{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 0.Overview\n\n**Edit:**\n\n* In my previous notebooks([here](https://www.kaggle.com/code/astrung/lstm-sequential-modelwith-item-features-tutorial) and [here](https://www.kaggle.com/code/astrung/lstm-sequential-modelwith-item-features-tutorial)), we have used test_data with `full_sort_topk`,but due to the limit of full_sort_topk we have missed last item for submited recommendation. Someone asked me about how can use all items as input features for recommendation in this [comment](https://www.kaggle.com/code/astrung/recbole-lstm-sequential-for-recomendation-tutorial/comments#1723707). \n* So i created a notebook [here](https://www.kaggle.com/code/astrung/recbole-using-all-items-for-prediction) for address there questions in detail, and this notebook is an improved of my [previous notebook](https://www.kaggle.com/code/astrung/lstm-sequential-modelwith-item-features-tutorial), applying our new function (using all item as input features without `full_sort_topk`) for this competition. In this notebook, we also use item features as input.\n* If you only want to use interaction as input feature, please check this [notebook](https://www.kaggle.com/astrung/lstm-model-with-item-infor-fix-missing-last-item).\n\n- - -\n\nIn previous [my notebook](https://www.kaggle.com/code/astrung/sequential-model-fixed-missing-last-item/), we tried to use GRU/LSTM model for testing effect of sequential model for recommendation with only iteration.\nIn this notebook, i showed how we can enhance sequential model with item features \n\nDue to memory limit and faster testing purpose, we will just use data in 2020.\n\nIf you want to use with all of interactions in all time, i have created a new atomic dataset here for you: \n\n* only interations data: https://www.kaggle.com/astrung/hm-atomic-interation\n* iterations + item features data: https://www.kaggle.com/astrung/hm-atomic-interation-with-item-feature \n\nWe also have other limit: we only train model and predict with users who buy more than 40 items and items which is bought by more than 40 people.\n\nWe will follow below steps for creating model:\n\n1. In order to use Recbole, we create atomic file from interaction data and item data\n2. Because we only use Recbole model for predicting with users who buy more than 40 items, other users will need to fill by default recomendation items. We create most viewed items in last month as defautl recomendation\n3. We create dataset and train model in recbole.\n4. We create prediction result by trained model\n5. We combine recomendation result from most viewed items in last month and Recbole predicted model.\n\nI will explain more detail in following cells.\n\n","metadata":{}},{"cell_type":"code","source":"!pip install recbole","metadata":{"execution":{"iopub.status.busy":"2022-03-20T03:28:15.95332Z","iopub.execute_input":"2022-03-20T03:28:15.953679Z","iopub.status.idle":"2022-03-20T03:28:32.484142Z","shell.execute_reply.started":"2022-03-20T03:28:15.953594Z","shell.execute_reply":"2022-03-20T03:28:32.483336Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1. Create atomic file","metadata":{}},{"cell_type":"markdown","source":"### 1.A create atomic of item features\nwe will create item features for feeding with iteration features into GRU4REC model ","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport gc\ndf = pd.read_csv(r\"/kaggle/input/h-and-m-personalized-fashion-recommendations/articles.csv\", dtype={'article_id': 'str'})\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-20T03:28:32.486105Z","iopub.execute_input":"2022-03-20T03:28:32.486393Z","iopub.status.idle":"2022-03-20T03:28:33.490226Z","shell.execute_reply.started":"2022-03-20T03:28:32.486357Z","shell.execute_reply":"2022-03-20T03:28:33.489571Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for col in df.columns:\n    print(col)\n    print(len(pd.unique(df[col])))","metadata":{"execution":{"iopub.status.busy":"2022-03-20T03:28:33.491599Z","iopub.execute_input":"2022-03-20T03:28:33.491863Z","iopub.status.idle":"2022-03-20T03:28:33.644072Z","shell.execute_reply.started":"2022-03-20T03:28:33.491829Z","shell.execute_reply":"2022-03-20T03:28:33.643362Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### we see so many couple of columns are [category_text, encoded_value]. So in order to avoid [Multicollinearity](https://link.springer.com/chapter/10.1007/978-0-585-25657-3_37), we will keep only one columns in each couple\nWe can see below couple of columns in item features, and we will keep one of them:\n\n* use product_type_no - skip product_type_name\n* use graphical_appearance_no - skip graphical_appearance_name\n* use colour_group_code - skip colour_group_name\n* use perceived_colour_value_id - skip perceived_colour_value_name\n* use perceived_colour_master_id - skip perceived_colour_master_name\n* use index_code - skip index_name\n* use index_group_no - skip index_group_name\n* use section_no - skip section_name\n* use garment_group_no - skip garment_group_name\n* use product_code, skip product_name\n* use department_no, skip department_name","metadata":{}},{"cell_type":"code","source":"df = df.drop(columns = ['product_type_name', 'graphical_appearance_name', 'colour_group_name', 'perceived_colour_value_name',\n                        'perceived_colour_master_name', 'index_name', 'index_group_name', 'section_name', \n                        'garment_group_name', 'prod_name', 'department_name', 'detail_desc'])\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-20T03:28:33.646162Z","iopub.execute_input":"2022-03-20T03:28:33.646415Z","iopub.status.idle":"2022-03-20T03:28:33.665466Z","shell.execute_reply.started":"2022-03-20T03:28:33.646379Z","shell.execute_reply":"2022-03-20T03:28:33.664504Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"temp = df.rename(\n    columns={'article_id': 'item_id:token', 'product_code': 'product_code:token', 'product_type_no': 'product_type_no:float',\n             'product_group_name': 'product_group_name:token_seq', 'graphical_appearance_no': 'graphical_appearance_no:token', \n             'colour_group_code': 'colour_group_code:token', 'perceived_colour_value_id': 'perceived_colour_value_id:token', \n             'perceived_colour_master_id': 'perceived_colour_master_id:token', 'department_no': 'department_no:token', \n             'index_code': 'index_code:token', 'index_group_no': 'index_group_no:token', 'section_no': 'section_no:token', \n             'garment_group_no': 'garment_group_no:token'})\ntemp.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-20T03:28:33.667191Z","iopub.execute_input":"2022-03-20T03:28:33.667589Z","iopub.status.idle":"2022-03-20T03:28:33.689815Z","shell.execute_reply.started":"2022-03-20T03:28:33.667553Z","shell.execute_reply":"2022-03-20T03:28:33.689028Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!mkdir /kaggle/working/recbox_data\ntemp.to_csv(r'/kaggle/working/recbox_data/recbox_data.item', index=False, sep='\\t')","metadata":{"execution":{"iopub.status.busy":"2022-03-20T03:28:33.691191Z","iopub.execute_input":"2022-03-20T03:28:33.691437Z","iopub.status.idle":"2022-03-20T03:28:34.837272Z","shell.execute_reply.started":"2022-03-20T03:28:33.691403Z","shell.execute_reply":"2022-03-20T03:28:34.836272Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 1.B create atomic of iteration features\nwe will create iteration features for GRU4REC model ","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv(r\"/kaggle/input/h-and-m-personalized-fashion-recommendations/transactions_train.csv\", \n                 dtype={'article_id': 'str'})\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-20T03:28:34.839271Z","iopub.execute_input":"2022-03-20T03:28:34.83957Z","iopub.status.idle":"2022-03-20T03:30:08.592215Z","shell.execute_reply.started":"2022-03-20T03:28:34.839528Z","shell.execute_reply":"2022-03-20T03:30:08.591532Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['t_dat'] = pd.to_datetime(df['t_dat'], format=\"%Y-%m-%d\")\ndf","metadata":{"execution":{"iopub.status.busy":"2022-03-20T03:30:08.593579Z","iopub.execute_input":"2022-03-20T03:30:08.594007Z","iopub.status.idle":"2022-03-20T03:30:14.347186Z","shell.execute_reply.started":"2022-03-20T03:30:08.593969Z","shell.execute_reply":"2022-03-20T03:30:14.346507Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\ndf['timestamp'] = df.t_dat.values.astype(np.int64) // 10 ** 9\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-20T03:30:14.348478Z","iopub.execute_input":"2022-03-20T03:30:14.348732Z","iopub.status.idle":"2022-03-20T03:30:14.859245Z","shell.execute_reply.started":"2022-03-20T03:30:14.348681Z","shell.execute_reply":"2022-03-20T03:30:14.856331Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**We fill with data in only 2020(timestamp > > 1585620000) and create inter file**\nFor anyone need instruction about inter file, please check below links:\n* https://recbole.io/docs/user_guide/data_intro.html\n* https://recbole.io/docs/user_guide/data/atomic_files.html\n\nif you want a full of iterations without limiting timestamp, please check here:\n* ","metadata":{}},{"cell_type":"code","source":"temp = df[df['timestamp'] > 1585620000][['customer_id', 'article_id', 'timestamp']].rename(\n    columns={'customer_id': 'user_id:token', 'article_id': 'item_id:token', 'timestamp': 'timestamp:float'})\ntemp","metadata":{"execution":{"iopub.status.busy":"2022-03-20T03:30:14.86588Z","iopub.execute_input":"2022-03-20T03:30:14.866154Z","iopub.status.idle":"2022-03-20T03:30:16.318768Z","shell.execute_reply.started":"2022-03-20T03:30:14.866117Z","shell.execute_reply":"2022-03-20T03:30:16.318078Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We save atomic file in dataset format for using with recbole","metadata":{}},{"cell_type":"code","source":"temp.to_csv('/kaggle/working/recbox_data/recbox_data.inter', index=False, sep='\\t')\ndel temp\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-03-20T03:30:16.320215Z","iopub.execute_input":"2022-03-20T03:30:16.320496Z","iopub.status.idle":"2022-03-20T03:30:50.635226Z","shell.execute_reply.started":"2022-03-20T03:30:16.32046Z","shell.execute_reply":"2022-03-20T03:30:50.634551Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. We create defautl recomendation for user who can not be predicted by sequential model.\nI use this approach in notebook: https://www.kaggle.com/hervind/h-m-faster-trending-products-weekly You can check it for more detail information. I will juse copy only code here","metadata":{}},{"cell_type":"code","source":"import os\nimport numpy as np\nimport pandas as pd","metadata":{"execution":{"iopub.status.busy":"2022-03-20T03:30:50.636395Z","iopub.execute_input":"2022-03-20T03:30:50.636724Z","iopub.status.idle":"2022-03-20T03:30:50.64104Z","shell.execute_reply.started":"2022-03-20T03:30:50.636671Z","shell.execute_reply":"2022-03-20T03:30:50.640002Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub0 = pd.read_csv('../input/hm-pre-recommendation/submissio_byfone_chris.csv').sort_values('customer_id').reset_index(drop=True)\nsub1 = pd.read_csv('../input/hm-pre-recommendation/submission_trending.csv').sort_values('customer_id').reset_index(drop=True)\nsub2 = pd.read_csv('../input/hm-pre-recommendation/submission_exponential_decay.csv').sort_values('customer_id').reset_index(drop=True)\n\nsub0.shape, sub1.shape, sub2.shape","metadata":{"execution":{"iopub.status.busy":"2022-03-20T03:30:50.642559Z","iopub.execute_input":"2022-03-20T03:30:50.642856Z","iopub.status.idle":"2022-03-20T03:31:44.175281Z","shell.execute_reply.started":"2022-03-20T03:30:50.642822Z","shell.execute_reply":"2022-03-20T03:31:44.174559Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub0.columns = ['customer_id', 'prediction0']\nsub0['prediction1'] = sub1['prediction']\nsub0['prediction2'] = sub2['prediction']\ndel sub1, sub2\ngc.collect()\nsub0.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-20T03:31:44.176416Z","iopub.execute_input":"2022-03-20T03:31:44.17682Z","iopub.status.idle":"2022-03-20T03:31:44.406286Z","shell.execute_reply.started":"2022-03-20T03:31:44.176782Z","shell.execute_reply":"2022-03-20T03:31:44.405611Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def cust_blend(dt, W = [1,1,1]):\n    #Global ensemble weights\n    #W = [1.15,0.95,0.85]\n    \n    #Create a list of all model predictions\n    REC = []\n    REC.append(dt['prediction0'].split())\n    REC.append(dt['prediction1'].split())\n    REC.append(dt['prediction2'].split())\n    \n    #Create a dictionary of items recommended. \n    #Assign a weight according the order of appearance and multiply by global weights\n    res = {}\n    for M in range(len(REC)):\n        for n, v in enumerate(REC[M]):\n            if v in res:\n                res[v] += (W[M]/(n+1))\n            else:\n                res[v] = (W[M]/(n+1))\n    \n    # Sort dictionary by item weights\n    res = list(dict(sorted(res.items(), key=lambda item: -item[1])).keys())\n    \n    # Return the top 12 itens only\n    return ' '.join(res[:12])\n\nsub0['prediction'] = sub0.apply(cust_blend, W = [1.05,1.00,0.95], axis=1)\nsub0.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-20T03:31:44.407335Z","iopub.execute_input":"2022-03-20T03:31:44.408039Z","iopub.status.idle":"2022-03-20T03:34:31.62021Z","shell.execute_reply.started":"2022-03-20T03:31:44.407999Z","shell.execute_reply":"2022-03-20T03:34:31.619549Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del sub0['prediction0']\ndel sub0['prediction1']\ndel sub0['prediction2']\ngc.collect()\nsub0.to_csv(f'submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-03-20T03:34:31.621522Z","iopub.execute_input":"2022-03-20T03:34:31.621777Z","iopub.status.idle":"2022-03-20T03:34:42.720925Z","shell.execute_reply.started":"2022-03-20T03:34:31.621742Z","shell.execute_reply":"2022-03-20T03:34:42.72018Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del sub0\ndel df\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-03-20T03:34:42.722062Z","iopub.execute_input":"2022-03-20T03:34:42.724059Z","iopub.status.idle":"2022-03-20T03:34:42.809975Z","shell.execute_reply.started":"2022-03-20T03:34:42.724028Z","shell.execute_reply":"2022-03-20T03:34:42.808313Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. Create dataset and train model with Recbole\n\nFor anyone need instruction document, please check this link: https://recbole.io/docs/user_guide/usage/use_modules.html","metadata":{}},{"cell_type":"code","source":"import logging\nfrom logging import getLogger\nfrom recbole.config import Config\nfrom recbole.data import create_dataset, data_preparation\nfrom recbole.model.sequential_recommender import GRU4RecF\nfrom recbole.trainer import Trainer\nfrom recbole.utils import init_seed, init_logger","metadata":{"execution":{"iopub.status.busy":"2022-03-20T03:34:42.8114Z","iopub.execute_input":"2022-03-20T03:34:42.81195Z","iopub.status.idle":"2022-03-20T03:34:45.133228Z","shell.execute_reply.started":"2022-03-20T03:34:42.811908Z","shell.execute_reply":"2022-03-20T03:34:45.132482Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"for limiting memory and time traning, we will filter for only using user who bought more than 40 items and item which is sold more than 40 times. If you want to train with more data, please change below config\n* user_inter_num_interval\n* item_inter_num_interval","metadata":{}},{"cell_type":"code","source":"parameter_dict = {\n    'data_path': '/kaggle/working',\n    'USER_ID_FIELD': 'user_id',\n    'ITEM_ID_FIELD': 'item_id',\n    'TIME_FIELD': 'timestamp',\n    'user_inter_num_interval': \"[40,inf)\",\n    'item_inter_num_interval': \"[40,inf)\",\n    'load_col': {'inter': ['user_id', 'item_id', 'timestamp'],\n                 'item': ['item_id', 'product_code', 'product_type_no', 'product_group_name', 'graphical_appearance_no',\n                      'colour_group_code', 'perceived_colour_value_id', 'perceived_colour_master_id',\n                      'department_no', 'index_code', 'index_group_no', 'section_no', 'garment_group_no']\n             },\n    'selected_features': ['product_code', 'product_type_no', 'product_group_name', 'graphical_appearance_no',\n                          'colour_group_code', 'perceived_colour_value_id', 'perceived_colour_master_id',\n                          'department_no', 'index_code', 'index_group_no', 'section_no', 'garment_group_no'],\n    'neg_sampling': None,\n    'epochs': 30,\n    'eval_args': {\n        'split': {'RS': [10, 0, 0]},\n        'group_by': 'user',\n        'order': 'TO',\n        'mode': 'full'}\n}\n\nconfig = Config(model='GRU4RecF', dataset='recbox_data', config_dict=parameter_dict)\n\n# init random seed\ninit_seed(config['seed'], config['reproducibility'])\n\n# logger initialization\ninit_logger(config)\nlogger = getLogger()\n# Create handlers\nc_handler = logging.StreamHandler()\nc_handler.setLevel(logging.INFO)\nlogger.addHandler(c_handler)\n\n# write config info into log\nlogger.info(config)","metadata":{"execution":{"iopub.status.busy":"2022-03-20T03:34:45.135619Z","iopub.execute_input":"2022-03-20T03:34:45.136054Z","iopub.status.idle":"2022-03-20T03:34:45.453796Z","shell.execute_reply.started":"2022-03-20T03:34:45.136018Z","shell.execute_reply":"2022-03-20T03:34:45.453245Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dataset = create_dataset(config)\nlogger.info(dataset)","metadata":{"execution":{"iopub.status.busy":"2022-03-20T03:34:45.454742Z","iopub.execute_input":"2022-03-20T03:34:45.454984Z","iopub.status.idle":"2022-03-20T03:36:01.446816Z","shell.execute_reply.started":"2022-03-20T03:34:45.454951Z","shell.execute_reply":"2022-03-20T03:36:01.446128Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# dataset splitting\ntrain_data, valid_data, test_data = data_preparation(config, dataset)","metadata":{"execution":{"iopub.status.busy":"2022-03-20T03:36:01.447896Z","iopub.execute_input":"2022-03-20T03:36:01.448163Z","iopub.status.idle":"2022-03-20T03:36:17.472516Z","shell.execute_reply.started":"2022-03-20T03:36:01.448124Z","shell.execute_reply":"2022-03-20T03:36:17.471845Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# model loading and initialization\nmodel = GRU4RecF(config, train_data.dataset).to(config['device'])\nlogger.info(model)\n\n# trainer loading and initialization\ntrainer = Trainer(config, model)\n\n# model training\nbest_valid_score, best_valid_result = trainer.fit(train_data)","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2022-03-20T03:36:17.473738Z","iopub.execute_input":"2022-03-20T03:36:17.473981Z","iopub.status.idle":"2022-03-20T04:00:34.160516Z","shell.execute_reply.started":"2022-03-20T03:36:17.473947Z","shell.execute_reply":"2022-03-20T04:00:34.159747Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. Create recommendation result from trained model\n\nI note document here for any one want to customize it: https://recbole.io/docs/user_guide/usage/case_study.html","metadata":{}},{"cell_type":"code","source":"from recbole.utils.case_study import full_sort_topk\nexternal_user_ids = dataset.id2token(\n    dataset.uid_field, list(range(dataset.user_num)))[1:]#fist element in array is 'PAD'(default of Recbole) ->remove it ","metadata":{"execution":{"iopub.status.busy":"2022-03-20T04:00:34.567899Z","iopub.status.idle":"2022-03-20T04:00:34.568681Z","shell.execute_reply.started":"2022-03-20T04:00:34.568423Z","shell.execute_reply":"2022-03-20T04:00:34.568452Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import torch\nfrom recbole.data.interaction import Interaction\n\ndef add_last_item(old_interaction, last_item_id, max_len=50):\n    new_seq_items = old_interaction['item_id_list'][-1]\n    if old_interaction['item_length'][-1].item() < max_len:\n        new_seq_items[old_interaction['item_length'][-1].item()] = last_item_id\n    else:\n        new_seq_items = torch.roll(new_seq_items, -1)\n        new_seq_items[-1] = last_item_id\n    return new_seq_items.view(1, len(new_seq_items))\n\ndef predict_for_all_item(external_user_id, dataset, model):\n    model.eval()\n    with torch.no_grad():\n        uid_series = dataset.token2id(dataset.uid_field, [external_user_id])\n        index = np.isin(dataset.inter_feat[dataset.uid_field].numpy(), uid_series)\n        input_interaction = dataset[index]\n        test = {\n            'item_id_list': add_last_item(input_interaction, \n                                          input_interaction['item_id'][-1].item(), model.max_seq_length),\n            'item_length': torch.tensor(\n                [input_interaction['item_length'][-1].item() + 1\n                 if input_interaction['item_length'][-1].item() < model.max_seq_length else model.max_seq_length])\n        }\n        new_inter = Interaction(test)\n        new_inter = new_inter.to(config['device'])\n        new_scores = model.full_sort_predict(new_inter)\n        new_scores = new_scores.view(-1, test_data.dataset.item_num)\n        new_scores[:, 0] = -np.inf  # set scores of [pad] to -inf\n    return torch.topk(new_scores, 12)","metadata":{"execution":{"iopub.status.busy":"2022-03-20T04:00:34.569761Z","iopub.status.idle":"2022-03-20T04:00:34.570516Z","shell.execute_reply.started":"2022-03-20T04:00:34.570281Z","shell.execute_reply":"2022-03-20T04:00:34.570307Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"predict_for_all_item('0109ad0b5a76924a1b58be677409bb601cc8bead9a87b8ce5b08a4a1f5bc71ef', \n                     dataset, model)","metadata":{"execution":{"iopub.status.busy":"2022-03-20T04:00:34.571717Z","iopub.status.idle":"2022-03-20T04:00:34.572472Z","shell.execute_reply.started":"2022-03-20T04:00:34.572226Z","shell.execute_reply":"2022-03-20T04:00:34.572261Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"topk_items = []\nfor external_user_id in external_user_ids:\n    _, topk_iid_list = predict_for_all_item(external_user_id, dataset, model)\n    last_topk_iid_list = topk_iid_list[-1]\n    external_item_list = dataset.id2token(dataset.iid_field, last_topk_iid_list.cpu()).tolist()\n    topk_items.append(external_item_list)\nprint(len(topk_items))","metadata":{"execution":{"iopub.status.busy":"2022-03-20T04:00:34.573662Z","iopub.status.idle":"2022-03-20T04:00:34.57443Z","shell.execute_reply.started":"2022-03-20T04:00:34.574183Z","shell.execute_reply":"2022-03-20T04:00:34.57421Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"external_item_str = [' '.join(x) for x in topk_items]\nresult = pd.DataFrame(external_user_ids, columns=['customer_id'])\nresult['prediction'] = external_item_str\nresult.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-20T04:00:34.575455Z","iopub.status.idle":"2022-03-20T04:00:34.575762Z","shell.execute_reply.started":"2022-03-20T04:00:34.575588Z","shell.execute_reply":"2022-03-20T04:00:34.575607Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del external_item_str\ndel topk_items\ndel external_user_ids\ndel train_data\ndel valid_data\ndel test_data\ndel model\ndel Trainer\ndel logger\ndel dataset\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-03-20T04:00:34.577589Z","iopub.status.idle":"2022-03-20T04:00:34.578222Z","shell.execute_reply.started":"2022-03-20T04:00:34.577985Z","shell.execute_reply":"2022-03-20T04:00:34.578009Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5. Combine result from most bought items and GRU model","metadata":{}},{"cell_type":"code","source":"submit_df = pd.read_csv('submission.csv')\nsubmit_df.shape","metadata":{"execution":{"iopub.status.busy":"2022-03-20T04:00:34.579444Z","iopub.status.idle":"2022-03-20T04:00:34.580077Z","shell.execute_reply.started":"2022-03-20T04:00:34.579843Z","shell.execute_reply":"2022-03-20T04:00:34.579868Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submit_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-20T04:00:34.58128Z","iopub.status.idle":"2022-03-20T04:00:34.581908Z","shell.execute_reply.started":"2022-03-20T04:00:34.581657Z","shell.execute_reply":"2022-03-20T04:00:34.581681Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submit_df = pd.merge(submit_df, result, on='customer_id', how='outer')\nsubmit_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-20T04:00:34.583085Z","iopub.status.idle":"2022-03-20T04:00:34.583732Z","shell.execute_reply.started":"2022-03-20T04:00:34.583483Z","shell.execute_reply":"2022-03-20T04:00:34.583508Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submit_df = submit_df.fillna(-1)\nsubmit_df['prediction'] = submit_df.apply(\n    lambda x: x['prediction_y'] if x['prediction_y'] != -1 else x['prediction_x'], axis=1)\nsubmit_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-20T04:00:34.584922Z","iopub.status.idle":"2022-03-20T04:00:34.58555Z","shell.execute_reply.started":"2022-03-20T04:00:34.58531Z","shell.execute_reply":"2022-03-20T04:00:34.585335Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submit_df = submit_df.drop(columns=['prediction_y', 'prediction_x'])\nsubmit_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-20T04:00:34.586749Z","iopub.status.idle":"2022-03-20T04:00:34.587378Z","shell.execute_reply.started":"2022-03-20T04:00:34.587134Z","shell.execute_reply":"2022-03-20T04:00:34.587159Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submit_df.to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-03-20T04:00:34.588558Z","iopub.status.idle":"2022-03-20T04:00:34.589283Z","shell.execute_reply.started":"2022-03-20T04:00:34.588995Z","shell.execute_reply":"2022-03-20T04:00:34.589028Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}