{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"In this notebook we will train a <code>Word2Vec</code> model, aproximated nearest neighbour model and test it on local validation set.\n\nWe will use the <code>gensim</code> library for <code>Word2Vec</code>, aproximated nearest neighbour model form <code>annoy</code>.\n\nFor the following operation, we divide the training set into 2 subsets: training and testing. For this purpose, I used the [dataset](https://www.kaggle.com/datasets/radek1/otto-train-and-test-data-for-local-validation) provided by Radek Osmulski. The prepared sets allow you to test the developed recommendation system. A higher score on the validation set indicates a higher score on the test set and a better place on the LB (see [post](https://www.kaggle.com/competitions/otto-recommender-system/discussion/364991)).\n\nThe developed code in this notebook is based on the example prepared by Radek (see [nootebook](https://www.kaggle.com/code/radek1/word2vec-how-to-training-and-submission)). The fragment \"Evaluating submission on local validation set\" applies the code presented in [post](https://www.kaggle.com/competitions/otto-recommender-system/discussion/364991). We introduce some modifications that improve the products proposed by the system.\n\nTips for improving the model\n* While experimenting with the model, I noticed that changing the hyperparameters of the <code>Word2vec</code> model impacts the overall score. Especially parameters: <code>vector_size</code>, <code>window</code> and <code>negative</code>,\n* The number of trees in <code>annoy</code> model affects results.\n* To prepare the recommended aids list, it is worth using the last few aids. \n* <b>Version 3:</b> Recently discovred the <code>ns_exponent</code> parameter value influence.\n* <b>Version 3:</b> It is worth noting that selecting fewer than 20 similar products for each aid and summing up can produce a higher score.\n* <b>Version 4:</b> Using <code>skip-gram</code> instead of <code>CBOW</code> improves the result. Unfortunately, longer training time is required.\n* <b>Version 5:</b> For reproducibility purposes, a <code>seed</code> parameter value is fixed. Because of <code>worker</code> parameter is set to 4, it is recommended to provide your hashing function as an argument in <code>Word2vec</code>.\n* And there is more to find... 😉\n\n","metadata":{}},{"cell_type":"markdown","source":"# Data preprocessing","metadata":{}},{"cell_type":"code","source":"!pip install polars\nimport gc\nimport polars as pl\nfrom gensim.test.utils import common_texts\nfrom gensim.models import Word2Vec\n\ntrain_local = pl.read_parquet('../input/otto-train-and-test-data-for-local-validation/train.parquet')\ntest_local = pl.read_parquet('../input/otto-train-and-test-data-for-local-validation/test.parquet')","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-01-21T12:51:36.957586Z","iopub.execute_input":"2023-01-21T12:51:36.958128Z","iopub.status.idle":"2023-01-21T12:52:12.552647Z","shell.execute_reply.started":"2023-01-21T12:51:36.958036Z","shell.execute_reply":"2023-01-21T12:52:12.551479Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data = pl.concat([train_local, test_local]).unique()\n\nsentences_df = data.groupby('session').agg(\n    pl.col('aid').alias('sentence')\n)\ndel data; gc.collect() \n\nsentences = sentences_df['sentence'].to_list()\ndel sentences_df; gc.collect() ","metadata":{"execution":{"iopub.status.busy":"2023-01-21T12:52:12.554119Z","iopub.execute_input":"2023-01-21T12:52:12.554420Z","iopub.status.idle":"2023-01-21T12:54:17.016057Z","shell.execute_reply.started":"2023-01-21T12:52:12.554389Z","shell.execute_reply":"2023-01-21T12:54:17.014537Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we removed the repeated in the dataframe.","metadata":{}},{"cell_type":"markdown","source":"# Training word2vec model","metadata":{}},{"cell_type":"code","source":"def simple_hash_function(key):\n    return sum(\n        index * ord(character)\n        for index, character in enumerate(repr(key), start=1)\n    )","metadata":{"execution":{"iopub.status.busy":"2023-01-21T12:54:17.018312Z","iopub.execute_input":"2023-01-21T12:54:17.018694Z","iopub.status.idle":"2023-01-21T12:54:17.025013Z","shell.execute_reply.started":"2023-01-21T12:54:17.018664Z","shell.execute_reply":"2023-01-21T12:54:17.023733Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nw2vec = Word2Vec(sentences=sentences, vector_size= 64, window = 3, negative = 8, ns_exponent = 0.2, sg = 1, min_count=1, workers=4, seed = 1, hashfxn=simple_hash_function)","metadata":{"execution":{"iopub.status.busy":"2023-01-21T12:54:17.027606Z","iopub.execute_input":"2023-01-21T12:54:17.028883Z","iopub.status.idle":"2023-01-21T14:26:29.911761Z","shell.execute_reply.started":"2023-01-21T12:54:17.028838Z","shell.execute_reply":"2023-01-21T14:26:29.910321Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Preparing aproximated nearest neighbour model","metadata":{}},{"cell_type":"code","source":"%%time\n\nfrom annoy import AnnoyIndex\n\naid2idx = {aid: i for i, aid in enumerate(w2vec.wv.index_to_key)}\nindex = AnnoyIndex(64, 'euclidean')\n\nfor aid, idx in aid2idx.items():\n    index.add_item(idx, w2vec.wv.vectors[idx])\n    \nindex.build(32)","metadata":{"execution":{"iopub.status.busy":"2023-01-21T14:26:29.913832Z","iopub.execute_input":"2023-01-21T14:26:29.914698Z","iopub.status.idle":"2023-01-21T14:27:29.725439Z","shell.execute_reply.started":"2023-01-21T14:26:29.914657Z","shell.execute_reply":"2023-01-21T14:27:29.724388Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Creating aids predictions","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\n\nfrom collections import defaultdict\nimport collections\n\nsession_types = ['clicks', 'carts', 'orders']\ntest_session_AIDs = test_local.to_pandas().reset_index(drop=True).groupby('session')['aid'].apply(list)\ntest_session_types = test_local.to_pandas().reset_index(drop=True).groupby('session')['type'].apply(list)","metadata":{"execution":{"iopub.status.busy":"2023-01-21T14:27:29.726956Z","iopub.execute_input":"2023-01-21T14:27:29.727259Z","iopub.status.idle":"2023-01-21T14:28:32.496873Z","shell.execute_reply.started":"2023-01-21T14:27:29.727232Z","shell.execute_reply":"2023-01-21T14:28:32.495133Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"labels = []\n\ntype_weight_multipliers = {0: 1, 1: 6, 2: 3}\n\nsession_num = len(test_session_AIDs)\n\nfor AIDs, types in zip(test_session_AIDs[:session_num], test_session_types[:session_num]):\n    if len(AIDs) >= 20:\n        # if we have enough aids (over equals 20) we don't need to look for candidates! we just use the old logic\n        weights=np.logspace(0.1,1,len(AIDs),base=2, endpoint=True)-1\n        aids_temp=defaultdict(lambda: 0)\n        for aid,w,t in zip(AIDs,weights,types): \n            aids_temp[aid]+= w * type_weight_multipliers[t]\n            \n        sorted_aids=[k for k, v in sorted(aids_temp.items(), key=lambda item: -item[1])]\n        labels.append(sorted_aids[:20])\n    else:\n        # here we don't have 20 aids to output -- we will use word2vec embeddings to generate candidates!\n        AIDs = list(dict.fromkeys(AIDs[::-1]))\n        \n        # let's grab the up to 3 recent aids\n        recent_len = max(min(3,len(AIDs)),1)\n        \n        # how many aids for each aid\n        AIDs_num = round((20-len(AIDs))/recent_len) + 2\n        \n        # let's look for some neighbors!        \n        nns_it = []\n        for it in range(0,recent_len):\n            nns_it += [w2vec.wv.index_to_key[i] for i in index.get_nns_by_item(aid2idx[AIDs[it]], AIDs_num)[1:]]\n        \n        # select repeating and unique neighbors\n        nns_repeated = [item for item, count in collections.Counter(nns_it).items() if count > 1]\n        nns_once = [item for item, count in collections.Counter(nns_it).items() if count == 1]\n\n        # prepare selection\n        nns = (nns_repeated+nns_once)[:20]\n        labels.append((AIDs+nns)[:20])","metadata":{"execution":{"iopub.status.busy":"2023-01-21T14:28:32.498379Z","iopub.execute_input":"2023-01-21T14:28:32.498712Z","iopub.status.idle":"2023-01-21T14:35:49.716920Z","shell.execute_reply.started":"2023-01-21T14:28:32.498683Z","shell.execute_reply":"2023-01-21T14:35:49.715512Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Preparing submission dataframe","metadata":{}},{"cell_type":"code","source":"labels_as_strings = [' '.join([str(l) for l in lls]) for lls in labels]\n\npredictions = pd.DataFrame(data={'session_type': test_session_AIDs.index, 'labels': labels_as_strings})\n\nprediction_dfs = []\n\nfor st in session_types:\n    modified_predictions = predictions.copy()\n    modified_predictions.session_type = modified_predictions.session_type.astype('str') + f'_{st}'\n    prediction_dfs.append(modified_predictions)\n\nsubmission = pd.concat(prediction_dfs).reset_index(drop=True)\n\ndel labels, labels_as_strings, predictions, prediction_dfs\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2023-01-21T14:35:49.718424Z","iopub.execute_input":"2023-01-21T14:35:49.718796Z","iopub.status.idle":"2023-01-21T14:36:16.685711Z","shell.execute_reply.started":"2023-01-21T14:35:49.718763Z","shell.execute_reply":"2023-01-21T14:36:16.684666Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.head()","metadata":{"execution":{"iopub.status.busy":"2023-01-21T14:36:16.687066Z","iopub.execute_input":"2023-01-21T14:36:16.687982Z","iopub.status.idle":"2023-01-21T14:36:16.702976Z","shell.execute_reply.started":"2023-01-21T14:36:16.687913Z","shell.execute_reply":"2023-01-21T14:36:16.701770Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Evaluating submission on local validation set","metadata":{}},{"cell_type":"code","source":"submission['session'] = submission.session_type.apply(lambda x: int(x.split('_')[0]))\nsubmission['type'] = submission.session_type.apply(lambda x: x.split('_')[1])\nsubmission.labels = submission.labels.apply(lambda x: [int(i) for i in x.split(' ')[:20]])\n\ntest_labels = pd.read_parquet('../input/otto-train-and-test-data-for-local-validation/test_labels.parquet')\ntest_labels = test_labels.merge(submission, how='left', on=['session', 'type'])\ntest_labels['hits'] = test_labels.apply(lambda df: len(set(df.ground_truth).intersection(set(df.labels))), axis=1)\ntest_labels['gt_count'] = test_labels.ground_truth.str.len().clip(0,20)\n\nrecall_per_type = test_labels.groupby(['type'])['hits'].sum() / test_labels.groupby(['type'])['gt_count'].sum() \n\nscore = (recall_per_type * pd.Series({'clicks': 0.10, 'carts': 0.30, 'orders': 0.60})).sum()","metadata":{"execution":{"iopub.status.busy":"2023-01-21T14:36:16.705660Z","iopub.execute_input":"2023-01-21T14:36:16.705925Z","iopub.status.idle":"2023-01-21T14:38:01.568047Z","shell.execute_reply.started":"2023-01-21T14:36:16.705901Z","shell.execute_reply":"2023-01-21T14:38:01.566651Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Recall clicks: {:f} carts: {:f} orders {:f}\".format(recall_per_type[1],recall_per_type[0],recall_per_type[2]))\nprint(\"Score on validation data: {:f}\".format(score))","metadata":{"execution":{"iopub.status.busy":"2023-01-21T14:38:01.569467Z","iopub.execute_input":"2023-01-21T14:38:01.569808Z","iopub.status.idle":"2023-01-21T14:38:01.577107Z","shell.execute_reply.started":"2023-01-21T14:38:01.569778Z","shell.execute_reply":"2023-01-21T14:38:01.575150Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And you should have your score.\n\nWishing you all the best in the challenge! 🤞","metadata":{}}]}