{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"A co-visitation matrix is essentially an \"analog\" approximation to matrix factorization! I talk a bit more about this idea here: [💡 What is the co-visitation matrix, really?](https://www.kaggle.com/competitions/otto-recommender-system/discussion/365358).\n\nBut matrix factorization has a lot of advantages as compared to co-visitation matrices. First of all, it can make better use of data -- it operates on the notion of similarity between categories. We can construct a more powerful representation if our model understands that aid `1` is similar to aid `142` as opposed to it treating each aid as an atomic entity (this is the jump from unigram/bigram/trigram models to word2vec in NLP).\n\nLet us thus train a matrix factorization model and replace the co-visitation matrices with it!\n\nNow, I don't expect that the first version of the model will be particularly well tuned. There has already been a lot of work put into co-visitation matrices and in the later versions we work off 3 different matrices, one for each category of actions! A similar progression can and will happen with matrix factorization 🙂 This notebook hopefully will enable us to jumpstart this type of exploration 🙂\n\nTo streamline the work, we will use data in `parquet` format. (Here is the notebook [💡 [Howto] Full dataset as parquet/csv files](https://www.kaggle.com/code/radek1/howto-full-dataset-as-parquet-csv-files) and here is [the most up-to-date version of the dataset](https://www.kaggle.com/datasets/radek1/otto-full-optimized-memory-footprint), no need for dealing with `jasonl` files and the associated mess any longer! Please upvote if you find this useful!)\n\nFor data processing we will use [polars](https://www.pola.rs/). `Polars` has a much smaller memory footprint than `pandas` and is quite fast. Plus it has really clean, intuitive API.\n\nLet's get to work! 🙂\n\n\n## Other resources you might find useful:\n\n* [💡 Training an XGBoost Ranker on the GPU with Merlin Models 🔥🔥🔥](https://www.kaggle.com/competitions/otto-recommender-system/discussion/368848)\n* [How to train a Word2Vec model 🚀🚀🚀](https://www.kaggle.com/competitions/otto-recommender-system/discussion/368384)\n* [💡 Can you beat static rules with a ranker model without additional features?](https://www.kaggle.com/competitions/otto-recommender-system/discussion/366474)\n* [🐘 the elephant in the room -- high cardinality of targets and what to do about this](https://www.kaggle.com/competitions/otto-recommender-system/discussion/364722)\n* [📖 What are some good resources to learn about how gradient-boosted tree ranking models work?](https://www.kaggle.com/competitions/otto-recommender-system/discussion/366477)\n* [💡How to ensemble predictions -- a key component to every strong solution 🏅](https://www.kaggle.com/competitions/otto-recommender-system/discussion/368747)\n* [from zero to 60 in 2 seconds or less 🏎️🚓🚓🚓](https://www.kaggle.com/competitions/otto-recommender-system/discussion/367058)\n* [💡What is a good initial goal in the competition? How to improve beyond it? 📈](https://www.kaggle.com/competitions/otto-recommender-system/discussion/368685)\n* [💡How to improve the results of your Approximate Nearest Neighbor search! (annoy)](https://www.kaggle.com/competitions/otto-recommender-system/discussion/368385)\n* [📅 Dataset for local validation created using organizer's repository (parquet files)](https://www.kaggle.com/competitions/otto-recommender-system/discussion/364534)\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"code","source":"!pip install merlin-dataloader==0.0.2\n\nfrom merlin.loader.torch import Loader ","metadata":{"execution":{"iopub.status.busy":"2022-12-09T13:39:51.749438Z","iopub.execute_input":"2022-12-09T13:39:51.750284Z","iopub.status.idle":"2022-12-09T13:41:44.862562Z","shell.execute_reply.started":"2022-12-09T13:39:51.750233Z","shell.execute_reply":"2022-12-09T13:41:44.861285Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!du -sh *","metadata":{"execution":{"iopub.status.busy":"2022-12-09T12:57:55.227577Z","iopub.execute_input":"2022-12-09T12:57:55.228008Z","iopub.status.idle":"2022-12-09T12:57:55.512427Z","shell.execute_reply.started":"2022-12-09T12:57:55.227975Z","shell.execute_reply":"2022-12-09T12:57:55.510473Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"torch.save(the_model.state_dict(), PATH)# Data Preprocessing","metadata":{}},{"cell_type":"code","source":"!pip install \"modin[dask]\"\n\nimport os\n\nos.environ[\"MODIN_ENGINE\"] = \"ray\"  # Modin will use Dask\n\nimport modin.pandas as mpd\nimport pandas as pd\n","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2022-12-09T02:02:52.338908Z","iopub.execute_input":"2022-12-09T02:02:52.339367Z","iopub.status.idle":"2022-12-09T02:03:05.949146Z","shell.execute_reply.started":"2022-12-09T02:02:52.339305Z","shell.execute_reply":"2022-12-09T02:03:05.947348Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from merlin.loader.torch import Loader \nfrom merlin.io import Dataset\nfrom torch import nn","metadata":{"execution":{"iopub.status.busy":"2022-12-09T02:03:05.952117Z","iopub.execute_input":"2022-12-09T02:03:05.952771Z","iopub.status.idle":"2022-12-09T02:03:05.961088Z","shell.execute_reply.started":"2022-12-09T02:03:05.952717Z","shell.execute_reply":"2022-12-09T02:03:05.959456Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# import json\n# import ray\n\n# ray.init(\n#     _system_config={\n#         # Allow spilling until the local disk is 99% utilized.\n#         # This only affects spilling to the local file system.\n#         \"local_fs_capacity_threshold\": 0.95,\n#         \"object_spilling_config\": json.dumps({\n#               \"type\": \"filesystem\",\n#               \"params\": {\n#                 \"directory_path\": \"/kaggle/working/spill\",\n#             }\n#         })\n#     },\n# )\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# from distributed import Client\n\n# client = Client(processes=True, threads_per_worker=1)\n# import ray\n# ray.init()\ntrain = pd.read_parquet('../input/otto-full-optimized-memory-footprint/train.parquet')\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\ntrain['ts'] = pd.to_datetime(train['ts'], unit='s')\ntrain.head(3)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.shape","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\ntrain = train[(train.ts > np.datetime64('2022-08-21'))]\ntrain.shape","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import gc\ngc.collect()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!rm -r spill/","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"unique_sessions_split = np.array_split(train.session.unique(), 20)\n\ni = 0\nfor split_session in unique_sessions_split:\n    print(i)\n    train[train.session.isin(split_session)].to_parquet(f\"split_train_{i}\")\n    i+=1","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[train.session.isin(split_session)].info()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test = pl.read_parquet('../input/otto-full-optimized-memory-footprint/test.parquet')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We need to create `aid-aid` pairs to train our matrix factorization model!\n\nLet's us grab the pairs both from the train and test set.","metadata":{}},{"cell_type":"code","source":"%%time\n\ntrain_pairs = (pl.concat([train, test])\n    .groupby('session').agg([\n        pl.col('aid'),\n        pl.col('aid').shift(-1).alias('aid_next')\n    ])\n    .explode(['aid', 'aid_next'])\n    .drop_nulls()\n)[['aid', 'aid_next']]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_pairs.shape[0] / 1_000_000","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"That is 209 million pairs created in 40 seconds without running out of RAM! 🙂 Not too bad","metadata":{}},{"cell_type":"code","source":"train_pairs.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's see what is the cardinality of our aids -- we will need this to create the embedding layer.","metadata":{}},{"cell_type":"code","source":"cardinality_aids = max(train_pairs['aid'].max(), train_pairs['aid_next'].max())\ncardinality_aids","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cardinality_aids = 1855602 ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We will have up to `1855602` -- that is a lot! But our matrix factorization model will be able to handle this.\n\nLet's construct a `PyTorch` dataset and `dataloader`.","metadata":{}},{"cell_type":"code","source":"\n\nclass ClicksDataset(Dataset):\n    def __init__(self, pairs):\n        self.aid1 = pairs['aid'].to_numpy()\n        self.aid2 = pairs['aid_next'].to_numpy()\n    def __getitem__(self, idx):\n        aid1 = self.aid1[idx]\n        aid2 = self.aid2[idx]\n        return [aid1, aid2]\n    def __len__(self):\n        return len(self.aid1)\n\n# train_ds = ClicksDataset(train_pairs[:-10_000_000])\n# valid_ds = ClicksDataset(train_pairs[10_000_000:])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let us see how quickly we can iterate over a single epoch with a batch size of `65536`.","metadata":{}},{"cell_type":"code","source":"train_ds = ClicksDataset(train_pairs)\ntrain_dl_pytorch = DataLoader(train_ds, 65536, True, num_workers=2)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\n# for batch in train_dl_pytorch:\n#     aid1, aid2 = batch[0], batch[1]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Oh dear, that took forever! Mind you, were are not doing anything here, apart from iterating over the dataset for a single epoch (and that is without validation!).\n\nThe reason this is taking so long is that indexing into the the arrays and collating results into batches is very computationally expensive.\n\nThere are ways to work around this but they require writing a lot of code (you could use the iterable-style dataset). And still our solution wouldn't be particularly well optimized.\n\nLet us do something else instead!\n\nWe will use a brand new [Merlin Dataloader](https://github.com/NVIDIA-Merlin/dataloader). It is a library that my team launched just a couple of days ago 🙂\n\nNow this library shines when you have a GPU, which is what you generally want when training DL models. But, alas, Kaggle gives you only 13 GB of RAM on a kernel with a GPU, and that wouldn't allow us to process our dataset!\n\nLet's see how far we can get with CPU only.","metadata":{}},{"cell_type":"markdown","source":"We can read data directly from the disk -- even better!\n\nLet's write our datasets to disk.","metadata":{}},{"cell_type":"code","source":"train_pairs[:-10_000_000].to_pandas().to_parquet('train_pairs.parquet')\ntrain_pairs[-10_000_000:].to_pandas().to_parquet('valid_pairs.parquet')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\ntrain_ds = pl.DataFrame(pd.read_parquet(\"train_pairs.parquet\"))\nvalid_ds = pl.DataFrame(pd.read_parquet(\"valid_pairs.parquet\"))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\ntrain_ds = Dataset('train_pairs.parquet')\ntrain_dl_merlin = Loader(train_ds, 65536, True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# %%time\n\n# for batch in train_dl_merlin:\n#     aid1, aid2 = batch[0], batch[1]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"That is much better 🙂. Let's train our matrix factorization model!","metadata":{}},{"cell_type":"code","source":"class MatrixFactorization(nn.Module):\n    def __init__(self, n_aids, n_factors):\n        super().__init__()\n        self.aid_factors = nn.Embedding(n_aids, n_factors, sparse=True)\n        \n    def forward(self, aid1, aid2):\n        aid1 = self.aid_factors(aid1)\n        aid2 = self.aid_factors(aid2)\n        \n        return (aid1 * aid2).sum(dim=1)\n    \nclass AverageMeter(object):\n    \"\"\"Computes and stores the average and current value\"\"\"\n    def __init__(self, name, fmt=':f'):\n        self.name = name\n        self.fmt = fmt\n        self.reset()\n\n    def reset(self):\n        self.val = 0\n        self.avg = 0\n        self.sum = 0\n        self.count = 0\n\n    def update(self, val, n=1):\n        self.val = val\n        self.sum += val * n\n        self.count += n\n        self.avg = self.sum / self.count\n\n    def __str__(self):\n        fmtstr = '{name} {val' + self.fmt + '} ({avg' + self.fmt + '})'\n        return fmtstr.format(**self.__dict__)\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nvalid_ds = Dataset('valid_pairs.parquet')\nvalid_dl_merlin = Loader(valid_ds, 65536, True)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from torch.optim import SparseAdam\n\nnum_epochs=1\nlr=0.1\ncardinality_aids = 1855602 \nmodel = MatrixFactorization(cardinality_aids+1, 32)\noptimizer = SparseAdam(model.parameters(), lr=lr)\ncriterion = nn.BCEWithLogitsLoss()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import torch\ntorch.save(model.state_dict(), 'model_weights.pth')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import torch\ndevice = torch.device(\"cuda\")\nmodel.cuda()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"num_epochs = 50","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\nfor epoch in range(num_epochs):\n    print(\"Current epoch\", epoch)\n    for batch, _ in train_dl_merlin:\n\n        model.train()\n\n        losses = AverageMeter('Loss', ':.4e')\n\n        aid1, aid2 = batch['aid'], batch['aid_next']\n\n        output_pos = model(aid1, aid2)\n\n        output_neg = model(aid1, aid2[torch.randperm(aid2.shape[0])])\n\n        \n        output = torch.cat([output_pos, output_neg])\n        targets = torch.cat([torch.ones_like(output_pos), torch.zeros_like(output_pos)])\n\n        loss = criterion(output, targets)\n        losses.update(loss.item())\n        \n        optimizer.zero_grad()\n        loss.backward()\n        optimizer.step()\n        \n    model.eval()\n    \n    with torch.no_grad():\n        accuracy = AverageMeter('accuracy')\n        for batch, _ in valid_dl_merlin:\n            aid1, aid2 = batch['aid'], batch['aid_next']\n            output_pos = model(aid1, aid2)\n            output_neg = model(aid1, aid2[torch.randperm(aid2.shape[0])])\n            accuracy_batch = torch.cat([output_pos.sigmoid() > 0.5, output_neg.sigmoid() < 0.5]).float().mean()\n            accuracy.update(accuracy_batch, aid1.shape[0])\n            \n    print(f'{epoch+1:02d}: * TrainLoss {losses.avg:.3f}  * Accuracy {accuracy.avg:.3f}')","metadata":{"scrolled":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import torch\ntorch.save(model.state_dict(), 'model_weights.pth')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Later to restore:\nimport torch\nmodel.load_state_dict(torch.load(\"model_weights.pth\", map_location=torch.device('cpu')))\nmodel.eval()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's grab the embeddings!","metadata":{}},{"cell_type":"code","source":"embeddings = model.aid_factors.weight.detach().cpu().numpy()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And construct create the index for approximate nearest neighbor search.","metadata":{}},{"cell_type":"code","source":"import pickle\npickle.dump(embeddings, open(\"embeddings.pkl\", \"wb\"))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pickle\n\n\nembeddings = pickle.load(open(\"embeddings.pkl\", \"rb\"))","metadata":{"execution":{"iopub.status.busy":"2022-12-09T02:03:44.465805Z","iopub.execute_input":"2022-12-09T02:03:44.466389Z","iopub.status.idle":"2022-12-09T02:03:51.109266Z","shell.execute_reply.started":"2022-12-09T02:03:44.466339Z","shell.execute_reply":"2022-12-09T02:03:51.107660Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\nfrom annoy import AnnoyIndex\n\nindex = AnnoyIndex(32, 'euclidean')\nfor i, v in enumerate(embeddings):\n#     print(i, v)\n#     input()\n    index.add_item(i, v)\n    \nindex.build(10)","metadata":{"execution":{"iopub.status.busy":"2022-12-09T02:03:51.111885Z","iopub.execute_input":"2022-12-09T02:03:51.112364Z","iopub.status.idle":"2022-12-09T02:04:23.358144Z","shell.execute_reply.started":"2022-12-09T02:03:51.112300Z","shell.execute_reply":"2022-12-09T02:04:23.356662Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now for any `aid`, we can find its nearest neighbor!","metadata":{}},{"cell_type":"code","source":"index.get_nns_by_item(123, 10)","metadata":{"execution":{"iopub.status.busy":"2022-12-09T02:04:23.360200Z","iopub.execute_input":"2022-12-09T02:04:23.360743Z","iopub.status.idle":"2022-12-09T02:04:23.370765Z","shell.execute_reply.started":"2022-12-09T02:04:23.360699Z","shell.execute_reply":"2022-12-09T02:04:23.369005Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's create a submission! 🙂","metadata":{}},{"cell_type":"code","source":"from collections import defaultdict\nsession_types = ['clicks', 'carts', 'orders']\ntest_session_AIDs = test.to_pandas().reset_index(drop=True).groupby('session')['aid'].apply(list)\ntest_session_types = test.to_pandas().reset_index(drop=True).groupby('session')['type'].apply(list)\n\nlabels = []\n\ntype_weight_multipliers = {0: 1, 1: 6, 2: 3}\ndmf_test_candidates = defaultdict(set)\ni = 0\nc = 0\nfor AIDs, types in zip(test_session_AIDs, test_session_types):\n    session = test_session_AIDs.index[i]\n    rec = None\n    if len(AIDs) >= 20:\n        # if we have enough aids (over equals 20) we don't need to look for candidates! we just use the old logic\n        weights=np.logspace(0.1,1,len(AIDs),base=2, endpoint=True)-1\n        aids_temp=defaultdict(lambda: 0)\n        for aid,w,t in zip(AIDs,weights,types): \n            aids_temp[aid]+= w * type_weight_multipliers[t]\n            \n        sorted_aids=[k for k, v in sorted(aids_temp.items(), key=lambda item: -item[1])]\n        rec = set(sorted_aids[:20])\n        labels.append(rec)\n#         dmf_test_candidates[session] = set(rec)\n    else:\n        # here we don't have 20 aids to output -- we will use approximate nearest neighbor search and our embeddings\n        # to generate candidates!\n        AIDs = list(dict.fromkeys(AIDs[::-1]))\n        \n        # let's grab the most recent aid\n        most_recent_aid = AIDs[0]\n        \n        # and look for some neighbors!\n        nns = index.get_nns_by_item(most_recent_aid, 21)[1:]\n        rec = (AIDs+nns)[:20]\n        labels.append(rec)\n        dmf_test_candidates[session] = set(rec)\n        c+=1\n    i+=1\n    \nprint(c)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pickle\npickle.dump(dmf_test_candidates, open(\"dmf_test_candidates.pkl\", \"wb\"))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"scrolled":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from collections import defaultdict\nimport numpy as np\nimport gc\nimport glob\ngc.collect()\n\n# labels = []\n\n\ntype_weight_multipliers = {0: 1, 1: 6, 2: 3}\ndmf_train_candidates = defaultdict(set)\nc = 0\n\n\nfor chunk_name in glob.glob(\"split_train_*\"):\n    \n    chunk = pd.read_parquet(chunk_name)\n    print(chunk_name, chunk.shape[0])\n    session_types = ['clicks', 'carts', 'orders']\n    test_session_AIDs = chunk.reset_index(drop=True).groupby('session')['aid'].apply(list)\n    test_session_types = chunk.reset_index(drop=True).groupby('session')['type'].apply(list)\n    \n    i = 0\n    for AIDs, types in zip(test_session_AIDs, test_session_types):\n#         gc.collect()\n        if i % 100 == 0:\n            print(i/chunk.shape[0])\n        session = test_session_AIDs.index[i]\n        rec = None\n        if len(AIDs) >= 20:\n#             if we have enough aids (over equals 20) we don't need to look for candidates! we just use the old logic\n            weights=np.logspace(0.1,1,len(AIDs),base=2, endpoint=True)-1\n            aids_temp=defaultdict(lambda: 0)\n            for aid,w,t in zip(AIDs,weights,types): \n                aids_temp[aid]+= w * type_weight_multipliers[t]\n\n            sorted_aids=[k for k, v in sorted(aids_temp.items(), key=lambda item: -item[1])]\n            rec = set(sorted_aids[:20])\n#             labels.append(rec)\n            dmf_train_candidates[session] = set(rec)\n        else:\n            # here we don't have 20 aids to output -- we will use approximate nearest neighbor search and our embeddings\n            # to generate candidates!\n            AIDs = list(dict.fromkeys(AIDs[::-1]))\n            # let's grab the most recent aid\n            most_recent_aid = int(AIDs[0])\n\n            # and look for some neighbors!\n            nns = index.get_nns_by_item(most_recent_aid, 21)[1:]\n            rec = (AIDs+nns)[:20]\n\n            dmf_train_candidates[session] = set(rec)\n\n            del AIDs\n#             gc.collect()\n            c+=1\n        i+=1\n    print(c)\nprint(c)","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2022-12-09T02:04:38.090868Z","iopub.execute_input":"2022-12-09T02:04:38.091468Z","iopub.status.idle":"2022-12-09T02:24:26.570453Z","shell.execute_reply.started":"2022-12-09T02:04:38.091420Z","shell.execute_reply":"2022-12-09T02:24:26.568074Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pickle\npickle.dump(dmf_train_candidates, open(\"dmf_train_candidates.pkl\", \"wb\"))","metadata":{"execution":{"iopub.status.busy":"2022-12-09T02:24:26.575007Z","iopub.execute_input":"2022-12-09T02:24:26.575875Z","iopub.status.idle":"2022-12-09T02:26:54.184014Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's now pull it all together and write to a file,","metadata":{}},{"cell_type":"code","source":"labels_as_strings = [' '.join([str(l) for l in lls]) for lls in labels]\n\npredictions = pd.DataFrame(data={'session_type': test_session_AIDs.index, 'labels': labels_as_strings})\n\nprediction_dfs = []\n\nfor st in session_types:\n    modified_predictions = predictions.copy()\n    modified_predictions.session_type = modified_predictions.session_type.astype('str') + f'_{st}'\n    prediction_dfs.append(modified_predictions)\n\nsubmission = pd.concat(prediction_dfs).reset_index(drop=True)\nsubmission.to_csv('submission2.csv', index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And we are done!\n\n\n**If you like this notebook, please smash the upvote button! Thank you! 😊**\n\nThere are many ways in which this can be expanded:\n* we can train on the GPU\n* we can train for longer\n* maybe we would get better results if we were to filter our train data by type?\n* should we train only on adjacent aids? maybe we should expand the neighborhood we train on\n\nWe can keep asking ourselves many questions like this 🙂 Now we have a framework to start answering them!\n\nThank you for reading! Happy Kaggling! 🙌","metadata":{"execution":{"iopub.status.busy":"2022-11-18T02:49:02.940358Z","iopub.execute_input":"2022-11-18T02:49:02.940858Z","iopub.status.idle":"2022-11-18T02:49:02.973867Z","shell.execute_reply.started":"2022-11-18T02:49:02.94076Z","shell.execute_reply":"2022-11-18T02:49:02.972223Z"}}},{"cell_type":"code","source":"submission","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\nfrom IPython.display import FileLink\n\nFileLink(r'submission.csv')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}