{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Experiment on @radek1's notebook on local validation framework","metadata":{}},{"cell_type":"markdown","source":"This notebook builds on my previous work -- [co-visitation matrix - simplified, imprvd logic 🔥](https://www.kaggle.com/code/radek1/co-visitation-matrix-simplified-imprvd-logic?scriptVersionId=110068977) that achieves 0.558 on the LB.\n\nHere we take the functionality from that notebook, run on 1/1000 of the data (it achieves ~0.487 on public LB).\n\nThe next step in improving our results is to create a robust local validation framework to facilitate experimentation. This can be a stepping stone towards a much stronger result.\n\nLet's take a stab at implementing a local validation framework in this notebook!\n\n<strong>Please smash that thumbs up button if you like this notebook! Thank you! 🙂</strong>","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"code","source":"import os\n\ntry: import fastkaggle\nexcept ModuleNotFoundError:\n    os.system(\"pip install -Uq fastkaggle\")\n\nfrom fastkaggle import *\n\n# use fastdebug.utils \nif iskaggle: os.system(\"pip install nbdev snoop\")\n\nif iskaggle:\n    path = \"../input/fastdebugutils0\"\n    import sys\n    sys.path\n    sys.path.insert(1, path)\n    import utils as fu\n    from utils import *\nelse: \n    from fastdebug.utils import *\n    import fastdebug.utils as fu","metadata":{"execution":{"iopub.status.busy":"2022-11-19T13:42:53.195429Z","iopub.execute_input":"2022-11-19T13:42:53.195949Z","iopub.status.idle":"2022-11-19T13:43:24.234849Z","shell.execute_reply.started":"2022-11-19T13:42:53.195855Z","shell.execute_reply":"2022-11-19T13:43:24.233997Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\n!pip install pickle5\nimport pickle5 as pickle","metadata":{"execution":{"iopub.status.busy":"2022-11-19T13:43:24.236356Z","iopub.execute_input":"2022-11-19T13:43:24.237634Z","iopub.status.idle":"2022-11-19T13:43:37.319956Z","shell.execute_reply.started":"2022-11-19T13:43:24.237597Z","shell.execute_reply":"2022-11-19T13:43:37.318602Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ntrain = pd.read_parquet('../input/otto-full-optimized-memory-footprint/train.parquet')","metadata":{"execution":{"iopub.status.busy":"2022-11-19T13:43:37.323229Z","iopub.execute_input":"2022-11-19T13:43:37.323577Z","iopub.status.idle":"2022-11-19T13:43:56.493495Z","shell.execute_reply.started":"2022-11-19T13:43:37.323544Z","shell.execute_reply":"2022-11-19T13:43:56.492576Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### rd: recsys - otto - local validation - As we only use the last week of training set to split into the local test and local validation set\n# test = pd.read_parquet('../input/otto-full-optimized-memory-footprint/test.parquet') ","metadata":{"execution":{"iopub.status.busy":"2022-11-19T08:09:38.367041Z","iopub.execute_input":"2022-11-19T08:09:38.367346Z","iopub.status.idle":"2022-11-19T08:09:38.371432Z","shell.execute_reply.started":"2022-11-19T08:09:38.367319Z","shell.execute_reply":"2022-11-19T08:09:38.370278Z"}}},{"cell_type":"code","source":"DO_LOCAL_VALIDATION = True # set it True if we are doing local validation instead of submit to the public leaderboard","metadata":{"execution":{"iopub.status.busy":"2022-11-19T13:43:56.494866Z","iopub.execute_input":"2022-11-19T13:43:56.495401Z","iopub.status.idle":"2022-11-19T13:43:56.499477Z","shell.execute_reply.started":"2022-11-19T13:43:56.495368Z","shell.execute_reply":"2022-11-19T13:43:56.498587Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Local CV","metadata":{}},{"cell_type":"markdown","source":"For local CV, we will use the last weeks data of train for validation.\n\nEssentially, without modifying the calculations in the notebook, we can run evaluation locally if we replace the contents of the `train` and `test` variables.\n\nWhen doing local validation, we will print out local results. And without it, we will train on full data and submit to Kaggle LB.","metadata":{}},{"cell_type":"markdown","source":"### rd: recsys - otto - local validation - Find the start and end datetime of training sessions","metadata":{}},{"cell_type":"code","source":"ts_min, ts_max = train.ts.min(), train.ts.max()\nts_min, ts_max","metadata":{"execution":{"iopub.status.busy":"2022-11-19T13:43:56.502822Z","iopub.execute_input":"2022-11-19T13:43:56.503169Z","iopub.status.idle":"2022-11-19T13:43:56.888203Z","shell.execute_reply.started":"2022-11-19T13:43:56.503138Z","shell.execute_reply":"2022-11-19T13:43:56.887243Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### rd: recsys - otto - local validation - Version 2 of the Radek's dataset is second accuracy,`7*24*60*60` capture the length of actual 7 days; version 1 is millisecond accuracy and  using `7*24*60*60*1000` to capture 7 days length. see the accuracy difference in details [here](https://www.kaggle.com/code/danielliao/process-data-otto?scriptVersionId=111357696&cellId=29)","metadata":{}},{"cell_type":"code","source":"import datetime\ndatetime.date.fromtimestamp(ts_max), datetime.date.fromtimestamp(ts_max - 7*24*60*60), datetime.date.fromtimestamp(ts_max - 7*24*60*60*1000)","metadata":{"execution":{"iopub.status.busy":"2022-11-19T13:43:56.889562Z","iopub.execute_input":"2022-11-19T13:43:56.890022Z","iopub.status.idle":"2022-11-19T13:43:56.901266Z","shell.execute_reply.started":"2022-11-19T13:43:56.889978Z","shell.execute_reply":"2022-11-19T13:43:56.899983Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### rd: recsys - otto - local validation - ts where to cut - train_cutoff = ts_max - seven_days # 1_056_923_999 = 1_661_723_999 - 604_800","metadata":{}},{"cell_type":"code","source":"seven_days = 7*24*60*60 # 604_800\ntrain_cutoff = ts_max - seven_days # 1_056_923_999 = 1_661_723_999 - 604_800\nts_max > train_cutoff > ts_min","metadata":{"execution":{"iopub.status.busy":"2022-11-19T13:43:56.903068Z","iopub.execute_input":"2022-11-19T13:43:56.903532Z","iopub.status.idle":"2022-11-19T13:43:56.914974Z","shell.execute_reply.started":"2022-11-19T13:43:56.903484Z","shell.execute_reply":"2022-11-19T13:43:56.913821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"datetime.datetime.fromtimestamp(0), datetime.datetime.fromtimestamp(train_cutoff), \\\ndatetime.datetime.fromtimestamp(ts_min), datetime.datetime.fromtimestamp(ts_max), \\","metadata":{"execution":{"iopub.status.busy":"2022-11-19T13:43:56.916324Z","iopub.execute_input":"2022-11-19T13:43:56.916658Z","iopub.status.idle":"2022-11-19T13:43:56.925976Z","shell.execute_reply.started":"2022-11-19T13:43:56.916629Z","shell.execute_reply":"2022-11-19T13:43:56.924990Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### rd: recsys - otto - local validation - split train into local_train and local_test - local_train = train[train.ts <= train_cutoff] - local_test = train[train.ts > train_cutoff]","metadata":{}},{"cell_type":"code","source":"train.shape, train.memory_usage() # 216_716_096 rows","metadata":{"execution":{"iopub.status.busy":"2022-11-19T13:43:56.927053Z","iopub.execute_input":"2022-11-19T13:43:56.927458Z","iopub.status.idle":"2022-11-19T13:43:56.941241Z","shell.execute_reply.started":"2022-11-19T13:43:56.927425Z","shell.execute_reply":"2022-11-19T13:43:56.939931Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"local_train = train[train.ts <= train_cutoff] # 163_955_181 rows, and RAM on session metrics doubled from 4.6G to 9.1G","metadata":{"execution":{"iopub.status.busy":"2022-11-19T13:43:56.943252Z","iopub.execute_input":"2022-11-19T13:43:56.943732Z","iopub.status.idle":"2022-11-19T13:44:03.819717Z","shell.execute_reply.started":"2022-11-19T13:43:56.943691Z","shell.execute_reply":"2022-11-19T13:44:03.818394Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# run\nlocal_train.shape, local_train.memory_usage() # the Index is a huge number on RAM","metadata":{"execution":{"iopub.status.busy":"2022-11-19T13:44:03.821308Z","iopub.execute_input":"2022-11-19T13:44:03.821677Z","iopub.status.idle":"2022-11-19T13:44:03.831281Z","shell.execute_reply.started":"2022-11-19T13:44:03.821645Z","shell.execute_reply":"2022-11-19T13:44:03.829822Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# run\nlocal_test = train[train.ts > train_cutoff] # 52_760_915 rows, and RAM raise from 9.1G to 10.1G","metadata":{"execution":{"iopub.status.busy":"2022-11-19T13:44:03.833025Z","iopub.execute_input":"2022-11-19T13:44:03.833362Z","iopub.status.idle":"2022-11-19T13:44:06.166099Z","shell.execute_reply.started":"2022-11-19T13:44:03.833332Z","shell.execute_reply":"2022-11-19T13:44:06.164696Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# run \nlocal_test.shape, local_test.memory_usage() # the Index is a huge number on RAM","metadata":{"execution":{"iopub.status.busy":"2022-11-19T13:44:06.167646Z","iopub.execute_input":"2022-11-19T13:44:06.168020Z","iopub.status.idle":"2022-11-19T13:44:06.178308Z","shell.execute_reply.started":"2022-11-19T13:44:06.167988Z","shell.execute_reply":"2022-11-19T13:44:06.177099Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### rd: recsys - otto - local validation - How train.reset_index work? - help(train.reset_index) ","metadata":{}},{"cell_type":"code","source":"# help(train.reset_index) \n#     >>> df = pd.DataFrame([('bird', 389.0),\n#     ...                    ('bird', 24.0),\n#     ...                    ('mammal', 80.5),\n#     ...                    ('mammal', np.nan)],\n#     ...                   index=['falcon', 'parrot', 'lion', 'monkey'],\n#     ...                   columns=('class', 'max_speed'))\n#     >>> df\n#              class  max_speed\n#     falcon    bird      389.0\n#     parrot    bird       24.0\n#     lion    mammal       80.5\n#     monkey  mammal        NaN\n    \n#     When we reset the index, the old index is added as a column, and a\n#     new sequential index is used:\n    \n#     >>> df.reset_index()\n#         index   class  max_speed\n#     0  falcon    bird      389.0\n#     1  parrot    bird       24.0\n#     2    lion  mammal       80.5\n#     3  monkey  mammal        NaN\n    \n#     We can use the `drop` parameter to avoid the old index being added as\n#     a column:\n    \n#     >>> df.reset_index(drop=True)\n#         class  max_speed\n#     0    bird      389.0\n#     1    bird       24.0\n#     2  mammal       80.5\n#     3  mammal        NaN","metadata":{"execution":{"iopub.status.busy":"2022-11-19T13:44:06.183465Z","iopub.execute_input":"2022-11-19T13:44:06.184390Z","iopub.status.idle":"2022-11-19T13:44:06.194587Z","shell.execute_reply.started":"2022-11-19T13:44:06.184344Z","shell.execute_reply":"2022-11-19T13:44:06.193199Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### rd: recsys - otto - local validation - Save RAM by converting local_train.index from Int64Index to RangeIndex like train.index? - train.index, local_train.index, local_test.index - local_train.reset_index(inplace=True, drop=True)\nhttps://stackoverflow.com/questions/54603378/pandas-convert-from-int64index-to-rangeindex","metadata":{}},{"cell_type":"code","source":"train.index, local_train.index, local_test.index # RangeIndex vs Int64Index","metadata":{"execution":{"iopub.status.busy":"2022-11-19T13:44:06.196616Z","iopub.execute_input":"2022-11-19T13:44:06.197086Z","iopub.status.idle":"2022-11-19T13:44:06.216229Z","shell.execute_reply.started":"2022-11-19T13:44:06.197045Z","shell.execute_reply":"2022-11-19T13:44:06.214850Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### convert index from Int64Range to RangeIndex","metadata":{}},{"cell_type":"code","source":"local_train.reset_index(inplace=True, drop=True) # no effect on RAM from the session metrics board\nlocal_train.index, local_train.memory_usage() # but the number for Index dropped drastically\n# previously the local_train.index RAM usage is 1311641448, now is 128","metadata":{"execution":{"iopub.status.busy":"2022-11-19T13:44:06.217678Z","iopub.execute_input":"2022-11-19T13:44:06.218785Z","iopub.status.idle":"2022-11-19T13:44:06.231722Z","shell.execute_reply.started":"2022-11-19T13:44:06.218713Z","shell.execute_reply":"2022-11-19T13:44:06.230380Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"local_test.reset_index(inplace=True, drop=True) # no effect on RAM from the session metrics board\nlocal_test.index, local_test.memory_usage() # but the number for Index dropped drastically\n# the previous RAM usage of local_test.index is 422087320","metadata":{"execution":{"iopub.status.busy":"2022-11-19T13:44:06.233195Z","iopub.execute_input":"2022-11-19T13:44:06.233641Z","iopub.status.idle":"2022-11-19T13:44:06.247270Z","shell.execute_reply.started":"2022-11-19T13:44:06.233597Z","shell.execute_reply":"2022-11-19T13:44:06.245958Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del train # RAM dropped from 10.1G to 7.5G according to session metrics","metadata":{"execution":{"iopub.status.busy":"2022-11-19T13:44:06.248695Z","iopub.execute_input":"2022-11-19T13:44:06.249278Z","iopub.status.idle":"2022-11-19T13:44:06.256438Z","shell.execute_reply.started":"2022-11-19T13:44:06.249220Z","shell.execute_reply":"2022-11-19T13:44:06.255565Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### rd: recsys - otto - local validation - what are the benefits of removing intersecting sessions between local_train and local_test to simulate real world - overlapping_sessions = set(local_train.session).intersection(set(local_test.session))","metadata":{}},{"cell_type":"code","source":"%%time\noverlapping_sessions = set(local_train.session).intersection(set(local_test.session)) # not use use RAM","metadata":{"execution":{"iopub.status.busy":"2022-11-19T13:44:06.257425Z","iopub.execute_input":"2022-11-19T13:44:06.258173Z","iopub.status.idle":"2022-11-19T13:44:33.861715Z","shell.execute_reply.started":"2022-11-19T13:44:06.258139Z","shell.execute_reply":"2022-11-19T13:44:33.860473Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### rd: recsys - otto - local validation - the portion of intersection sessions on local_train and local_test is large. What would happen when adding those sessions back? better score or worse score? (question)","metadata":{}},{"cell_type":"code","source":"len(overlapping_sessions), len(overlapping_sessions)/local_train.session.unique().shape[0], len(overlapping_sessions)/local_test.session.unique().shape[0]\n\n# 3_521_833, 11_098_528, 5_323_084","metadata":{"execution":{"iopub.status.busy":"2022-11-19T13:44:33.863135Z","iopub.execute_input":"2022-11-19T13:44:33.863591Z","iopub.status.idle":"2022-11-19T13:44:36.195476Z","shell.execute_reply.started":"2022-11-19T13:44:33.863557Z","shell.execute_reply":"2022-11-19T13:44:36.194175Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nlocal_test = local_test[~local_test.session.isin(overlapping_sessions)] # RAM raise from 7.5 to 7.9\nlocal_test.index, local_test.memory_usage()","metadata":{"execution":{"iopub.status.busy":"2022-11-19T13:44:36.197497Z","iopub.execute_input":"2022-11-19T13:44:36.197873Z","iopub.status.idle":"2022-11-19T13:44:38.551161Z","shell.execute_reply.started":"2022-11-19T13:44:36.197842Z","shell.execute_reply":"2022-11-19T13:44:38.549803Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"local_test.reset_index(inplace=True, drop=True) # but not reduce RAM according to session metrics, stays at 7.9G\nlocal_test.index, local_test.memory_usage()","metadata":{"execution":{"iopub.status.busy":"2022-11-19T13:44:38.552980Z","iopub.execute_input":"2022-11-19T13:44:38.553316Z","iopub.status.idle":"2022-11-19T13:44:38.562214Z","shell.execute_reply.started":"2022-11-19T13:44:38.553286Z","shell.execute_reply":"2022-11-19T13:44:38.561039Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### rd: recsys - otto - local validation - any empty rows in any sessions of local_test - local_test.groupby('session')['aid'].count().apply(lambda x: x == 0)","metadata":{}},{"cell_type":"code","source":"\ncount_zero = local_test.groupby('session')['aid'].count().apply(lambda x: x == 0)\ncount_one = local_test.groupby('session')['aid'].count().apply(lambda x: x > 0)","metadata":{"execution":{"iopub.status.busy":"2022-11-19T13:44:38.563616Z","iopub.execute_input":"2022-11-19T13:44:38.564501Z","iopub.status.idle":"2022-11-19T13:44:40.595117Z","shell.execute_reply.started":"2022-11-19T13:44:38.564465Z","shell.execute_reply":"2022-11-19T13:44:40.593998Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sum(count_zero), sum(count_one), local_test.session.unique().shape[0]","metadata":{"execution":{"iopub.status.busy":"2022-11-19T13:44:40.596305Z","iopub.execute_input":"2022-11-19T13:44:40.597218Z","iopub.status.idle":"2022-11-19T13:44:41.092966Z","shell.execute_reply.started":"2022-11-19T13:44:40.597182Z","shell.execute_reply":"2022-11-19T13:44:41.091731Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### rd: recsys - otto - local validation - split local_test into test and validation two parts - for grp in local_test.groupby('session'): -     cutoff = np.random.randint(1, grp[1].shape[0]) - new_test.append(grp[1].iloc[:cutoff]) -     data_to_calculate_validation_score.append(grp[1].iloc[cutoff:])","metadata":{}},{"cell_type":"code","source":"%%time\nnew_test = []\ndata_to_calculate_validation_score = []\n\nfor grp in local_test.groupby('session'): # loop each session of local_test\n    # select a random row from the session as cutoff row, we want at least a single item in our validation data for each session\n    cutoff = np.random.randint(1, grp[1].shape[0]) \n    new_test.append(grp[1].iloc[:cutoff]) # take the left part from cutoff as data samples for local test\n    data_to_calculate_validation_score.append(grp[1].iloc[cutoff:]) # take the right part from the cutoff as data samples for local validation","metadata":{"execution":{"iopub.status.busy":"2022-11-19T13:44:41.094898Z","iopub.execute_input":"2022-11-19T13:44:41.095241Z","iopub.status.idle":"2022-11-19T13:47:36.257263Z","shell.execute_reply.started":"2022-11-19T13:44:41.095211Z","shell.execute_reply":"2022-11-19T13:47:36.256451Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### rd: recsys - otto - local validation - stack a list of smaller dfs onto each other - test = pd.concat(new_test).reset_index(drop=True) - valid = pd.concat(data_to_calculate_validation_score).reset_index(drop=True)","metadata":{}},{"cell_type":"code","source":"%%time\ntest = pd.concat(new_test).reset_index(drop=True) # stack a list of smaller dfs onto each otehr\nvalid = pd.concat(data_to_calculate_validation_score).reset_index(drop=True) # maximum to 24G","metadata":{"execution":{"iopub.status.busy":"2022-11-19T13:49:12.883985Z","iopub.execute_input":"2022-11-19T13:49:12.884406Z","iopub.status.idle":"2022-11-19T13:55:09.959515Z","shell.execute_reply.started":"2022-11-19T13:49:12.884369Z","shell.execute_reply":"2022-11-19T13:55:09.958158Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test.shape, test.memory_usage(), valid.shape, valid.memory_usage()","metadata":{"execution":{"iopub.status.busy":"2022-11-19T13:55:09.961747Z","iopub.execute_input":"2022-11-19T13:55:09.962132Z","iopub.status.idle":"2022-11-19T13:55:09.974950Z","shell.execute_reply.started":"2022-11-19T13:55:09.962096Z","shell.execute_reply":"2022-11-19T13:55:09.973663Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ntest.to_parquet('_test.parquet') # save to a parquet file\nvalid.to_parquet('_valid.parquet')\n\ndel new_test, data_to_calculate_validation_score # now dropped to 9.8G","metadata":{"execution":{"iopub.status.busy":"2022-11-19T13:55:09.976282Z","iopub.execute_input":"2022-11-19T13:55:09.976787Z","iopub.status.idle":"2022-11-19T13:55:28.750165Z","shell.execute_reply.started":"2022-11-19T13:55:09.976716Z","shell.execute_reply":"2022-11-19T13:55:28.748913Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del local_test # from 9.8G down to 9.3G according to session metrics","metadata":{"execution":{"iopub.status.busy":"2022-11-19T13:55:28.753780Z","iopub.execute_input":"2022-11-19T13:55:28.754819Z","iopub.status.idle":"2022-11-19T13:55:28.764068Z","shell.execute_reply.started":"2022-11-19T13:55:28.754770Z","shell.execute_reply":"2022-11-19T13:55:28.763188Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = local_train","metadata":{"execution":{"iopub.status.busy":"2022-11-19T13:55:28.765372Z","iopub.execute_input":"2022-11-19T13:55:28.766157Z","iopub.status.idle":"2022-11-19T13:55:28.780888Z","shell.execute_reply.started":"2022-11-19T13:55:28.766117Z","shell.execute_reply":"2022-11-19T13:55:28.779564Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"DO_LOCAL_VALIDATION = True","metadata":{"execution":{"iopub.status.busy":"2022-11-19T13:55:28.782560Z","iopub.execute_input":"2022-11-19T13:55:28.782964Z","iopub.status.idle":"2022-11-19T13:55:28.791204Z","shell.execute_reply.started":"2022-11-19T13:55:28.782930Z","shell.execute_reply":"2022-11-19T13:55:28.789926Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have now swapped the train and test sets for the ones we conjured and can now proceed to train as we would normally.","metadata":{}},{"cell_type":"markdown","source":"# Train","metadata":{}},{"cell_type":"markdown","source":"### rd: recsys - otto - local validation - create subset on both train and test - lucky_sessions_train = train.drop_duplicates(['session']).sample(frac=fraction_of_sessions_to_use)['session'] - subset_of_train = train[train.session.isin(lucky_sessions_train)]","metadata":{}},{"cell_type":"code","source":"fraction_of_sessions_to_use = 1/1000","metadata":{"execution":{"iopub.status.busy":"2022-11-19T13:55:28.793063Z","iopub.execute_input":"2022-11-19T13:55:28.793412Z","iopub.status.idle":"2022-11-19T13:55:28.802021Z","shell.execute_reply.started":"2022-11-19T13:55:28.793381Z","shell.execute_reply":"2022-11-19T13:55:28.800630Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\nlucky_sessions_train = train.drop_duplicates(['session']).sample(frac=fraction_of_sessions_to_use)['session']\nsubset_of_train = train[train.session.isin(lucky_sessions_train)]\n\nlucky_sessions_test = test.drop_duplicates(['session']).sample(frac=fraction_of_sessions_to_use)['session']\nsubset_of_test = test[test.session.isin(lucky_sessions_test)]\n\n# now session metrics reports RAM to be 10.5GB","metadata":{"execution":{"iopub.status.busy":"2022-11-19T13:55:28.804089Z","iopub.execute_input":"2022-11-19T13:55:28.804482Z","iopub.status.idle":"2022-11-19T13:55:38.273726Z","shell.execute_reply.started":"2022-11-19T13:55:28.804446Z","shell.execute_reply":"2022-11-19T13:55:38.272140Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### rd: recsys - otto - local validation - Add session as index for the subsets (train and test) - subset_of_train.index = pd.MultiIndex.from_frame(subset_of_train[['session']])","metadata":{}},{"cell_type":"code","source":"subset_of_train.index = pd.MultiIndex.from_frame(subset_of_train[['session']])\nsubset_of_test.index = pd.MultiIndex.from_frame(subset_of_test[['session']]) # now effect on RAM","metadata":{"execution":{"iopub.status.busy":"2022-11-19T13:55:38.275798Z","iopub.execute_input":"2022-11-19T13:55:38.276347Z","iopub.status.idle":"2022-11-19T13:55:38.296912Z","shell.execute_reply.started":"2022-11-19T13:55:38.276292Z","shell.execute_reply":"2022-11-19T13:55:38.295364Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"subset_of_train.shape, subset_of_train.session.unique().shape[0], subset_of_test.shape, subset_of_test.session.unique().shape[0],","metadata":{"execution":{"iopub.status.busy":"2022-11-19T13:55:38.300158Z","iopub.execute_input":"2022-11-19T13:55:38.301454Z","iopub.status.idle":"2022-11-19T13:55:38.314541Z","shell.execute_reply.started":"2022-11-19T13:55:38.301415Z","shell.execute_reply":"2022-11-19T13:55:38.313095Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### rd: recsys - otto - local validation - each the last 30 events of each session, make a cartesian product on each event, remove rows with the same aids, only select rows two aids occurred consecutively within a day, and doing it in large chunk/batch of sessions each loop, put each chunk of sessions as an item into a list (see src below)","metadata":{}},{"cell_type":"code","source":"%%time\n\nall_consecutive_AIDs = [] \nchunk_size = 60_000\n\nsessions = subset_of_train.session.unique() # all unique sessions of subset_of_train\nfor i in range(0, sessions.shape[0], chunk_size): # loop every 60_000 sessions, not rows (each session has multiple rows)\n    # take every 60_000 sessions (with all rows in each session) put into a new df named current_chunk (also remove session as index)\n    current_chunk = subset_of_train.loc[sessions[i]:sessions[min(sessions.shape[0]-1, i+chunk_size-1)]].reset_index(drop=True)\n    # update current_chunk by selecting the last 30 rows of each session (make sure no additional index)\n    current_chunk = current_chunk.groupby('session').apply(lambda g: g.tail(30)).reset_index(drop=True)\n    # @radek1 has a nice explanation on this line of code, see the cell below\n    # consecutive_AIDs contains 60_000 sessions, each aid has paired with every aids    \n    consecutive_AIDs = current_chunk.merge(current_chunk, on='session')\n    # remove all rows where the pair of aids are the same\n    consecutive_AIDs = consecutive_AIDs[consecutive_AIDs.aid_x != consecutive_AIDs.aid_y]\n    # add a column named 'days_elapsed' to record how many days passed between two aids\n    # whether divided by 1000 or not should make no difference in RAM, as they are all float64 type\n    consecutive_AIDs['days_elapsed'] = (consecutive_AIDs.ts_y - consecutive_AIDs.ts_x) / (24 * 60 * 60) # not 24*60*60*1000\n    # select only rows where first aid comes before second aid and both occurred in the same day\n    consecutive_AIDs = consecutive_AIDs[(consecutive_AIDs.days_elapsed > 0) & (consecutive_AIDs.days_elapsed <= 1)]\n    # put every 60_000 session df processed above into a list\n    all_consecutive_AIDs.append(\n        consecutive_AIDs\n    )","metadata":{"execution":{"iopub.status.busy":"2022-11-19T14:01:14.772615Z","iopub.execute_input":"2022-11-19T14:01:14.773262Z","iopub.status.idle":"2022-11-19T14:01:18.471656Z","shell.execute_reply.started":"2022-11-19T14:01:14.773221Z","shell.execute_reply":"2022-11-19T14:01:18.470197Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### How @radek1 explains `current_chunk.merge(current_chunk, on='session')` above from [discussion](https://www.kaggle.com/code/radek1/co-visitation-matrix-simplified-imprvd-logic/comments#2031136)\nso that we can create a cartesian product of aids\n\nessentially, it is a trick to create aid pairs by session\n\nif a user had three aids in a session 1, 2, 3 this will create all possible pairs [1, 1], [1,2], [1,3], [2,1]… etc","metadata":{}},{"cell_type":"markdown","source":"### rd: recsys - otto - local validaiton - apply the same logic above to the subset_of_test, and append them to consecutive_AIDs which is the same list that stores sessions in subset_of_train (see src)","metadata":{}},{"cell_type":"code","source":"%%time\nsessions = subset_of_test.session.unique()\nfor i in range(0, sessions.shape[0], chunk_size):\n    current_chunk = subset_of_test.loc[sessions[i]:sessions[min(sessions.shape[0]-1, i+chunk_size-1)]].reset_index(drop=True)\n    current_chunk = current_chunk.groupby('session').apply(lambda g: g.tail(30)).reset_index(drop=True)\n    consecutive_AIDs = current_chunk.merge(current_chunk, on='session')\n    consecutive_AIDs = consecutive_AIDs[consecutive_AIDs.aid_x != consecutive_AIDs.aid_y]\n    consecutive_AIDs['days_elapsed'] = (consecutive_AIDs.ts_y - consecutive_AIDs.ts_x) / (24 * 60 * 60 * 1000)\n    consecutive_AIDs = consecutive_AIDs[(consecutive_AIDs.days_elapsed > 0) & (consecutive_AIDs.days_elapsed <= 1)]\n    all_consecutive_AIDs.append(\n        consecutive_AIDs\n    )","metadata":{"execution":{"iopub.status.busy":"2022-11-19T14:02:27.996860Z","iopub.execute_input":"2022-11-19T14:02:27.997385Z","iopub.status.idle":"2022-11-19T14:02:28.531969Z","shell.execute_reply.started":"2022-11-19T14:02:27.997348Z","shell.execute_reply":"2022-11-19T14:02:28.530792Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### rd: recsys - otto - local validation - check the rows with duplicated values on 3 specified columns - all_yet.duplicated(['session', 'aid_x', 'aid_y']) - and remove rows from the dataframe - all_yet.drop_duplicates(['session', 'aid_x', 'aid_y'])","metadata":{}},{"cell_type":"code","source":"all_yet = pd.concat(all_consecutive_AIDs)","metadata":{"execution":{"iopub.status.busy":"2022-11-19T14:04:51.550372Z","iopub.execute_input":"2022-11-19T14:04:51.550898Z","iopub.status.idle":"2022-11-19T14:04:51.567101Z","shell.execute_reply.started":"2022-11-19T14:04:51.550860Z","shell.execute_reply":"2022-11-19T14:04:51.565933Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"all_yet.head()","metadata":{"execution":{"iopub.status.busy":"2022-11-19T14:04:59.067064Z","iopub.execute_input":"2022-11-19T14:04:59.067435Z","iopub.status.idle":"2022-11-19T14:04:59.088648Z","shell.execute_reply.started":"2022-11-19T14:04:59.067400Z","shell.execute_reply":"2022-11-19T14:04:59.087565Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"all_yet.duplicated(['session', 'aid_x', 'aid_y']).sum(), all_yet.shape[0]\nall_yet.drop_duplicates(['session', 'aid_x', 'aid_y']).shape\nall_yet[all_yet.duplicated(['session', 'aid_x', 'aid_y'])]","metadata":{"execution":{"iopub.status.busy":"2022-11-19T14:35:16.349241Z","iopub.execute_input":"2022-11-19T14:35:16.349748Z","iopub.status.idle":"2022-11-19T14:35:16.569885Z","shell.execute_reply.started":"2022-11-19T14:35:16.349707Z","shell.execute_reply":"2022-11-19T14:35:16.568262Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dup = all_yet[all_yet.duplicated(['session', 'aid_x', 'aid_y'])]\ndup1 = dup.loc[dup.aid_y == 398187]\ndup1.loc[dup1.aid_x == 1762221]","metadata":{"execution":{"iopub.status.busy":"2022-11-19T14:36:43.619868Z","iopub.execute_input":"2022-11-19T14:36:43.620275Z","iopub.status.idle":"2022-11-19T14:36:43.702565Z","shell.execute_reply.started":"2022-11-19T14:36:43.620240Z","shell.execute_reply":"2022-11-19T14:36:43.701223Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### rd: recsys - otto - local validation - question - selection with two conditionals - all_yet.loc[(all_yet.session == 1890 & all_yet.aid_x == 1762221), :]","metadata":{}},{"cell_type":"code","source":"# all_yet.loc[(all_yet.session == 1890 & all_yet.aid_x == 1762221), :] # ","metadata":{"execution":{"iopub.status.busy":"2022-11-19T14:37:57.182699Z","iopub.execute_input":"2022-11-19T14:37:57.183108Z","iopub.status.idle":"2022-11-19T14:37:57.187294Z","shell.execute_reply.started":"2022-11-19T14:37:57.183076Z","shell.execute_reply":"2022-11-19T14:37:57.186100Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### rd: recsys - otto - local validation - stack all dfs inside all_consecutive_AIDs into a single df and remove the rows when their session, aid_x, aid_y are the same - all_consecutive_AIDs = pd.concat(all_consecutive_AIDs).drop_duplicates(['session', 'aid_x', 'aid_y'])[['aid_x', 'aid_y']]","metadata":{}},{"cell_type":"code","source":"%%time\nall_consecutive_AIDs = pd.concat(all_consecutive_AIDs).drop_duplicates(['session', 'aid_x', 'aid_y'])[['aid_x', 'aid_y']]","metadata":{"execution":{"iopub.status.busy":"2022-11-19T14:41:03.288691Z","iopub.execute_input":"2022-11-19T14:41:03.289126Z","iopub.status.idle":"2022-11-19T14:41:03.409230Z","shell.execute_reply.started":"2022-11-19T14:41:03.289092Z","shell.execute_reply":"2022-11-19T14:41:03.407799Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"all_consecutive_AIDs.head()","metadata":{"execution":{"iopub.status.busy":"2022-11-19T14:41:04.527144Z","iopub.execute_input":"2022-11-19T14:41:04.528210Z","iopub.status.idle":"2022-11-19T14:41:04.538629Z","shell.execute_reply.started":"2022-11-19T14:41:04.528169Z","shell.execute_reply":"2022-11-19T14:41:04.537231Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### rd: recsys - otto - local validation - across all sessions, for each (aid_x, aid_y) pair, count and accumulate the occurrences of aid_y - next_AIDs = defaultdict(Counter) - for row in all_consecutive_AIDs.itertuples(): - next_AIDs[row.aid_x][row.aid_y] += 1","metadata":{}},{"cell_type":"code","source":"%%time\n\nfrom collections import defaultdict, Counter\n\nnext_AIDs = defaultdict(Counter)\n\nfor row in all_consecutive_AIDs.itertuples():\n    next_AIDs[row.aid_x][row.aid_y] += 1","metadata":{"execution":{"iopub.status.busy":"2022-11-19T14:49:06.912419Z","iopub.execute_input":"2022-11-19T14:49:06.913344Z","iopub.status.idle":"2022-11-19T14:49:07.568719Z","shell.execute_reply.started":"2022-11-19T14:49:06.913293Z","shell.execute_reply":"2022-11-19T14:49:07.567102Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for k,v in next_AIDs.items(): \n    print(k)\n    print(v)\n    break","metadata":{"execution":{"iopub.status.busy":"2022-11-19T14:49:15.686550Z","iopub.execute_input":"2022-11-19T14:49:15.687021Z","iopub.status.idle":"2022-11-19T14:49:15.693865Z","shell.execute_reply.started":"2022-11-19T14:49:15.686985Z","shell.execute_reply":"2022-11-19T14:49:15.692582Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(next_AIDs), all_consecutive_AIDs.aid_x.unique().shape[0]","metadata":{"execution":{"iopub.status.busy":"2022-11-19T14:50:10.026785Z","iopub.execute_input":"2022-11-19T14:50:10.027276Z","iopub.status.idle":"2022-11-19T14:50:10.040612Z","shell.execute_reply.started":"2022-11-19T14:50:10.027237Z","shell.execute_reply":"2022-11-19T14:50:10.039300Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## rd: recsys - otto - local validation - Now let's generate the predictions or labels from test set and validation set is to provide ground truth - get all aids of each test session into a list - test_session_AIDs = test.groupby('session')['aid'].apply(list)","metadata":{"execution":{"iopub.status.busy":"2022-11-04T00:06:55.37083Z","iopub.execute_input":"2022-11-04T00:06:55.371203Z","iopub.status.idle":"2022-11-04T00:06:55.389977Z","shell.execute_reply.started":"2022-11-04T00:06:55.371178Z","shell.execute_reply":"2022-11-04T00:06:55.388167Z"}}},{"cell_type":"code","source":"%%time\n# get aid into a list for every session in the processed local_test\ntest_session_AIDs = test.groupby('session')['aid'].apply(list)\ntest_session_AIDs.head()","metadata":{"execution":{"iopub.status.busy":"2022-11-19T14:52:16.408353Z","iopub.execute_input":"2022-11-19T14:52:16.408703Z","iopub.status.idle":"2022-11-19T14:52:50.088918Z","shell.execute_reply.started":"2022-11-19T14:52:16.408671Z","shell.execute_reply":"2022-11-19T14:52:50.087583Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"execution":{"iopub.status.busy":"2022-11-16T14:50:19.166340Z","iopub.execute_input":"2022-11-16T14:50:19.166725Z","iopub.status.idle":"2022-11-16T14:50:19.172459Z","shell.execute_reply.started":"2022-11-16T14:50:19.166687Z","shell.execute_reply":"2022-11-16T14:50:19.170930Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### rd: recsys - otto - robust local validation - debug a block of code by making it a func and use print and return","metadata":{}},{"cell_type":"markdown","source":"**The original code** here and see debuggable version in the next cell\n```python\nlabels = []\n\nfor AIDs in test_session_AIDs: # loop each session to get its list of aids\n    AIDs = list(dict.fromkeys(AIDs[::-1])) # reverse the order of the list of aids and remove duplicated aids and return a list\n    if len(AIDs) >= 20: # if there are more than 20 aids in the session, then takes the first 20 aids into labels (a list)\n        labels.append(AIDs[:20])\n    else:\n        counter = Counter()\n        for AID in AIDs: # loop every aid of the list of aid from the session\n            subsequent_AID_counter = next_AIDs.get(AID) # use next_AIDs to access all the pair partners of AID, which is a counter\n            # if the counter exist, merge subsequent_AID_counter to counter (counter is still a single counter with a lot more items each time)\n            if subsequent_AID_counter:\n                counter += subsequent_AID_counter\n        # now, counter is a Counter contains a Counter of other aids for every aid in a session (which means a lot of other aids counts)\n        # take the 40 most common other aids, if they are not already in AIDs, then add them to AIDs\n        AIDs += [AID for AID, cnt in counter.most_common(40) if AID not in AIDs]\n        # in the end only take the first 20 aids into labels for each session\n        labels.append(AIDs[:20])\n```","metadata":{}},{"cell_type":"markdown","source":"### rd: recsys - otto - local validation - create the labels/predictions for each test session/user - reverse the list of aids of each session, remove the duplicated aids, and select the first 20 aids as labels - and save it into a list 'labels' - question: should we use the learning from training set here? (see src below)","metadata":{}},{"cell_type":"markdown","source":"### rd: recsys - otto - local validation - if there are less than 20 aids in each session then we can borrow aids from next_AIDs which is learnt from training - get all aid_ys for each aid of a test session - find 40 the most common aid_ys - if they are not already exist in the test session, then add them into the list of aids of the test session - then take the first 20 from the new list of aids of the test session (see src below)","metadata":{}},{"cell_type":"code","source":"%%time\ndef f():\n    labels = []\n\n    for AIDs in test_session_AIDs: # loop each session to get its list of aids\n        AIDs = list(dict.fromkeys(AIDs[::-1])) # reverse the order of the list of aids and remove duplicated aids and return a list\n        if len(AIDs) >= 20: # if there are more than 20 aids in the session, then takes the first 20 aids into labels (a list)\n            labels.append(AIDs[:20])\n        else:\n            counter = Counter()\n#             idx = 0\n            for AID in AIDs: # loop every aid of the list of aid from the session\n                subsequent_AID_counter = next_AIDs.get(AID) # use next_AIDs to access all the pair partners of AID, which is a counter\n                # if the counter exist, merge subsequent_AID_counter to counter (counter is still a single counter with a lot more items each time)\n                if subsequent_AID_counter:\n#                     idx+=1\n                    counter += subsequent_AID_counter\n#                     print(f\"subsequent_AID_counter: {subsequent_AID_counter}\")\n#                     print(f\"counter: {counter}\")\n#                     print()\n#                     if idx>1:\n#                         return\n\n            # now, counter is a Counter contains a Counter of other aids for every aid in a session (which means a lot of other aids counts)\n            # take the 40 most common other aids, if they are not already in AIDs, then add them to AIDs\n#             print(counter.most_common(40))\n#             help(counter.most_common)\n#             return\n            AIDs += [AID for AID, cnt in counter.most_common(40) if AID not in AIDs]\n            # in the end only take the first 20 aids into labels for each session\n            labels.append(AIDs[:20])\n    \n    return labels\n\nlabels = f()","metadata":{"execution":{"iopub.status.busy":"2022-11-19T14:57:34.811087Z","iopub.execute_input":"2022-11-19T14:57:34.811626Z","iopub.status.idle":"2022-11-19T14:58:19.346444Z","shell.execute_reply.started":"2022-11-19T14:57:34.811581Z","shell.execute_reply":"2022-11-19T14:58:19.345208Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from matplotlib import pyplot as plt\n\nplt.hist([len(l) for l in labels]);","metadata":{"execution":{"iopub.status.busy":"2022-11-19T15:07:45.486565Z","iopub.execute_input":"2022-11-19T15:07:45.487093Z","iopub.status.idle":"2022-11-19T15:07:52.347266Z","shell.execute_reply.started":"2022-11-19T15:07:45.487055Z","shell.execute_reply":"2022-11-19T15:07:52.346028Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Prepare the submission format","metadata":{}},{"cell_type":"markdown","source":"### rd: recsys - otto - local validation - make the list of aids into a string - labels_as_strings = [' '.join([str(l) for l in lls]) for lls in labels] - make a df from a dict of lists - predictions = pd.DataFrame(data={'session_type': test_session_AIDs.index, 'labels': labels_as_strings})","metadata":{}},{"cell_type":"code","source":"labels_as_strings = [' '.join([str(l) for l in lls]) for lls in labels]\npredictions = pd.DataFrame(data={'session_type': test_session_AIDs.index, 'labels': labels_as_strings})\nlabels_as_strings[:2]\npredictions.head()","metadata":{"execution":{"iopub.status.busy":"2022-11-19T15:08:58.510449Z","iopub.execute_input":"2022-11-19T15:08:58.510926Z","iopub.status.idle":"2022-11-19T15:09:04.542671Z","shell.execute_reply.started":"2022-11-19T15:08:58.510889Z","shell.execute_reply":"2022-11-19T15:09:04.541565Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### rd: recsys - ottp - local validation - make predictions/labels for clicks, carts and orders (no difference) - and prepare and create the submission dataframe","metadata":{}},{"cell_type":"code","source":"prediction_dfs = []\nsession_types = ['clicks', 'carts', 'orders']\n\nfor st in session_types:\n    modified_predictions = predictions.copy()\n    modified_predictions.session_type = modified_predictions.session_type.astype('str') + f'_{st}'\n    prediction_dfs.append(modified_predictions)","metadata":{"execution":{"iopub.status.busy":"2022-11-19T15:09:43.419777Z","iopub.execute_input":"2022-11-19T15:09:43.420273Z","iopub.status.idle":"2022-11-19T15:09:47.630082Z","shell.execute_reply.started":"2022-11-19T15:09:43.420235Z","shell.execute_reply":"2022-11-19T15:09:47.628836Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"prediction_dfs[0].head()","metadata":{"execution":{"iopub.status.busy":"2022-11-19T15:09:47.632087Z","iopub.execute_input":"2022-11-19T15:09:47.632516Z","iopub.status.idle":"2022-11-19T15:09:47.644736Z","shell.execute_reply.started":"2022-11-19T15:09:47.632479Z","shell.execute_reply":"2022-11-19T15:09:47.643224Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"prediction_dfs[1].head()","metadata":{"execution":{"iopub.status.busy":"2022-11-19T15:12:16.571040Z","iopub.execute_input":"2022-11-19T15:12:16.571534Z","iopub.status.idle":"2022-11-19T15:12:16.583572Z","shell.execute_reply.started":"2022-11-19T15:12:16.571492Z","shell.execute_reply":"2022-11-19T15:12:16.582419Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"prediction_dfs[2].head()","metadata":{"execution":{"iopub.status.busy":"2022-11-19T15:12:16.934261Z","iopub.execute_input":"2022-11-19T15:12:16.935023Z","iopub.status.idle":"2022-11-19T15:12:16.946238Z","shell.execute_reply.started":"2022-11-19T15:12:16.934983Z","shell.execute_reply":"2022-11-19T15:12:16.944930Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = pd.concat(prediction_dfs).reset_index(drop=True)\nsubmission.head()","metadata":{"execution":{"iopub.status.busy":"2022-11-19T15:12:17.308155Z","iopub.execute_input":"2022-11-19T15:12:17.308949Z","iopub.status.idle":"2022-11-19T15:12:18.814599Z","shell.execute_reply.started":"2022-11-19T15:12:17.308909Z","shell.execute_reply":"2022-11-19T15:12:18.813422Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# We either submit to Kaggle or run validation locally\n\nWe need to now reverse the processing we applied to our predictions to shape them into a submission.\n\nI am undoing this work here on purpose. I will replace the code I use for predictions down the road, so I want my evaluation framework to work with data formatted like for making a submission.","metadata":{}},{"cell_type":"markdown","source":"### rd: load id2type dict and type2id list from pickle file","metadata":{}},{"cell_type":"code","source":"with open('../input/otto-full-optimized-memory-footprint/id2type.pkl', \"rb\") as fh:\n    id2type = pickle.load(fh)\nwith open('../input/otto-full-optimized-memory-footprint/type2id.pkl', \"rb\") as fh:\n    type2id = pickle.load(fh)\n    \nsample_sub = pd.read_csv('../input/otto-recommender-system/sample_submission.csv')","metadata":{"execution":{"iopub.status.busy":"2022-11-19T15:12:41.095044Z","iopub.execute_input":"2022-11-19T15:12:41.095923Z","iopub.status.idle":"2022-11-19T15:12:49.199446Z","shell.execute_reply.started":"2022-11-19T15:12:41.095878Z","shell.execute_reply":"2022-11-19T15:12:49.198164Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_sub.head()","metadata":{"execution":{"iopub.status.busy":"2022-11-19T15:12:52.027811Z","iopub.execute_input":"2022-11-19T15:12:52.028825Z","iopub.status.idle":"2022-11-19T15:12:52.048283Z","shell.execute_reply.started":"2022-11-19T15:12:52.028773Z","shell.execute_reply":"2022-11-19T15:12:52.047139Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### rd: validation set and test set must be in the same session so that we can use test set to make predictions and validaiton set can provide ground truth to compare against","metadata":{}},{"cell_type":"markdown","source":"```python\n%%time\nif DO_LOCAL_VALIDATION:\n    # convert back for experiment\n    submission['session'] = submission.session_type.apply(lambda x: int(x.split('_')[0]))\n    submission['type'] = submission.session_type.apply(lambda x: x.split('_')[1])\n    submission.labels = submission.labels.apply(lambda x: [int(i) for i in x.split(' ')])\n\n    # convert type from idx to string-type for validation set\n    valid.type = valid.type.map(lambda idx: id2type[idx])\n    # group by session, then group by type, to access all aids, and make all aids into a list\n    ground_truth = valid.groupby(['session', 'type'])['aid'].apply(list)\n    # remove index and rename column aid to labels\n    ground_truth = ground_truth.reset_index().rename(columns={'aid': 'labels'})\n    # \n    ground_truth.loc[ground_truth.type == 'clicks', 'labels'] = ground_truth.loc[ground_truth.type == 'clicks', 'labels'].str[-1:]\n\n    submission_with_gt = submission.merge(ground_truth[['session', 'type', 'labels']], how='left', on=['session', 'type'])\n    submission_with_gt = submission_with_gt[~submission_with_gt.labels_y.isna()]\n    submission_with_gt['hits'] = submission_with_gt.apply(lambda df: len(set(df.labels_x).intersection(set(df.labels_y))), axis=1)\n    submission_with_gt['gt_count'] = submission_with_gt.labels_y.str.len()\n\n    recall_per_type = submission_with_gt.groupby(['type'])['hits'].sum() / submission_with_gt.groupby(['type'])['gt_count'].sum() \n    local_validation_score = (recall_per_type * pd.Series({'clicks': 0.10, 'carts': 0.30, 'orders': 0.60})).sum()\n    print(f'Local validation score: {local_validation_score}')\n\nelse:\n    submission.to_csv('submission.csv', index=False)\n```","metadata":{"execution":{"iopub.status.busy":"2022-11-19T15:22:12.912394Z","iopub.execute_input":"2022-11-19T15:22:12.912919Z","iopub.status.idle":"2022-11-19T15:22:12.931121Z","shell.execute_reply.started":"2022-11-19T15:22:12.912872Z","shell.execute_reply":"2022-11-19T15:22:12.929532Z"}}},{"cell_type":"markdown","source":"## Experiment to figure what some lines do","metadata":{}},{"cell_type":"code","source":"submission = pd.concat(prediction_dfs).reset_index(drop=True)\nsubmission.head()","metadata":{"execution":{"iopub.status.busy":"2022-11-19T15:22:38.146200Z","iopub.execute_input":"2022-11-19T15:22:38.146673Z","iopub.status.idle":"2022-11-19T15:22:39.640727Z","shell.execute_reply.started":"2022-11-19T15:22:38.146635Z","shell.execute_reply":"2022-11-19T15:22:39.639372Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\ndef f():\n    if DO_LOCAL_VALIDATION:\n        # convert back for experiment\n        submission['session'] = submission.session_type.apply(lambda x: int(x.split('_')[0]))\n        submission['type'] = submission.session_type.apply(lambda x: x.split('_')[1])\n        submission.labels = submission.labels.apply(lambda x: [int(i) for i in x.split(' ')])\n\n        # convert type from idx to string-type for validation set\n        valid.type = valid.type.map(lambda idx: id2type[idx])\n        # group by session, then group by type, to access all aids, and make all aids into a list\n        ground_truth = valid.groupby(['session', 'type'])['aid'].apply(list)\n#         print(ground_truth)\n#         return\n        # remove index and rename column aid to labels (no more aid as column)\n        ground_truth = ground_truth.reset_index().rename(columns={'aid': 'labels'})\n#         print(ground_truth)\n#         return\n        # debug to figure out what does .str[-1:] do: to get the last aid of the labels as the only label for each row if the type is clicks\n        ground_truth.loc[ground_truth.type == 'clicks', 'labels'] = ground_truth.loc[ground_truth.type == 'clicks', 'labels'].str[-1:]\n#         print(ground_truth)\n#         return        \n        # how to merge submission with ground_truth on session and type\n        submission_with_gt = submission.merge(ground_truth[['session', 'type', 'labels']], how='left', on=['session', 'type'])\n#         print(submission_with_gt)\n#         return         \n        # make sure rows with ground_truth.label as na is not selected\n        submission_with_gt = submission_with_gt[~submission_with_gt.labels_y.isna()]\n        # create a column for submission_with_gt to check which rows has labels_x and labels_y have aid intersections (true or false)\n        submission_with_gt['hits'] = submission_with_gt.apply(lambda df: len(set(df.labels_x).intersection(set(df.labels_y))), axis=1)\n        # add a column to count the ground truth or the number of aids of labels_y (turn labels_y into a string, \n        # len() of it can tell us how many aids are there)\n        submission_with_gt['gt_count'] = submission_with_gt.labels_y.str.len()\n#         print(submission_with_gt)\n#         return         \n\n        # calc recall for each type\n        recall_per_type = submission_with_gt.groupby(['type'])['hits'].sum() / submission_with_gt.groupby(['type'])['gt_count'].sum() \n        # calc validation score with recall and then print it\n        local_validation_score = (recall_per_type * pd.Series({'clicks': 0.10, 'carts': 0.30, 'orders': 0.60})).sum()\n        print(f'Local validation score: {local_validation_score}')\n\n    else:\n        submission.to_csv('submission.csv', index=False)\n\nsubmission = pd.concat(prediction_dfs).reset_index(drop=True)\nvalid = pd.read_parquet('_valid.parquet')\nf()","metadata":{"execution":{"iopub.status.busy":"2022-11-19T15:22:39.642930Z","iopub.execute_input":"2022-11-19T15:22:39.643318Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}