{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.7.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":38760,"databundleVersionId":4493939,"sourceType":"competition"},{"sourceId":4436180,"sourceType":"datasetVersion","datasetId":2597726}],"dockerImageVersionId":30302,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Candidate ReRank Model using Handcrafted Rules\nIn this notebook, we present a \"candidate rerank\" model using handcrafted rules. We can improve this model by engineering features, merging them unto items and users, and training a reranker model (such as XGB) to choose our final 20. Furthermore to tune and improve this notebook, we should build a local CV scheme to experiment new logic and/or models.\n\nUPDATE: I published a notebook to compute validation score [here][10] using Radek's scheme described [here][11].\n\nNote in this competition, a \"session\" actually means a unique \"user\". So our task is to predict what each of the `1,671,803` test \"users\" (i.e. \"sessions\") will do in the future. For each test \"user\" (i.e. \"session\") we must predict what they will `click`, `cart`, and `order` during the remainder of the week long test period.\n\n### Step 1 - Generate Candidates\nFor each test user, we generate possible choices, i.e. candidates. In this notebook, we generate candidates from 5 sources:\n* User history of clicks, carts, orders\n* Most popular 20 clicks, carts, orders during test week\n* Co-visitation matrix of click/cart/order to cart/order with type weighting\n* Co-visitation matrix of cart/order to cart/order called buy2buy\n* Co-visitation matrix of click/cart/order to clicks with time weighting\n\n### Step 2 - ReRank and Choose 20\nGiven the list of candidates, we must select 20 to be our predictions. In this notebook, we do this with a set of handcrafted rules. We can improve our predictions by training an XGBoost model to select for us. Our handcrafted rules give priority to:\n* Most recent previously visited items\n* Items previously visited multiple times\n* Items previously in cart or order\n* Co-visitation matrix of cart/order to cart/order\n* Current popular items\n\n![](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Nov-2022/c_r_model.png)\n  \n# Credits\nWe thank many Kagglers who have shared ideas. We use co-visitation matrix idea from Vladimir [here][1]. We use groupby sort logic from Sinan in comment section [here][4]. We use duplicate prediction removal logic from Radek [here][5]. We use multiple visit logic from Pietro [here][2]. We use type weighting logic from Ingvaras [here][3]. We use leaky test data from my previous notebook [here][4]. And some ideas may have originated from Tawara [here][6] and KJ [here][7]. We use Colum2131's parquets [here][8]. Above image is from Ravi's discussion about candidate rerank models [here][9]\n\n[1]: https://www.kaggle.com/code/vslaykovsky/co-visitation-matrix\n[2]: https://www.kaggle.com/code/pietromaldini1/multiple-clicks-vs-latest-items\n[3]: https://www.kaggle.com/code/ingvarasgalinskas/item-type-vs-multiple-clicks-vs-latest-items\n[4]: https://www.kaggle.com/code/cdeotte/test-data-leak-lb-boost\n[5]: https://www.kaggle.com/code/radek1/co-visitation-matrix-simplified-imprvd-logic\n[6]: https://www.kaggle.com/code/ttahara/otto-mors-aid-frequency-baseline\n[7]: https://www.kaggle.com/code/whitelily/co-occurrence-baseline\n[8]: https://www.kaggle.com/datasets/columbia2131/otto-chunk-data-inparquet-format\n[9]: https://www.kaggle.com/competitions/otto-recommender-system/discussion/364721\n[10]: https://www.kaggle.com/cdeotte/compute-validation-score-cv-564\n[11]: https://www.kaggle.com/competitions/otto-recommender-system/discussion/364991","metadata":{"papermill":{"duration":0.005198,"end_time":"2022-11-10T16:03:20.966987","exception":false,"start_time":"2022-11-10T16:03:20.961789","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Notes\nBelow are notes about versions:\n* **Version 1 LB 0.573** Uses popular ideas from public notebooks and adds additional co-visitation matrices and additional logic. Has CV `0.563`. See validation notebook version 2 [here][1].\n* **Version 2 LB 573** Refactor logic for `suggest_buys(df)` to make it clear how new co-visitation matrices are reranking the candidates by adding to candidate weights. Also new logic boosts CV by `+0.0003`. Also LB is slightly better too. See validation notebook version 3 [here][1]\n* **Version 3** is the same as version 2 but 1.5x faster co-visitation matrix computation!\n* **Version 4 LB 575** Use top20 for clicks and top15 for carts and buys (instead of top40 and top40). This boosts CV `+0.0015` hooray! New CV is `0.5647`. See validation version 5 [here][1]\n* **Version 5** is the same as version 4 but 2x faster co-visitation matrix computation! (and 3x faster than version 1)\n* **Version 6** Stay tuned for more versions...\n\n[1]: https://www.kaggle.com/code/cdeotte/compute-validation-score-cv-564","metadata":{}},{"cell_type":"markdown","source":"# Step 1 - Candidate Generation with RAPIDS\nFor candidate generation, we build three co-visitation matrices. One computes the popularity of cart/order given a user's previous click/cart/order. We apply type weighting to this matrix. One computes the popularity of cart/order given a user's previous cart/order. We call this \"buy2buy\" matrix. One computes the popularity of clicks given a user previously click/cart/order.  We apply time weighting to this matrix. We will use RAPIDS cuDF GPU to compute these matrices quickly!","metadata":{"papermill":{"duration":0.00373,"end_time":"2022-11-10T16:03:20.9748","exception":false,"start_time":"2022-11-10T16:03:20.97107","status":"completed"},"tags":[]}},{"cell_type":"code","source":"VER = 5\n\nimport pandas as pd, numpy as np\nfrom tqdm.notebook import tqdm\nimport os, sys, pickle, glob, gc\nfrom collections import Counter\nimport cudf, itertools\nprint('We will use RAPIDS version',cudf.__version__)","metadata":{"papermill":{"duration":3.036143,"end_time":"2022-11-10T16:03:24.014816","exception":false,"start_time":"2022-11-10T16:03:20.978673","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2025-12-16T18:26:53.147629Z","iopub.execute_input":"2025-12-16T18:26:53.147862Z","iopub.status.idle":"2025-12-16T18:26:55.562874Z","shell.execute_reply.started":"2025-12-16T18:26:53.147833Z","shell.execute_reply":"2025-12-16T18:26:55.561900Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Compute Three Co-visitation Matrices with RAPIDS\nWe will compute 3 co-visitation matrices using RAPIDS cuDF on GPU. This is 30x faster than using Pandas CPU like other public notebooks! For maximum speed, set the variable `DISK_PIECES` to the smallest number possible based on the GPU you are using without incurring memory errors. If you run this code offline with 32GB GPU ram, then you can use `DISK_PIECES = 1` and compute each co-visitation matrix in almost 1 minute! Kaggle's GPU only has 16GB ram, so we use `DISK_PIECES = 4` and it takes an amazing 3 minutes each! Below are some of the tricks to speed up computation\n* Use RAPIDS cuDF GPU instead of Pandas CPU\n* Read disk once and save in CPU RAM for later GPU multiple use\n* Process largest amount of data possible on GPU at one time\n* Merge data in two stages. Multiple small to single medium. Multiple medium to single large.\n* Write result as parquet instead of dictionary","metadata":{"papermill":{"duration":0.00424,"end_time":"2022-11-10T16:03:24.023816","exception":false,"start_time":"2022-11-10T16:03:24.019576","status":"completed"},"tags":[]}},{"cell_type":"code","source":"%%time\n# CACHE FUNCTIONS\ndef read_file(f):\n    return cudf.DataFrame( data_cache[f] )\ndef read_file_to_cache(f):\n    df = pd.read_parquet(f)\n    df.ts = (df.ts/1000).astype('int32')\n    df['type'] = df['type'].map(type_labels).astype('int8')\n    return df\n\n# CACHE THE DATA ON CPU BEFORE PROCESSING ON GPU\ndata_cache = {}\ntype_labels = {'clicks':0, 'carts':1, 'orders':2}\nfiles = glob.glob('../input/otto-chunk-data-inparquet-format/*_parquet/*')\nfor f in files: data_cache[f] = read_file_to_cache(f)\n\n# CHUNK PARAMETERS\nREAD_CT = 5\nCHUNK = int( np.ceil( len(files)/6 ))\nprint(f'We will process {len(files)} files, in groups of {READ_CT} and chunks of {CHUNK}.')","metadata":{"papermill":{"duration":0.063943,"end_time":"2022-11-10T16:03:24.091816","exception":false,"start_time":"2022-11-10T16:03:24.027873","status":"completed"},"tags":[],"_kg_hide-input":true,"execution":{"iopub.status.busy":"2025-12-16T18:26:55.564387Z","iopub.execute_input":"2025-12-16T18:26:55.564671Z","iopub.status.idle":"2025-12-16T18:27:54.154723Z","shell.execute_reply.started":"2025-12-16T18:26:55.564646Z","shell.execute_reply":"2025-12-16T18:27:54.153707Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 1) \"Carts Orders\" Co-visitation Matrix - Type Weighted","metadata":{"papermill":{"duration":0.004089,"end_time":"2022-11-10T16:03:24.100502","exception":false,"start_time":"2022-11-10T16:03:24.096413","status":"completed"},"tags":[]}},{"cell_type":"code","source":"%%time\ntype_weight = {0:1, 1:6, 2:3}\n\n# USE SMALLEST DISK_PIECES POSSIBLE WITHOUT MEMORY ERROR\nDISK_PIECES = 4\nSIZE = 1.86e6/DISK_PIECES\n\n# COMPUTE IN PARTS FOR MEMORY MANGEMENT\nfor PART in range(DISK_PIECES):\n    print()\n    print('### DISK PART',PART+1)\n    \n    # MERGE IS FASTEST PROCESSING CHUNKS WITHIN CHUNKS\n    # => OUTER CHUNKS\n    for j in range(6):\n        a = j*CHUNK\n        b = min( (j+1)*CHUNK, len(files) )\n        print(f'Processing files {a} thru {b-1} in groups of {READ_CT}...')\n        \n        # => INNER CHUNKS\n        for k in range(a,b,READ_CT):\n            # READ FILE\n            df = [read_file(files[k])]\n            for i in range(1,READ_CT): \n                if k+i<b: df.append( read_file(files[k+i]) )\n            df = cudf.concat(df,ignore_index=True,axis=0)\n            df = df.sort_values(['session','ts'],ascending=[True,False])\n            # USE TAIL OF SESSION\n            df = df.reset_index(drop=True)\n            df['n'] = df.groupby('session').cumcount()\n            df = df.loc[df.n<30].drop('n',axis=1)\n            # CREATE PAIRS\n            df = df.merge(df,on='session')\n            df = df.loc[ ((df.ts_x - df.ts_y).abs()< 24 * 60 * 60) & (df.aid_x != df.aid_y) ]\n            # MEMORY MANAGEMENT COMPUTE IN PARTS\n            df = df.loc[(df.aid_x >= PART*SIZE)&(df.aid_x < (PART+1)*SIZE)]\n            # ASSIGN WEIGHTS\n            df = df[['session', 'aid_x', 'aid_y','type_y']].drop_duplicates(['session', 'aid_x', 'aid_y'])\n            df['wgt'] = df.type_y.map(type_weight)\n            df = df[['aid_x','aid_y','wgt']]\n            df.wgt = df.wgt.astype('float32')\n            df = df.groupby(['aid_x','aid_y']).wgt.sum()\n            # COMBINE INNER CHUNKS\n            if k==a: tmp2 = df\n            else: tmp2 = tmp2.add(df, fill_value=0)\n            print(k,', ',end='')\n        print()\n        # COMBINE OUTER CHUNKS\n        if a==0: tmp = tmp2\n        else: tmp = tmp.add(tmp2, fill_value=0)\n        del tmp2, df\n        gc.collect()\n    # CONVERT MATRIX TO DICTIONARY\n    tmp = tmp.reset_index()\n    tmp = tmp.sort_values(['aid_x','wgt'],ascending=[True,False])\n    # SAVE TOP 40\n    tmp = tmp.reset_index(drop=True)\n    tmp['n'] = tmp.groupby('aid_x').aid_y.cumcount()\n    tmp = tmp.loc[tmp.n<15].drop('n',axis=1)\n    # SAVE PART TO DISK (convert to pandas first uses less memory)\n    tmp.to_pandas().to_parquet(f'top_15_carts_orders_v{VER}_{PART}.pqt')","metadata":{"papermill":{"duration":566.561189,"end_time":"2022-11-10T16:12:50.666123","exception":false,"start_time":"2022-11-10T16:03:24.104934","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2025-12-16T18:27:54.157107Z","iopub.execute_input":"2025-12-16T18:27:54.157835Z","iopub.status.idle":"2025-12-16T18:31:08.642816Z","shell.execute_reply.started":"2025-12-16T18:27:54.157804Z","shell.execute_reply":"2025-12-16T18:31:08.641820Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 2) \"Buy2Buy\" Co-visitation Matrix","metadata":{"papermill":{"duration":0.03219,"end_time":"2022-11-10T16:12:50.730634","exception":false,"start_time":"2022-11-10T16:12:50.698444","status":"completed"},"tags":[]}},{"cell_type":"code","source":"%%time\n# USE SMALLEST DISK_PIECES POSSIBLE WITHOUT MEMORY ERROR\nDISK_PIECES = 1\nSIZE = 1.86e6/DISK_PIECES\n\n# COMPUTE IN PARTS FOR MEMORY MANGEMENT\nfor PART in range(DISK_PIECES):\n    print()\n    print('### DISK PART',PART+1)\n    \n    # MERGE IS FASTEST PROCESSING CHUNKS WITHIN CHUNKS\n    # => OUTER CHUNKS\n    for j in range(6):\n        a = j*CHUNK\n        b = min( (j+1)*CHUNK, len(files) )\n        print(f'Processing files {a} thru {b-1} in groups of {READ_CT}...')\n        \n        # => INNER CHUNKS\n        for k in range(a,b,READ_CT):\n            # READ FILE\n            df = [read_file(files[k])]\n            for i in range(1,READ_CT): \n                if k+i<b: df.append( read_file(files[k+i]) )\n            df = cudf.concat(df,ignore_index=True,axis=0)\n            df = df.loc[df['type'].isin([1,2])] # ONLY WANT CARTS AND ORDERS\n            df = df.sort_values(['session','ts'],ascending=[True,False])\n            # USE TAIL OF SESSION\n            df = df.reset_index(drop=True)\n            df['n'] = df.groupby('session').cumcount()\n            df = df.loc[df.n<30].drop('n',axis=1)\n            # CREATE PAIRS\n            df = df.merge(df,on='session')\n            df = df.loc[ ((df.ts_x - df.ts_y).abs()< 14 * 24 * 60 * 60) & (df.aid_x != df.aid_y) ] # 14 DAYS\n            # MEMORY MANAGEMENT COMPUTE IN PARTS\n            df = df.loc[(df.aid_x >= PART*SIZE)&(df.aid_x < (PART+1)*SIZE)]\n            # ASSIGN WEIGHTS\n            df = df[['session', 'aid_x', 'aid_y','type_y']].drop_duplicates(['session', 'aid_x', 'aid_y'])\n            df['wgt'] = 1\n            df = df[['aid_x','aid_y','wgt']]\n            df.wgt = df.wgt.astype('float32')\n            df = df.groupby(['aid_x','aid_y']).wgt.sum()\n            # COMBINE INNER CHUNKS\n            if k==a: tmp2 = df\n            else: tmp2 = tmp2.add(df, fill_value=0)\n            print(k,', ',end='')\n        print()\n        # COMBINE OUTER CHUNKS\n        if a==0: tmp = tmp2\n        else: tmp = tmp.add(tmp2, fill_value=0)\n        del tmp2, df\n        gc.collect()\n    # CONVERT MATRIX TO DICTIONARY\n    tmp = tmp.reset_index()\n    tmp = tmp.sort_values(['aid_x','wgt'],ascending=[True,False])\n    # SAVE TOP 40\n    tmp = tmp.reset_index(drop=True)\n    tmp['n'] = tmp.groupby('aid_x').aid_y.cumcount()\n    tmp = tmp.loc[tmp.n<15].drop('n',axis=1)\n    # SAVE PART TO DISK (convert to pandas first uses less memory)\n    tmp.to_pandas().to_parquet(f'top_15_buy2buy_v{VER}_{PART}.pqt')","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"papermill":{"duration":113.735315,"end_time":"2022-11-10T16:14:44.498182","exception":false,"start_time":"2022-11-10T16:12:50.762867","status":"completed"},"tags":[],"trusted":true,"execution":{"iopub.status.busy":"2025-12-16T18:31:08.644127Z","iopub.execute_input":"2025-12-16T18:31:08.644515Z","iopub.status.idle":"2025-12-16T18:31:38.683309Z","shell.execute_reply.started":"2025-12-16T18:31:08.644457Z","shell.execute_reply":"2025-12-16T18:31:38.682369Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 3) \"Clicks\" Co-visitation Matrix - Time Weighted","metadata":{"papermill":{"duration":0.04526,"end_time":"2022-11-10T16:14:44.58589","exception":false,"start_time":"2022-11-10T16:14:44.54063","status":"completed"},"tags":[]}},{"cell_type":"code","source":"%%time\n# USE SMALLEST DISK_PIECES POSSIBLE WITHOUT MEMORY ERROR\nDISK_PIECES = 4\nSIZE = 1.86e6/DISK_PIECES\n\n# COMPUTE IN PARTS FOR MEMORY MANGEMENT\nfor PART in range(DISK_PIECES):\n    print()\n    print('### DISK PART',PART+1)\n    \n    # MERGE IS FASTEST PROCESSING CHUNKS WITHIN CHUNKS\n    # => OUTER CHUNKS\n    for j in range(6):\n        a = j*CHUNK\n        b = min( (j+1)*CHUNK, len(files) )\n        print(f'Processing files {a} thru {b-1} in groups of {READ_CT}...')\n        \n        # => INNER CHUNKS\n        for k in range(a,b,READ_CT):\n            # READ FILE\n            df = [read_file(files[k])]\n            for i in range(1,READ_CT): \n                if k+i<b: df.append( read_file(files[k+i]) )\n            df = cudf.concat(df,ignore_index=True,axis=0)\n            df = df.sort_values(['session','ts'],ascending=[True,False])\n            # USE TAIL OF SESSION\n            df = df.reset_index(drop=True)\n            df['n'] = df.groupby('session').cumcount()\n            df = df.loc[df.n<30].drop('n',axis=1)\n            # CREATE PAIRS\n            df = df.merge(df,on='session')\n            df = df.loc[ ((df.ts_x - df.ts_y).abs()< 24 * 60 * 60) & (df.aid_x != df.aid_y) ]\n            # MEMORY MANAGEMENT COMPUTE IN PARTS\n            df = df.loc[(df.aid_x >= PART*SIZE)&(df.aid_x < (PART+1)*SIZE)]\n            # ASSIGN WEIGHTS\n            df = df[['session', 'aid_x', 'aid_y','ts_x']].drop_duplicates(['session', 'aid_x', 'aid_y'])\n            df['wgt'] = 1 + 3*(df.ts_x - 1659304800)/(1662328791-1659304800)\n            df = df[['aid_x','aid_y','wgt']]\n            df.wgt = df.wgt.astype('float32')\n            df = df.groupby(['aid_x','aid_y']).wgt.sum()\n            # COMBINE INNER CHUNKS\n            if k==a: tmp2 = df\n            else: tmp2 = tmp2.add(df, fill_value=0)\n            print(k,', ',end='')\n        print()\n        # COMBINE OUTER CHUNKS\n        if a==0: tmp = tmp2\n        else: tmp = tmp.add(tmp2, fill_value=0)\n        del tmp2, df\n        gc.collect()\n    # CONVERT MATRIX TO DICTIONARY\n    tmp = tmp.reset_index()\n    tmp = tmp.sort_values(['aid_x','wgt'],ascending=[True,False])\n    # SAVE TOP 40\n    tmp = tmp.reset_index(drop=True)\n    tmp['n'] = tmp.groupby('aid_x').aid_y.cumcount()\n    tmp = tmp.loc[tmp.n<20].drop('n',axis=1)\n    # SAVE PART TO DISK (convert to pandas first uses less memory)\n    tmp.to_pandas().to_parquet(f'top_20_clicks_v{VER}_{PART}.pqt')","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"papermill":{"duration":null,"end_time":null,"exception":false,"start_time":"2022-11-10T16:14:44.629032","status":"running"},"tags":[],"trusted":true,"execution":{"iopub.status.busy":"2025-12-16T18:31:38.684819Z","iopub.execute_input":"2025-12-16T18:31:38.685180Z","iopub.status.idle":"2025-12-16T18:34:49.654256Z","shell.execute_reply.started":"2025-12-16T18:31:38.685143Z","shell.execute_reply":"2025-12-16T18:34:49.653210Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# FREE MEMORY\ndel data_cache, tmp\n_ = gc.collect()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-16T18:34:49.658167Z","iopub.execute_input":"2025-12-16T18:34:49.658761Z","iopub.status.idle":"2025-12-16T18:34:49.825535Z","shell.execute_reply.started":"2025-12-16T18:34:49.658721Z","shell.execute_reply":"2025-12-16T18:34:49.824516Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Step 2 - ReRank (choose 20) using handcrafted rules\nFor description of the handcrafted rules, read this notebook's intro.","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"def load_test():    \n    dfs = []\n    for e, chunk_file in enumerate(glob.glob('../input/otto-chunk-data-inparquet-format/test_parquet/*')):\n        chunk = pd.read_parquet(chunk_file)\n        chunk.ts = (chunk.ts/1000).astype('int32')\n        chunk['type'] = chunk['type'].map(type_labels).astype('int8')\n        dfs.append(chunk)\n    return pd.concat(dfs).reset_index(drop=True) #.astype({\"ts\": \"datetime64[ms]\"})\n\ntest_df = load_test()\nprint('Test data has shape',test_df.shape)\ntest_df.head()","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"trusted":true,"execution":{"iopub.status.busy":"2025-12-16T18:34:49.826764Z","iopub.execute_input":"2025-12-16T18:34:49.827213Z","iopub.status.idle":"2025-12-16T18:34:50.988735Z","shell.execute_reply.started":"2025-12-16T18:34:49.827185Z","shell.execute_reply":"2025-12-16T18:34:50.987785Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def load_train():    \n    dfs = []\n    for e, chunk_file in enumerate(glob.glob('../input/otto-chunk-data-inparquet-format/train_parquet/*')):\n        chunk = pd.read_parquet(chunk_file)\n        chunk.ts = (chunk.ts/1000).astype('int32')\n        chunk['type'] = chunk['type'].map(type_labels).astype('int8')\n        dfs.append(chunk)\n    return pd.concat(dfs).reset_index(drop=True) #.astype({\"ts\": \"datetime64[ms]\"})\n\ntrain_df = load_train()\nprint('Test data has shape',train_df.shape)\ntrain_df.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-16T18:34:50.990399Z","iopub.execute_input":"2025-12-16T18:34:50.990762Z","iopub.status.idle":"2025-12-16T18:35:26.823370Z","shell.execute_reply.started":"2025-12-16T18:34:50.990727Z","shell.execute_reply":"2025-12-16T18:35:26.822454Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# %%time\n# def pqt_to_dict(df):\n#     return df.groupby('aid_x').aid_y.apply(list).to_dict()\n# # LOAD THREE CO-VISITATION MATRICES\n# top_20_clicks = pqt_to_dict( pd.read_parquet(f'top_20_clicks_v{VER}_0.pqt') )\n# for k in range(1,DISK_PIECES): \n#     top_20_clicks.update( pqt_to_dict( pd.read_parquet(f'top_20_clicks_v{VER}_{k}.pqt') ) )\n# top_20_buys = pqt_to_dict( pd.read_parquet(f'top_15_carts_orders_v{VER}_0.pqt') )\n# for k in range(1,DISK_PIECES): \n#     top_20_buys.update( pqt_to_dict( pd.read_parquet(f'top_15_carts_orders_v{VER}_{k}.pqt') ) )\n# top_20_buy2buy = pqt_to_dict( pd.read_parquet(f'top_15_buy2buy_v{VER}_0.pqt') )\n\n# # TOP CLICKS AND ORDERS IN TEST\n# top_clicks = test_df.loc[test_df['type']=='clicks','aid'].value_counts().index.values[:20]\n# top_orders = test_df.loc[test_df['type']=='orders','aid'].value_counts().index.values[:20]\n\n# print('Here are size of our 3 co-visitation matrices:')\n# print( len( top_20_clicks ), len( top_20_buy2buy ), len( top_20_buys ) )","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"trusted":true,"execution":{"iopub.status.busy":"2025-12-16T16:15:20.996043Z","iopub.status.idle":"2025-12-16T16:15:20.996599Z","shell.execute_reply.started":"2025-12-16T16:15:20.996303Z","shell.execute_reply":"2025-12-16T16:15:20.996332Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport gc, glob\nfrom tqdm import tqdm\nimport xgboost as xgb","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-16T18:35:26.824566Z","iopub.execute_input":"2025-12-16T18:35:26.825827Z","iopub.status.idle":"2025-12-16T18:35:27.244100Z","shell.execute_reply.started":"2025-12-16T18:35:26.825788Z","shell.execute_reply":"2025-12-16T18:35:27.243328Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def pqt_to_dict(df):\n    return df.groupby('aid_x').aid_y.apply(list).to_dict()\n\ntop_20_clicks = {}\nfor k in range(DISK_PIECES):\n    top_20_clicks.update(\n        pqt_to_dict(pd.read_parquet(f'top_20_clicks_v{VER}_{k}.pqt'))\n    )\n\ntop_20_buys = {}\nfor k in range(DISK_PIECES):\n    top_20_buys.update(\n        pqt_to_dict(pd.read_parquet(f'top_15_carts_orders_v{VER}_{k}.pqt'))\n    )\n\ntop_20_buy2buy = pqt_to_dict(\n    pd.read_parquet(f'top_15_buy2buy_v{VER}_0.pqt')\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-16T18:41:53.933563Z","iopub.execute_input":"2025-12-16T18:41:53.933945Z","iopub.status.idle":"2025-12-16T18:43:13.920139Z","shell.execute_reply.started":"2025-12-16T18:41:53.933918Z","shell.execute_reply":"2025-12-16T18:43:13.919331Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"FEATURE_COLS = [\n    'cnt', 'recency', 'max_type',\n    'in_click_cov', 'in_buy_cov', 'in_buy2buy'\n]\n\ndef generate_candidates(hist_aids, max_candidates=100):\n    recents = list(dict.fromkeys(hist_aids[::-1]))[:20]\n    covisit = []\n\n    for a in recents:\n        covisit += top_20_clicks.get(a, [])\n        covisit += top_20_buys.get(a, [])\n        covisit += top_20_buy2buy.get(a, [])\n\n    return list(dict.fromkeys(recents + covisit))[:max_candidates]\n\n\ndef make_features(hist_df, candidates):\n    last_ts = hist_df['ts'].max()\n\n    aid_counts = hist_df['aid'].value_counts()\n    aid_last_ts = hist_df.groupby('aid')['ts'].max()\n    aid_type_max = hist_df.groupby('aid')['type'].max()\n\n    X = np.zeros((len(candidates), 6), dtype=np.float32)\n\n    for i, aid in enumerate(candidates):\n        X[i, 0] = aid_counts.get(aid, 0)\n        X[i, 1] = last_ts - aid_last_ts.get(aid, last_ts)\n        X[i, 2] = aid_type_max.get(aid, -1)\n        X[i, 3] = aid in top_20_clicks\n        X[i, 4] = aid in top_20_buys\n        X[i, 5] = aid in top_20_buy2buy\n\n    return X\n\n\ndef make_labels(candidates, future_aids):\n    return np.fromiter(\n        (aid in future_aids for aid in candidates),\n        dtype=np.int8,\n        count=len(candidates)\n    )","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-16T17:56:56.854977Z","iopub.execute_input":"2025-12-16T17:56:56.855321Z","iopub.status.idle":"2025-12-16T17:56:56.866052Z","shell.execute_reply.started":"2025-12-16T17:56:56.855290Z","shell.execute_reply":"2025-12-16T17:56:56.865133Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport glob, os, gc\nimport xgboost as xgb\n\nos.makedirs(\"train_chunks\", exist_ok=True)\n\nMAX_SESSIONS_PER_CHUNK = 250_000  # adjust so each chunk fits in memory\nchunk_id = 0\n\nX_buf, y_buf, group_buf = [], [], []\nsession_count = 0\n\ntrain_files = glob.glob('../input/otto-chunk-data-inparquet-format/train_parquet/*')\n\ndef flush_chunk(X_buf, y_buf, group_buf, chunk_id):\n    X_chunk = np.vstack(X_buf)\n    y_chunk = np.concatenate(y_buf)\n    group_chunk = np.array(group_buf, dtype=np.int32)\n    np.savez_compressed(f\"train_chunks/chunk_{chunk_id}.npz\",\n                        X=X_chunk, y=y_chunk, group=group_chunk)\n    # print(f\"Saved chunk {chunk_id}, shape {X_chunk.shape}\")\n    X_buf.clear(); y_buf.clear(); group_buf.clear()\n    del X_chunk, y_chunk, group_chunk\n    gc.collect()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-16T17:57:03.919477Z","iopub.execute_input":"2025-12-16T17:57:03.920189Z","iopub.status.idle":"2025-12-16T17:57:03.928980Z","shell.execute_reply.started":"2025-12-16T17:57:03.920153Z","shell.execute_reply":"2025-12-16T17:57:03.928077Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"NUM_FEATURES = 6","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-16T17:57:09.202159Z","iopub.execute_input":"2025-12-16T17:57:09.202501Z","iopub.status.idle":"2025-12-16T17:57:09.206753Z","shell.execute_reply.started":"2025-12-16T17:57:09.202471Z","shell.execute_reply":"2025-12-16T17:57:09.205751Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import cudf, pandas as pd, numpy as np, gc, glob, os\nfrom tqdm import tqdm\nimport xgboost as xgb\n\nVER = 5\nDISK_PIECES = 4  # adjust based on GPU memory\nMAX_CANDIDATES = 100  # max candidates per session\nFEATURE_COLS = ['cnt','recency','max_type','in_click_cov','in_buy_cov','in_buy2buy']\nNUM_FEATURES = len(FEATURE_COLS)\nMAX_SESSIONS_PER_CHUNK = 250_000  # tune to fit GPU memory\n\nos.makedirs(\"train_chunks\", exist_ok=True)\n\n# # load co-visitation matrices (already computed with GPU)\n# def pqt_to_dict(df):\n#     return df.groupby('aid_x').aid_y.apply(list).to_dict()\n\n# top_20_clicks = {}\n# top_20_buys = {}\n# top_20_buy2buy = {}\n# for k in range(DISK_PIECES):\n#     top_20_clicks.update(pqt_to_dict(pd.read_parquet(f'top_20_clicks_v{VER}_{k}.pqt')))\n#     top_20_buys.update(pqt_to_dict(pd.read_parquet(f'top_15_carts_orders_v{VER}_{k}.pqt')))\n#     top_20_buy2buy.update(pqt_to_dict(pd.read_parquet(f'top_15_buy2buy_v{VER}_{k}.pqt')))\n\n# GPU-friendly generate_candidates\ndef generate_candidates(hist_aids):\n    recents = list(dict.fromkeys(hist_aids[::-1]))[:20]\n    covisit = []\n    for a in recents:\n        covisit += top_20_clicks.get(a, [])\n        covisit += top_20_buys.get(a, [])\n        covisit += top_20_buy2buy.get(a, [])\n    candidates = list(dict.fromkeys(recents + covisit))\n    return candidates[:MAX_CANDIDATES]\n\n# GPU-friendly feature generation\ndef make_features(hist_df, candidates):\n    last_ts = hist_df['ts'].max()\n    aid_counts = hist_df['aid'].value_counts()\n    aid_last_ts = hist_df.groupby('aid')['ts'].max()\n    aid_type_max = hist_df.groupby('aid')['type'].max()\n    feats = []\n    for aid in candidates:\n        feats.append([\n            aid_counts.get(aid,0),\n            last_ts - aid_last_ts.get(aid,last_ts),\n            aid_type_max.get(aid,-1),\n            int(aid in top_20_clicks),\n            int(aid in top_20_buys),\n            int(aid in top_20_buy2buy)\n        ])\n    return np.array(feats, dtype=np.float32)\n\ndef make_labels(candidates, future_aids):\n    return np.array([1 if aid in future_aids else 0 for aid in candidates], dtype=np.int8)\n\ndef flush_chunk(X_buf, y_buf, group_buf, chunk_id):\n    np.savez_compressed(f\"train_chunks/chunk_{chunk_id}.npz\",\n                        X=X_buf, y=y_buf, group=group_buf)\n    X_buf.fill(0); y_buf.fill(0); group_buf.fill(0)\n    gc.collect()\n\n# Read train files in chunks on GPU\ntrain_files = glob.glob('../input/otto-chunk-data-inparquet-format/train_parquet/*')\n\nfor PART in range(DISK_PIECES):\n    print(f\"\\nProcessing PART {PART+1}/{DISK_PIECES}\")\n    \n    # preallocate buffers\n    X_buf = np.zeros((MAX_SESSIONS_PER_CHUNK * MAX_CANDIDATES, NUM_FEATURES), dtype=np.float32)\n    y_buf = np.zeros((MAX_SESSIONS_PER_CHUNK * MAX_CANDIDATES,), dtype=np.int8)\n    group_buf = np.zeros((MAX_SESSIONS_PER_CHUNK,), dtype=np.int32)\n    buf_ptr = 0\n    session_ptr = 0\n    chunk_id = 0\n    \n    for f in tqdm(train_files, desc=\"Files\"):\n        # load chunk to GPU\n        df = cudf.read_parquet(f)\n        df.ts = (df.ts // 1000).astype('int32')\n        df['type'] = df['type'].map({'clicks':0,'carts':1,'orders':2}).astype('int8')\n\n        # process only aid range for this PART\n        aid_min = PART * 1.86e6 / DISK_PIECES\n        aid_max = (PART+1) * 1.86e6 / DISK_PIECES\n        df = df[(df['aid'] >= aid_min) & (df['aid'] < aid_max)]\n\n        # sort per session\n        df = df.sort_values(['session','ts'], ascending=[True,False])\n        df['n'] = df.groupby('session').cumcount()\n        df = df[df.n < 30].drop('n', axis=1)\n\n        sessions_in_file = df['session'].nunique()\n        for session, df_sess in df.groupby('session', sort=False):\n            if len(df_sess) < 3:\n                continue\n\n            split = int(len(df_sess) * 0.7)\n            hist = df_sess.iloc[:split].to_pandas()\n            future = df_sess.iloc[split:].to_pandas()\n            future_aids = set(future['aid'].values)\n\n            candidates = generate_candidates(hist['aid'].values)\n            X_sess = make_features(hist, candidates)\n            y_sess = make_labels(candidates, future_aids)\n            n_cand = len(candidates)\n\n            X_buf[buf_ptr:buf_ptr+n_cand, :] = X_sess\n            y_buf[buf_ptr:buf_ptr+n_cand] = y_sess\n            group_buf[session_ptr] = n_cand\n\n            buf_ptr += n_cand\n            session_ptr += 1\n\n            if session_ptr == MAX_SESSIONS_PER_CHUNK:\n                flush_chunk(X_buf[:buf_ptr,:], y_buf[:buf_ptr], group_buf[:session_ptr], chunk_id)\n                buf_ptr = 0\n                session_ptr = 0\n                chunk_id += 1\n\n        del df\n        gc.collect()\n\n    if session_ptr > 0:\n        flush_chunk(X_buf[:buf_ptr,:], y_buf[:buf_ptr], group_buf[:session_ptr], chunk_id)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-16T18:43:13.921907Z","iopub.execute_input":"2025-12-16T18:43:13.922176Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"model = xgb.XGBRanker(\n    objective='rank:pairwise',\n    n_estimators=300,\n    max_depth=6,\n    learning_rate=0.05,\n    subsample=0.8,\n    colsample_bytree=0.8,\n    tree_method='gpu_hist',\n    predictor='gpu_predictor',\n    random_state=42\n)\n\nchunk_files = sorted(glob.glob(\"train_chunks/chunk_*.npz\"))\n\nfor f in tqdm(chunk_files):\n    data = np.load(f)\n    X_chunk = data['X']\n    y_chunk = data['y']\n    group_chunk = data['group']\n    \n    model.fit(\n        X_chunk, y_chunk, group=group_chunk,\n        xgb_model=model if chunk_files.index(f) > 0 else None,\n        verbose=True\n    )\n    \n    del X_chunk, y_chunk, group_chunk, data\n    gc.collect()","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"test_files = glob.glob('../input/otto-chunk-data-inparquet-format/test_parquet/*')\npreds = {}\n\nfor f in tqdm(test_files):\n    df = pd.read_parquet(f)\n    df.ts = (df.ts / 1000).astype('int32')\n    df['type'] = df['type'].map({'clicks':0,'carts':1,'orders':2}).astype('int8')\n\n    for session, df_sess in df.groupby('session', sort=False):\n        candidates = generate_candidates(df_sess['aid'].values)\n        X = make_features(df_sess, candidates)\n        scores = model.predict(X)\n        top_idx = np.argsort(-scores)[:20]\n        preds[session] = np.array(candidates)[top_idx]\n\n    del df\n    gc.collect()","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"rows = []\n\nfor session, aids in preds.items():\n    labels = \" \".join(map(str, aids))\n    rows.append((f\"{session}_clicks\", labels))\n    rows.append((f\"{session}_carts\", labels))\n    rows.append((f\"{session}_orders\", labels))\n\nsubmission = pd.DataFrame(rows, columns=[\"session_type\", \"labels\"])\nsubmission.to_csv(\"submission.csv\", index=False)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-16T17:09:43.192693Z","iopub.status.idle":"2025-12-16T17:09:43.192976Z","shell.execute_reply.started":"2025-12-16T17:09:43.192837Z","shell.execute_reply":"2025-12-16T17:09:43.192850Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"X_train = np.vstack(X_buf)\ny_train = np.concatenate(y_buf)\ngroup = np.array(group_buf, dtype=np.int32)\n\ndel X_buf, y_buf, group_buf\ngc.collect()\n\nmodel = xgb.XGBRanker(\n    objective='rank:pairwise',\n    n_estimators=300,\n    max_depth=6,\n    learning_rate=0.05,\n    subsample=0.8,\n    colsample_bytree=0.8,\n    tree_method='gpu_hist',\n    predictor='gpu_predictor',\n    random_state=42\n)\n\nmodel.fit(X_train, y_train, group=group)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"test_files = glob.glob('../input/otto-chunk-data-inparquet-format/test_parquet/*')\npreds = {}\n\nfor f in tqdm(test_files):\n    df = pd.read_parquet(f)\n    df.ts = (df.ts / 1000).astype('int32')\n    df['type'] = df['type'].map({'clicks':0,'carts':1,'orders':2}).astype('int8')\n\n    for session, df_sess in df.groupby('session', sort=False):\n        candidates = generate_candidates(df_sess['aid'].values)\n        X = make_features(df_sess, candidates)\n\n        scores = model.predict(X)\n        top_idx = np.argsort(-scores)[:20]\n\n        preds[session] = np.array(candidates)[top_idx]\n\n    del df\n    gc.collect()","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"rows = []\n\nfor session, aids in preds.items():\n    labels = \" \".join(map(str, aids))\n    rows.append((f\"{session}_clicks\", labels))\n    rows.append((f\"{session}_carts\", labels))\n    rows.append((f\"{session}_orders\", labels))\n\nsubmission = pd.DataFrame(rows, columns=[\"session_type\", \"labels\"])\nsubmission.to_csv(\"submission.csv\", index=False)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# import pandas as pd\n# import numpy as np\n# import glob\n# import itertools\n# import xgboost as xgb\n# from collections import Counter\n# import os\n\n# type_labels = {'clicks': 0, 'carts': 1, 'orders': 2}\n# DISK_PIECES = 4\n# VER = 1","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-16T15:55:46.140391Z","iopub.execute_input":"2025-12-16T15:55:46.141279Z","iopub.status.idle":"2025-12-16T15:55:46.147570Z","shell.execute_reply.started":"2025-12-16T15:55:46.141234Z","shell.execute_reply":"2025-12-16T15:55:46.146536Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# def generate_candidates(hist_aids):\n#     unique_aids = list(dict.fromkeys(hist_aids[::-1]))\n#     covisit_clicks = list(itertools.chain(*[top_20_clicks.get(a, []) for a in unique_aids if a in top_20_clicks]))\n#     covisit_buy = list(itertools.chain(*[top_20_buy2buy.get(a, []) for a in unique_aids if a in top_20_buy2buy]))\n#     candidates = list(dict.fromkeys(unique_aids + covisit_clicks + covisit_buy))\n#     return candidates","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-16T08:11:33.856085Z","iopub.execute_input":"2025-12-16T08:11:33.856384Z","iopub.status.idle":"2025-12-16T08:11:33.865990Z","shell.execute_reply.started":"2025-12-16T08:11:33.856358Z","shell.execute_reply":"2025-12-16T08:11:33.865118Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# np.random.seed(42)\n\n# sess_with_buy = (\n#     train_df\n#     .groupby('session')['type']\n#     .max()\n#     .loc[lambda x: x >= 1]\n#     .index\n# )\n\n# keep_sessions = np.random.choice(\n#     sess_with_buy,\n#     # size=min(300_000, len(sess_with_buy)),\n#     size=len(sess_with_buy),\n#     replace=False\n# )\n\n# train_df = train_df[train_df['session'].isin(keep_sessions)]\n# train_df = train_df.sort_values(['session','ts'])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-16T15:13:52.127117Z","iopub.execute_input":"2025-12-16T15:13:52.127518Z","iopub.status.idle":"2025-12-16T15:13:57.221331Z","shell.execute_reply.started":"2025-12-16T15:13:52.127487Z","shell.execute_reply":"2025-12-16T15:13:57.220532Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"len(train_df)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-16T15:55:46.149758Z","iopub.execute_input":"2025-12-16T15:55:46.150053Z","iopub.status.idle":"2025-12-16T15:55:46.164843Z","shell.execute_reply.started":"2025-12-16T15:55:46.150012Z","shell.execute_reply":"2025-12-16T15:55:46.163911Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def generate_candidates(hist_aids, max_candidates=100):\n    recents = list(dict.fromkeys(hist_aids[::-1]))[:20]\n\n    covisit = []\n    for a in recents:\n        covisit += top_20_clicks.get(a, [])\n        covisit += top_20_buys.get(a, [])\n        covisit += top_20_buy2buy.get(a, [])\n\n    candidates = list(dict.fromkeys(recents + covisit))\n    return candidates[:max_candidates]\n\ndef make_features(hist_df, candidates):\n    feats = []\n    last_ts = hist_df['ts'].max()\n\n    aid_counts = hist_df['aid'].value_counts()\n    aid_last_ts = hist_df.groupby('aid')['ts'].max()\n    aid_type_max = hist_df.groupby('aid')['type'].max()\n\n    for aid in candidates:\n        feats.append([\n            aid,\n            aid_counts.get(aid, 0),\n            last_ts - aid_last_ts.get(aid, last_ts),\n            aid_type_max.get(aid, -1),\n            aid in top_20_clicks,\n            aid in top_20_buys,\n            aid in top_20_buy2buy,\n        ])\n\n    return pd.DataFrame(\n        feats,\n        columns=[\n            'aid',\n            'cnt',\n            'recency',\n            'max_type',\n            'in_click_cov',\n            'in_buy_cov',\n            'in_buy2buy'\n        ]\n    )\n    \ndef make_labels(candidates, future_aids):\n    return np.array([1 if aid in future_aids else 0 for aid in candidates], dtype=np.int8)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-16T15:55:46.166081Z","iopub.execute_input":"2025-12-16T15:55:46.167405Z","iopub.status.idle":"2025-12-16T15:55:46.218968Z","shell.execute_reply.started":"2025-12-16T15:55:46.167378Z","shell.execute_reply":"2025-12-16T15:55:46.218100Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# infer feature dimension\ndummy_feats = make_features(\n    train_df.iloc[:10],\n    generate_candidates(train_df.iloc[:10]['aid'].values)\n)\nFEATURE_COLS = [c for c in dummy_feats.columns if c != 'aid']\nD = len(FEATURE_COLS)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-16T16:10:49.532396Z","iopub.execute_input":"2025-12-16T16:10:49.533293Z","iopub.status.idle":"2025-12-16T16:10:49.548723Z","shell.execute_reply.started":"2025-12-16T16:10:49.533257Z","shell.execute_reply":"2025-12-16T16:10:49.547784Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"D","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-16T16:10:54.138209Z","iopub.execute_input":"2025-12-16T16:10:54.138585Z","iopub.status.idle":"2025-12-16T16:10:54.144450Z","shell.execute_reply.started":"2025-12-16T16:10:54.138553Z","shell.execute_reply":"2025-12-16T16:10:54.143448Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"from tqdm import tqdm\nimport numpy as np\n\nX_buf, y_buf, group_buf = [], [], []\n\nMAX_SESSIONS_PER_CHUNK = 5000\n\ndef flush_chunk(X_buf, y_buf, group_buf, chunk_id):\n    X = np.vstack(X_buf).astype(np.float32)\n    y = np.concatenate(y_buf).astype(np.int8)\n    group = np.array(group_buf, dtype=np.int32)\n\n    np.savez_compressed(\n        f\"xgb_chunk_{chunk_id}.npz\",\n        X=X,\n        y=y,\n        group=group\n    )\n\n    X_buf.clear()\n    y_buf.clear()\n    group_buf.clear()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-16T16:11:46.719044Z","iopub.execute_input":"2025-12-16T16:11:46.719869Z","iopub.status.idle":"2025-12-16T16:11:46.726002Z","shell.execute_reply.started":"2025-12-16T16:11:46.719833Z","shell.execute_reply":"2025-12-16T16:11:46.725058Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"chunk_id = 0\nsession_count = 0\n\nfor session, df_sess in tqdm(\n        train_df.groupby('session', sort=False),\n        total=train_df['session'].nunique()\n    ):\n\n    if len(df_sess) < 3:\n        continue\n\n    split = int(len(df_sess) * 0.7)\n    hist = df_sess.iloc[:split]\n    future = df_sess.iloc[split:]\n    future_aids = set(future['aid'].values)\n\n    candidates = generate_candidates(hist['aid'].values)\n    feats = make_features(hist, candidates)\n\n    X_buf.append(\n        feats[FEATURE_COLS].to_numpy(dtype=np.float32)\n    )\n    y_buf.append(\n        make_labels(candidates, future_aids)\n    )\n    group_buf.append(len(candidates))\n\n    session_count += 1\n\n    if session_count % MAX_SESSIONS_PER_CHUNK == 0:\n        flush_chunk(X_buf, y_buf, group_buf, chunk_id)\n        chunk_id += 1\n\n# flush remainder\nif X_buf:\n    flush_chunk(X_buf, y_buf, group_buf, chunk_id)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-16T16:11:52.192293Z","iopub.execute_input":"2025-12-16T16:11:52.192785Z","execution_failed":"2025-12-16T16:12:13.321Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# from tqdm import tqdm\n\n# X_all, y_all, group = [], [], []\n\n# for session, df_sess in tqdm(\n#         train_df.groupby('session'),\n#         total=train_df['session'].nunique()\n#     ):\n#     if len(df_sess) < 3:\n#         continue\n\n#     split = int(len(df_sess) * 0.7)\n#     hist = df_sess.iloc[:split]\n#     future = df_sess.iloc[split:]\n#     future_aids = set(future['aid'].values)\n\n#     candidates = generate_candidates(hist['aid'].values)\n#     feats = make_features(hist, candidates)\n    \n#     # Drop 'aid' before appending to training data\n#     X_all.append(feats.drop(columns='aid', errors='ignore'))\n#     y_all.append(make_labels(candidates, future_aids))\n#     group.append(len(candidates))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-16T10:33:33.529557Z","iopub.execute_input":"2025-12-16T10:33:33.530471Z","iopub.status.idle":"2025-12-16T10:54:22.083994Z","shell.execute_reply.started":"2025-12-16T10:33:33.530438Z","shell.execute_reply":"2025-12-16T10:54:22.083025Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"X_train = pd.concat(X_all, ignore_index=True)\ny_train = np.concatenate(y_all)\ngroup = np.array(group)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-16T10:54:22.085803Z","iopub.execute_input":"2025-12-16T10:54:22.086522Z","iopub.status.idle":"2025-12-16T10:55:15.422317Z","shell.execute_reply.started":"2025-12-16T10:54:22.086483Z","shell.execute_reply":"2025-12-16T10:55:15.421383Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import xgboost as xgb\n\nmodel = xgb.XGBRanker(\n    objective='rank:pairwise',\n    n_estimators=200,\n    max_depth=6,\n    learning_rate=0.05,\n    subsample=0.8,\n    colsample_bytree=0.8,\n    tree_method='hist',\n    random_state=42\n)\n\nmodel.fit(\n    X_train,\n    y_train,\n    group=group\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-16T10:55:15.429996Z","iopub.execute_input":"2025-12-16T10:55:15.430269Z","iopub.status.idle":"2025-12-16T11:00:17.584464Z","shell.execute_reply.started":"2025-12-16T10:55:15.430246Z","shell.execute_reply":"2025-12-16T11:00:17.583544Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"from tqdm import tqdm\n\nsessions = test_df['session'].values\nunique_sessions, session_ptrs = np.unique(sessions, return_index=True)\n\npreds = {}\n\nfor i, sess in enumerate(tqdm(unique_sessions, total=len(unique_sessions))):\n    start = session_ptrs[i]\n    end = session_ptrs[i+1] if i+1 < len(session_ptrs) else len(test_df)\n\n    df_sess = test_df.iloc[start:end]\n\n    candidates = generate_candidates(df_sess['aid'].values)\n    X = make_features(df_sess, candidates)\n    aids = X['aid'].values\n    \n    # convert all features to numeric\n    X_model = X.drop(columns='aid').copy()\n    X_model = X_model.fillna(0)        # optional: handle NaNs\n    X_model = X_model.astype(float)     # convert int/bool to float\n    \n    # prediction\n    scores = model.predict(X_model)\n    top_idx = np.argsort(-scores)[:20]\n    \n    preds[sess] = aids[top_idx]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-16T11:00:17.586125Z","iopub.execute_input":"2025-12-16T11:00:17.586451Z","iopub.status.idle":"2025-12-16T14:05:27.962661Z","shell.execute_reply.started":"2025-12-16T11:00:17.586426Z","shell.execute_reply":"2025-12-16T14:05:27.959326Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Create Submission CSV\nInferring test data with Pandas groupby is slow. We need to accelerate the following code.","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"import pandas as pd\n\nrows = []\n\nfor session, aids in preds.items():\n    labels = \" \".join(map(str, aids))\n\n    rows.append((f\"{session}_clicks\", labels))\n    rows.append((f\"{session}_carts\",  labels))\n    rows.append((f\"{session}_orders\", labels))\n\nsubmission = pd.DataFrame(rows, columns=[\"session_type\", \"labels\"])\nsubmission.to_csv(\"submission.csv\", index=False)\n\nsubmission.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-16T14:09:46.101503Z","iopub.execute_input":"2025-12-16T14:09:46.101851Z","iopub.status.idle":"2025-12-16T14:10:24.383638Z","shell.execute_reply.started":"2025-12-16T14:09:46.101823Z","shell.execute_reply":"2025-12-16T14:10:24.382740Z"}},"outputs":[],"execution_count":null}]}