It was a pleasure to work with @titericz @benediktschifferer @theoviel . We each made individual models and then ensembled all our work together by adding the ranks of each of our predictions per user target type. Below I describe my individual single model. My teammates will describe their work in their own discussion posts. You can read about Theo's LB 0.6029 model here! You can read about Benny's LB 0.601 model here!
\nMost of my solution was made public in my notebook here and discussion here during the competition. There were only 3 significant ideas missing from my public work. Let's discuss how to boost my public work to LB 0.601 single model. (UPDATE: All code published to GitHub here)
\nIt was explained here and here that the best approach was \"candidate rerank\" model. My public notebook here shows how to achieve LB 0.575. And my discussion post here explains how to improve my public notebook by adding a GBT ranker model. Below are the three missing pieces labeled (1), (2), and (3) to achieve LB 0.601
\nTo build a \"candidate rerank\", we need candidates. Where do we get candidates? The easiest way is to get them from my public notebook. In the def suggest_buys(df) and def suggest_clicks(df) function, the last lines are
top_aids = [aid for aid, ct in aids_counter.most_common(20)] \nreturn top_aids\nTo generate 50 candidates, we change 20 to 50 as in
\ntop_aids = [aid for aid, ct in aids_counter.most_common(50)] \nreturn top_aids\nWe now have 50 candidates per user from our public notebook above. Next we need to make features for our reranker model. The strongest and easiest way to make features is extract our co-visit counts by changing the last two lines of my public notebook to the following:
\ntop_counts = [ct for aid, ct in aids_counter.most_common(50)] \nreturn top_counts\nWhen we merge these counts to our candidates, we now have an interaction feature for each user item pair. To make more interaction features, we can extract the counts for each co-visit matrix individually. For example, imagine that we have 3 co-visit matrices named covisit2, covisit3, and covist4. Then one by one, we extract each covisit's counts:
\nEXTRACT = ['covisit2']\naids_counter = Counter()\nif 'covisit2' in EXTRACT:\n aids = list(itertools.chain(*[covisit2[aid] for aid in unique_aids if aid in covisit2]))\n for a in aids: aids_counter[a] += 1\nif 'covisit3' in EXTRACT:\n aids = list(itertools.chain(*[covisit3[aid] for aid in unique_aids if aid in covisit3]))\n for a in aids: aids_counter[a] += 1\nif 'covisit4' in EXTRACT:\n aids = list(itertools.chain(*[covisit4[aid] for aid in unique_aids if aid in covisit4]))\n for a in aids: aids_counter[a] += 1\ntop_counts = [ct for aid, ct in aids_counter.most_common(50)] \nreturn top_counts\nFirst we use the technique above to generate candidates and the technique above to extract covisit counts. Next we add some simple item and user features like counting the number of times an item is click cart or order. When we apply the XGB reranker described here, our CV and LB will boost by +0.011. For example, the public notebook will boost to LB 0.586.
To score over LB 0.600, we create more co-visit matrices to boost the original notebook's LB score. My public notebook uses 3 covisit matrices and achieves LB 0.575. If we make 17 more covisit matrices, we can boost my public \"rules only\" notebook to LB 0.590 (new notebook published here). Then when we extract the covisit counts explained above, the XGB reranker will boost +0.011 and achieve LB 0.601
\nBelow are a description of my 20 covisit matrices. These covisit matrices are the secret sauce enabling my single XGB ranker model to achieve LB 0.601. The following variable names are from my new LB 0.590 notebook posted here. (Example code showing how to compute covisit matrices on GPU is here)
\ndf['k'] = np.arange(len(df))df = df.merge(df, on=['session'])df = df.loc[ (df.k_y - df.k_x).abs()==1 ](df.k_y - df.k_x).abs()<=2df = df.loc[df['type'].isin(['carts','orders'])]df = df.merge(df, on=['session'])df = df.loc[ (df.k_y - df.k_x > 0) & (df.k_y - df.k_x <= 3) ](df.k_y - df.k_x).abs()<=3 and have time decay withdf['wgt'] = (1/2)**( (df.ts_x - df.ts_y).abs() /60/60)(df.k_y - df.k_x).abs()<=6df = df.loc[df.ts >= LAST_3_WEEKS ]df2 = df.loc[df['type'].isin(['carts','orders'])]df = df.merge(df2, on=['session'])df = df.loc[ df.ts_y - df.ts_x > 0 ]df['wgt'] = (1/2)**( (df.ts_x - df.ts_y).abs() /60/60)df['x'] = df.groupby('session').ts.transform('min')df = df.loc[df.x > train.ts.min() + TWO_WEEKS ]df['n'] = df.groupby('session').cumcount()df2 = df.loc[df['n']==0]df3 = df.loc[df['type'].isin(['carts','orders'])]df = df2.merge(df3, on='session')df = df.loc[df[COUNT]<THRESHOLD]df = df.merge(df, on='session')df2 = df.loc[df[HOUR]<14]df = df2.merge(df, on='session')To find the above 20 covisit matrices, i computed hundreds of covisit matrices and then computed local CV score. To make covisit matrices quickly, I used RAPIDS cuDF to make each covisit matrix on GPU in under 1 minute. Code to make covisit matrix is shown here. Matrices were made using Nvidia 4xV100 32GB GPUs.
\nI published all 261 jupyter notebooks in my GitHub here. Specially we can review all the code used to generate co-visititation matices. And we can see the pipeline for building, training, inferring a GBT reranker model. Our team's final Kaggle inference submit notebook is here. My notebook to generate 100 candidates for reranker is here. By itself it scores 49th place LB 0.590!
", "messages": [ { "id": 2125843, "postDate": "2023-02-01T23:45:34.770Z", "content": "It was a pleasure to work with @titericz @benediktschifferer @theoviel . We each made individual models and then ensembled all our work together by adding the ranks of each of our predictions per user target type. Below I describe my individual single model. My teammates will describe their work in their own discussion posts. You can read about Theo's LB 0.6029 model here! You can read about Benny's LB 0.601 model here!
\nMost of my solution was made public in my notebook here and discussion here during the competition. There were only 3 significant ideas missing from my public work. Let's discuss how to boost my public work to LB 0.601 single model. (UPDATE: All code published to GitHub here)
\nIt was explained here and here that the best approach was \"candidate rerank\" model. My public notebook here shows how to achieve LB 0.575. And my discussion post here explains how to improve my public notebook by adding a GBT ranker model. Below are the three missing pieces labeled (1), (2), and (3) to achieve LB 0.601
\nTo build a \"candidate rerank\", we need candidates. Where do we get candidates? The easiest way is to get them from my public notebook. In the def suggest_buys(df) and def suggest_clicks(df) function, the last lines are
top_aids = [aid for aid, ct in aids_counter.most_common(20)] \nreturn top_aids\nTo generate 50 candidates, we change 20 to 50 as in
\ntop_aids = [aid for aid, ct in aids_counter.most_common(50)] \nreturn top_aids\nWe now have 50 candidates per user from our public notebook above. Next we need to make features for our reranker model. The strongest and easiest way to make features is extract our co-visit counts by changing the last two lines of my public notebook to the following:
\ntop_counts = [ct for aid, ct in aids_counter.most_common(50)] \nreturn top_counts\nWhen we merge these counts to our candidates, we now have an interaction feature for each user item pair. To make more interaction features, we can extract the counts for each co-visit matrix individually. For example, imagine that we have 3 co-visit matrices named covisit2, covisit3, and covist4. Then one by one, we extract each covisit's counts:
\nEXTRACT = ['covisit2']\naids_counter = Counter()\nif 'covisit2' in EXTRACT:\n aids = list(itertools.chain(*[covisit2[aid] for aid in unique_aids if aid in covisit2]))\n for a in aids: aids_counter[a] += 1\nif 'covisit3' in EXTRACT:\n aids = list(itertools.chain(*[covisit3[aid] for aid in unique_aids if aid in covisit3]))\n for a in aids: aids_counter[a] += 1\nif 'covisit4' in EXTRACT:\n aids = list(itertools.chain(*[covisit4[aid] for aid in unique_aids if aid in covisit4]))\n for a in aids: aids_counter[a] += 1\ntop_counts = [ct for aid, ct in aids_counter.most_common(50)] \nreturn top_counts\nFirst we use the technique above to generate candidates and the technique above to extract covisit counts. Next we add some simple item and user features like counting the number of times an item is click cart or order. When we apply the XGB reranker described here, our CV and LB will boost by +0.011. For example, the public notebook will boost to LB 0.586.
To score over LB 0.600, we create more co-visit matrices to boost the original notebook's LB score. My public notebook uses 3 covisit matrices and achieves LB 0.575. If we make 17 more covisit matrices, we can boost my public \"rules only\" notebook to LB 0.590 (new notebook published here). Then when we extract the covisit counts explained above, the XGB reranker will boost +0.011 and achieve LB 0.601
\nBelow are a description of my 20 covisit matrices. These covisit matrices are the secret sauce enabling my single XGB ranker model to achieve LB 0.601. The following variable names are from my new LB 0.590 notebook posted here. (Example code showing how to compute covisit matrices on GPU is here)
\ndf['k'] = np.arange(len(df))df = df.merge(df, on=['session'])df = df.loc[ (df.k_y - df.k_x).abs()==1 ](df.k_y - df.k_x).abs()<=2df = df.loc[df['type'].isin(['carts','orders'])]df = df.merge(df, on=['session'])df = df.loc[ (df.k_y - df.k_x > 0) & (df.k_y - df.k_x <= 3) ](df.k_y - df.k_x).abs()<=3 and have time decay withdf['wgt'] = (1/2)**( (df.ts_x - df.ts_y).abs() /60/60)(df.k_y - df.k_x).abs()<=6df = df.loc[df.ts >= LAST_3_WEEKS ]df2 = df.loc[df['type'].isin(['carts','orders'])]df = df.merge(df2, on=['session'])df = df.loc[ df.ts_y - df.ts_x > 0 ]df['wgt'] = (1/2)**( (df.ts_x - df.ts_y).abs() /60/60)df['x'] = df.groupby('session').ts.transform('min')df = df.loc[df.x > train.ts.min() + TWO_WEEKS ]df['n'] = df.groupby('session').cumcount()df2 = df.loc[df['n']==0]df3 = df.loc[df['type'].isin(['carts','orders'])]df = df2.merge(df3, on='session')df = df.loc[df[COUNT]<THRESHOLD]df = df.merge(df, on='session')df2 = df.loc[df[HOUR]<14]df = df2.merge(df, on='session')To find the above 20 covisit matrices, i computed hundreds of covisit matrices and then computed local CV score. To make covisit matrices quickly, I used RAPIDS cuDF to make each covisit matrix on GPU in under 1 minute. Code to make covisit matrix is shown here. Matrices were made using Nvidia 4xV100 32GB GPUs.
\nI published all 261 jupyter notebooks in my GitHub here. Specially we can review all the code used to generate co-visititation matices. And we can see the pipeline for building, training, inferring a GBT reranker model. Our team's final Kaggle inference submit notebook is here. My notebook to generate 100 candidates for reranker is here. By itself it scores 49th place LB 0.590!
", "rawMarkdown": "# Team G & B & D & T\nIt was a pleasure to work with @titericz @benediktschifferer @theoviel . We each made individual models and then ensembled all our work together by adding the ranks of each of our predictions per user target type. Below I describe my individual single model. My teammates will describe their work in their own discussion posts. You can read about Theo's LB 0.6029 model [here][5]! You can read about Benny's LB 0.601 model [here][8]!\n\nMost of my solution was made public in my notebook [here][3] and discussion [here][2] during the competition. There were only 3 significant ideas missing from my public work. Let's discuss how to boost my public work to LB 0.601 single model. (UPDATE: All code published to GitHub [here][6])\n\n# How To Score LB 0.601 Single Model\nIt was explained [here][1] and [here][2] that the best approach was \"candidate rerank\" model. My public notebook [here][3] shows how to achieve LB 0.575. And my discussion post [here][2] explains how to improve my public notebook by adding a GBT ranker model. Below are the three missing pieces labeled (1), (2), and (3) to achieve LB 0.601\n\n# (1) Choosing Candidates\nTo build a \"candidate rerank\", we need candidates. Where do we get candidates? The easiest way is to get them from my public notebook. In the `def suggest_buys(df)` and `def suggest_clicks(df)` function, the last lines are\n\n top_aids = [aid for aid, ct in aids_counter.most_common(20)] \n return top_aids\n\nTo generate 50 candidates, we change 20 to 50 as in\n\n top_aids = [aid for aid, ct in aids_counter.most_common(50)] \n return top_aids\n\n# (2) Choosing Interaction Features\nWe now have 50 candidates per user from our public notebook above. Next we need to make features for our reranker model. The strongest and easiest way to make features is extract our co-visit counts by changing the last two lines of my public notebook to the following:\n\n top_counts = [ct for aid, ct in aids_counter.most_common(50)] \n return top_counts\n\nWhen we merge these counts to our candidates, we now have an interaction feature for each user item pair. To make more interaction features, we can extract the counts for each co-visit matrix individually. For example, imagine that we have 3 co-visit matrices named covisit2, covisit3, and covist4. Then one by one, we extract each covisit's counts:\n\n EXTRACT = ['covisit2']\n aids_counter = Counter()\n if 'covisit2' in EXTRACT:\n aids = list(itertools.chain(*[covisit2[aid] for aid in unique_aids if aid in covisit2]))\n for a in aids: aids_counter[a] += 1\n if 'covisit3' in EXTRACT:\n aids = list(itertools.chain(*[covisit3[aid] for aid in unique_aids if aid in covisit3]))\n for a in aids: aids_counter[a] += 1\n if 'covisit4' in EXTRACT:\n aids = list(itertools.chain(*[covisit4[aid] for aid in unique_aids if aid in covisit4]))\n for a in aids: aids_counter[a] += 1\n top_counts = [ct for aid, ct in aids_counter.most_common(50)] \n return top_counts\n\n# Reranker Boost CV and LB +0.011\nFirst we use the technique above to generate candidates and the technique above to extract covisit counts. Next we add some simple item and user features like counting the number of times an item is click cart or order. When we apply the XGB reranker described [here][2], our CV and LB will boost by `+0.011`. For example, the public notebook will boost to LB 0.586.\n\n# Using Rules Only (without reranker) Scores LB 0.590\nTo score over LB 0.600, we create more co-visit matrices to boost the original notebook's LB score. My public notebook uses 3 covisit matrices and achieves LB 0.575. If we make 17 more covisit matrices, we can boost my public \"rules only\" notebook to LB 0.590 (new notebook published [here][4]). Then when we extract the covisit counts explained above, the XGB reranker will boost +0.011 and achieve LB 0.601\n\n# (3) Twenty Covisit Matrices\nBelow are a description of my 20 covisit matrices. These covisit matrices are the secret sauce enabling my single XGB ranker model to achieve LB 0.601. The following variable names are from my new LB 0.590 notebook posted [here][4]. (Example code showing how to compute covisit matrices on GPU is [here][3])\n* **top_20** - this covisit matrix is in my original notebook\n* **top_20b** - all covisit pair counts are consecutive items. See code below.\n `df['k'] = np.arange(len(df))`\n `df = df.merge(df, on=['session'])`\n `df = df.loc[ (df.k_y - df.k_x).abs()==1 ]`\n* **top_20c** - all covisit pair counts are `(df.k_y - df.k_x).abs()<=2`\n* **top_20d** - all covisit pairs are carts/orders and forward at most 3 consecutive\n `df = df.loc[df['type'].isin(['carts','orders'])]`\n `df = df.merge(df, on=['session'])`\n `df = df.loc[ (df.k_y - df.k_x > 0) & (df.k_y - df.k_x <= 3) ]`\n* **top_20e** - all covisit pairs are `(df.k_y - df.k_x).abs()<=3` and have time decay with\n `df['wgt'] = (1/2)**( (df.ts_x - df.ts_y).abs() /60/60)`\n* **top_20f** - same as above but `(df.k_y - df.k_x).abs()<=6`\n* **top_20_orders** - this covisit matrix is in my original notebook\n* **top_20_buy2buy** - this covisit matrix is in my original notebook\n* **top_20_buy2buy2** - use most recent 3 weeks data and only carts/orders. Apply time decay shown above.\n* **top_20_test** - use most recent 3 weeks data. Only forward in time pairs. Use clicks/carts/orders to carts/orders. Add time decay\n `df = df.loc[df.ts >= LAST_3_WEEKS ]`\n `df2 = df.loc[df['type'].isin(['carts','orders'])]`\n `df = df.merge(df2, on=['session'])`\n `df = df.loc[ df.ts_y - df.ts_x > 0 ]`\n `df['wgt'] = (1/2)**( (df.ts_x - df.ts_y).abs() /60/60)`\n* **top_20_test2** - use most recent 2 weeks data with time decay.\n* **top_20_buy** - Limit to forward 2 hours. Use clicks/carts/orders to carts/orders. Apply time decay.\n* **top_20_new** - Find cold start users in train. Pairs using only their first history item. Use clicks/carts/orders to carts/orders.\n `df['x'] = df.groupby('session').ts.transform('min')`\n `df = df.loc[df.x > train.ts.min() + TWO_WEEKS ]`\n `df['n'] = df.groupby('session').cumcount()`\n `df2 = df.loc[df['n']==0]`\n `df3 = df.loc[df['type'].isin(['carts','orders'])]`\n `df = df2.merge(df3, on='session')`\n* **top_20_new2** - Find cold start users in train. Pairs using only their first history item. Use clicks/carts/orders to clicks/carts/orders. Apply time decay.\n* **top_40_day** - Use only last week data. Forward in time. Clicks/carts/orders to carts/orders. Time decay\n* **top_40_day2** - Use only last week data. Time decay\n* **top_40_less** - Train users with less than 6 history and test users with less than 3\n `df = df.loc[df[COUNT]Everything looks so easy when reading the describtion of your solution. I've even thought for a moment \"how could it happen that I didn't try it myself?\"
\nWith just kaggle notebooks running all the possible experiments is more tricky. And anyway, I didn't even think about some of your matrices, like before/after 2PM, only users with short history, only last week. I did make a matrice for exact next aid, but only used it for clicks and didn't even try to apply it to carts/orders.
And again, thank you for you clear and insightfull posts, both during and after the competition.
", "rawMarkdown": "Everything looks so easy when reading the describtion of your solution. I've even thought for a moment \"how could it happen that I didn't try it myself?\" \nWith just kaggle notebooks running all the possible experiments is more tricky. And anyway, I didn't even think about some of your matrices, like before/after 2PM, only users with short history, only last week. I did make a matrice for exact next aid, but only used it for clicks and didn't even try to apply it to carts/orders.\n\nAnd again, thank you for you clear and insightfull posts, both during and after the competition.", "votes": 3 }, { "id": 2127417, "postDate": "2023-02-02T21:45:54.857Z", "content": "Great job @cdeotte and the rest of the team! 🥳 Amazing performance!
\nThis write up reads like a good detective story, the plot is so clearly explained! So cool to understand what you did to achieve this phenomenal result. Love the progression from the public kernel with the co-visitation matrices there to the full solution using the reranker!
\nCongrats again and thank you for this great write-up!
", "rawMarkdown": "Great job @cdeotte and the rest of the team! 🥳 Amazing performance!\n\nThis write up reads like a good detective story, the plot is so clearly explained! So cool to understand what you did to achieve this phenomenal result. Love the progression from the public kernel with the co-visitation matrices there to the full solution using the reranker!\n\nCongrats again and thank you for this great write-up! ", "votes": 4, "replies": [ { "id": 2128306, "postDate": "2023-02-03T15:59:16.260Z", "content": "Thanks Radek. Thanks for all your sharing during the competition!
", "rawMarkdown": "Thanks Radek. Thanks for all your sharing during the competition!" } ] }, { "id": 2186719, "postDate": "2023-03-18T03:42:24.003Z", "content": "thx for sharing, really learn a lot from that
", "rawMarkdown": "thx for sharing, really learn a lot from that", "votes": 1 }, { "id": 2146750, "postDate": "2023-02-16T05:38:22.827Z", "content": "Congratulations @cdeotte excellent work.
", "rawMarkdown": "Congratulations @cdeotte excellent work.", "votes": 1 }, { "id": 2141778, "postDate": "2023-02-13T05:00:00.417Z", "content": "Wow, congratulations on taking 3rd place in the competition! Your solution is impressive OAO
\nI learnt your approach to using RAPID cuDF to quickly compute hundreds of covisit matrices and then compute the local CV score is clever. I can imagine that this saved a lot of time and allowed me to experiment more efficiently in the future!
\nThank for sharing!
", "rawMarkdown": "Wow, congratulations on taking 3rd place in the competition! Your solution is impressive OAO\n\nI learnt your approach to using RAPID cuDF to quickly compute hundreds of covisit matrices and then compute the local CV score is clever. I can imagine that this saved a lot of time and allowed me to experiment more efficiently in the future!\n\nThank for sharing!", "votes": 1 }, { "id": 2139626, "postDate": "2023-02-10T23:35:14.827Z", "content": "Congratulations! Great read!
", "rawMarkdown": "Congratulations! Great read!", "votes": 1 }, { "id": 2138097, "postDate": "2023-02-10T14:55:52.430Z", "content": "I saw that in your open source code, the saved file name of the feature is gpu-116
\n;gpu-220 and so on, why are they named so?
This was experiment 116 and experiment 220. I made hundreds of different co-visit matrices and computed validation score to see which were the best.
", "rawMarkdown": "This was experiment 116 and experiment 220. I made hundreds of different co-visit matrices and computed validation score to see which were the best.", "votes": 1 } ] }, { "id": 2131421, "postDate": "2023-02-06T05:14:31.107Z", "content": "Hello Chris,
\nGreat work and Congratulations.
\nI try to implement your idea but feel a little confused in some covisit items.
\nCould you slightly talk about if you don't mind:
Thanks ChickenBoy,
\ntop_20_new is clicks/carts/orders to carts/orders without time decay. And top_20_new2 is clicks/carts/orders to clicks/carts/orders with time decay.
Cold start is user_first_action - dataset_start_time > 2 weeks
df['x'] = df.groupby('session').ts.transform('min')
\n df = df.loc[df.x > train.ts.min() + TWO_WEEKS ]
\n df['n'] = df.groupby('session').cumcount()
\n df2 = df.loc[df['n']==0]
\n df3 = df.loc[df['type'].isin(['carts','orders'])]
\n df = df2.merge(df3, on='session')
In top_40_more/less, we filter users
\ndf = df.loc[df[COUNT]<THRESHOLD]
\n df = df.merge(df, on='session')
And in top_40_more2/less2, we filter first item.
\ndf2 = df.loc[df[HOUR]<14]\ndf = df2.merge(df, on='session')\nA useful share. Thanks I will use this in my project
", "rawMarkdown": "A useful share. Thanks I will use this in my project", "votes": 1 }, { "id": 2129759, "postDate": "2023-02-04T20:51:34.087Z", "content": "Nice write down
", "rawMarkdown": "Nice write down", "votes": 1 }, { "id": 2128550, "postDate": "2023-02-03T19:47:20.930Z", "content": "Great breakdown
", "rawMarkdown": "Great breakdown", "votes": 1 }, { "id": 2127381, "postDate": "2023-02-02T21:03:21.687Z", "content": "Congratulations! @cdeotte
\nAnd Thanks it helped me to understand about Competition ranking in Kaggle.
Thank you for sharing. It helped me a lot.
\nCould you share the code for building a reank model using your CV notebook and LB notebook?
\nI had a hard time making a reank model using candidates.
\nThank you.
", "rawMarkdown": "Thank you for sharing. It helped me a lot.\n\nCould you share the code for building a reank model using your CV notebook and LB notebook?\n\nI had a hard time making a reank model using candidates.\n\nThank you.", "votes": 1 }, { "id": 2125980, "postDate": "2023-02-02T03:27:05.887Z", "content": "Congratulations on your gold medal, and thanks for your great help - your code and suggestions were very influential (and have helped me get my first medal!)
\nYour work actually introduced me to cuDF, which I'm still new with and have a question I hope you could help. Beside reducing dtype, to preserve GPU memory I also remove unused stuffs, but from my experience del and gc.collect() do not always help with lowering GPU memory. Take the example code snippet below - the memory usage doubles after the merge, and does not reduce after del and gc.collect(). Is it because of some caching property of cuDF?
\ndf1 = cudf.from_pandas(pd.load_parquet(path_1)) \ndf2 = cudf.from_pandas(pd.load_parquet(path_2)) \ndf1 = df1.merge(df2, on=, how=) \n df1, df2 \n_ = gc.collect() \nThank you!
", "rawMarkdown": "Congratulations on your gold medal, and thanks for your great help - your code and suggestions were very influential (and have helped me get my first medal!)\nYour work actually introduced me to `cuDF`, which I'm still new with and have a question I hope you could help. Beside reducing `dtype`, to preserve GPU memory I also remove unused stuffs, but from my experience `del` and `gc.collect()` do not always help with lowering GPU memory. Take the example code snippet below - the memory usage doubles after the `merge`, and does not reduce after `del` and `gc.collect()`. Is it because of some caching property of `cuDF`?\n```python\n# GPU memory < 0.1 GBs\ndf1 = cudf.from_pandas(pd.load_parquet(path_1)) # GPU memory ~ 3.5 GBs\ndf2 = cudf.from_pandas(pd.load_parquet(path_2)) # GPU memory ~ 3.6 GBs\ndf1 = df1.merge(df2, on='session', how='left') # GPU memory ~ 7.0 GBs\ndel df1, df2 # GPU memory ~ 7.0 GBs\n_ = gc.collect() # GPU memory ~ 7.0 GBs\n```\nThank you!", "votes": 1, "replies": [ { "id": 2126051, "postDate": "2023-02-02T04:22:58.530Z", "content": "yes, sometimes memory usage does not decrease. In this situation, i save dataframe to disk as parquet. Then i shut down notebook and start a new notebook and read in the parquet. That will clear memory.
", "rawMarkdown": "yes, sometimes memory usage does not decrease. In this situation, i save dataframe to disk as parquet. Then i shut down notebook and start a new notebook and read in the parquet. That will clear memory.", "votes": 1, "replies": [ { "id": 2126144, "postDate": "2023-02-02T05:42:19.227Z", "content": "That's what I had to do to too! Also I google-ed around for an option to release unused GPU memory and found nothing related to cuDF but a few options for PyTorch (torch.cuda.empty_cache, if I'm correct?). Would be nice if cuDF has something similar.
\nThanks Chris!
I think this can help:
\n#Free The GPU Memory
\nfrom numba import cuda
\ncuda.select_device(0)
\ncuda.close()
\ncuda.select_device(0)
Thanks Mohamed! I have not tried this, but based on numba documentation this seems equivalent to restarting the kernel. Have you tried it?
", "rawMarkdown": "Thanks Mohamed! I have not tried this, but based on numba documentation this seems equivalent to restarting the kernel. Have you tried it?", "votes": 1 }, { "id": 2126678, "postDate": "2023-02-02T11:44:11.280Z", "content": "I used it several times in \"American Express Competition\". Sometimes I encounter a permission issue (Not sure honestly why it appears sometimes), but other than that, it works perfectly.
", "rawMarkdown": "I used it several times in \"American Express Competition\". Sometimes I encounter a permission issue (Not sure honestly why it appears sometimes), but other than that, it works perfectly.", "votes": 1 }, { "id": 2126917, "postDate": "2023-02-02T15:04:20.790Z", "content": "Sounds like what I'm looking for - will make sure to try it next time. Thanks again!
", "rawMarkdown": "Sounds like what I'm looking for - will make sure to try it next time. Thanks again!", "votes": 1 } ] }, { "id": 2129883, "postDate": "2023-02-05T01:20:17.170Z", "content": "Put your cudf operations inside a function. Then python releases GPU memory used inside the function after the return.
", "rawMarkdown": "Put your cudf operations inside a function. Then python releases GPU memory used inside the function after the return.", "votes": 4 } ] }, { "id": 2125898, "postDate": "2023-02-02T01:24:28.653Z", "content": "super cool!
", "rawMarkdown": "super cool!", "votes": 1, "replies": [ { "id": 2125916, "postDate": "2023-02-02T01:50:43.040Z", "content": "Thanks Adam
", "rawMarkdown": "Thanks Adam" } ] }, { "id": 2125886, "postDate": "2023-02-02T00:58:10.020Z", "content": "Thank you Chris, I won't get current ranking wihtout your public notebook。
\nI have some doubts in the your gbdt public notebook ：
\nI would like to know why there is no need to specify a group in Inference?
\nHow does the model judge whether the data is in the same group during inference?
\nDoes this mean that the inference group must be the same as the training group?
\nI looked at the docs but didn't find what I want to know, am I missing something?
Please forgive me for asking the question here again 😂because I really want to know the answer
", "rawMarkdown": "Thank you Chris, I won't get current ranking wihtout your public notebook。\n\nI have some doubts in the your gbdt public notebook ：\nI would like to know why there is no need to specify a group in Inference?\nHow does the model judge whether the data is in the same group during inference?\nDoes this mean that the inference group must be the same as the training group?\nI looked at the docs but didn't find what I want to know, am I missing something?\n\nPlease forgive me for asking the question here again 😂because I really want to know the answer\n", "votes": 1, "replies": [ { "id": 2125888, "postDate": "2023-02-02T01:08:33.187Z", "content": "Group is not needed during inference. The GBT will rank all rows during inference. So when we just look at each user during inference, the ranking will be correct.
\n(Group is only needed during training)
", "rawMarkdown": "Group is not needed during inference. The GBT will rank all rows during inference. So when we just look at each user during inference, the ranking will be correct.\n\n(Group is only needed during training)", "votes": 1, "replies": [ { "id": 2126217, "postDate": "2023-02-02T06:10:49.377Z", "content": "Thanks Chris!
", "rawMarkdown": "Thanks Chris!", "votes": 1 } ] } ] }, { "id": 2130617, "postDate": "2023-02-05T15:20:20.370Z", "content": "Nice work, it was very interesting to read!
", "rawMarkdown": "Nice work, it was very interesting to read! ", "votes": 2, "replies": [ { "id": 2136039, "postDate": "2023-02-09T04:09:49.487Z", "content": "Thanks Andrew!
", "rawMarkdown": "Thanks Andrew!", "votes": 1 } ] }, { "id": 2129630, "postDate": "2023-02-04T19:01:51.847Z", "content": "The information you share is very valuable. Thanks for sharing. I will use it in my next rule based project
", "rawMarkdown": "The information you share is very valuable. Thanks for sharing. I will use it in my next rule based project", "votes": 2 }, { "id": 2128312, "postDate": "2023-02-03T16:05:20.163Z", "content": "Congratulations, @cdeotte on your outstanding results! Your exceptional problem-solving skills and strong intuition have truly inspired me. I have gained so much valuable knowledge from you. I am grateful for all the lessons and experiences you have shared with us. Thank you!
", "rawMarkdown": "Congratulations, @cdeotte on your outstanding results! Your exceptional problem-solving skills and strong intuition have truly inspired me. I have gained so much valuable knowledge from you. I am grateful for all the lessons and experiences you have shared with us. Thank you!", "votes": 2 }, { "id": 2127634, "postDate": "2023-02-03T02:24:13.223Z", "content": "Congrats @cdeotte and team on great results. Thanks for sharing throughout the competition, again learn alot from you!
", "rawMarkdown": "Congrats @cdeotte and team on great results. Thanks for sharing throughout the competition, again learn alot from you!", "votes": 2 }, { "id": 2126779, "postDate": "2023-02-02T13:01:10.733Z", "content": "Congratulations! It's incredible how the rule based models could be pushed. My limit was 0.581 LB with rule based.
\nNext time I'll try out some of the ideas you presented.
Again congratulations!
", "rawMarkdown": "Congratulations! It's incredible how the rule based models could be pushed. My limit was 0.581 LB with rule based.\nNext time I'll try out some of the ideas you presented.\n\nAgain congratulations!", "votes": 2 }, { "id": 2126498, "postDate": "2023-02-02T09:08:12.020Z", "content": "I love your way to explain solution starting from baseline notebook.
\nI really learned a lot from your notebook and discusssions.
\nCongrats!
This is epic! Thanks for sharing @cdeotte!
\nOut of curiosity, what makes you stick with rule-based and creating various different covisit matrix while there's other techniques such as CF, MF and word2vec?
and given all the count features, did you tune the ranker?
", "rawMarkdown": "This is epic! Thanks for sharing @cdeotte!\nOut of curiosity, what makes you stick with rule-based and creating various different covisit matrix while there's other techniques such as CF, MF and word2vec?\n\nand given all the count features, did you tune the ranker?", "votes": 2, "replies": [ { "id": 2126053, "postDate": "2023-02-02T04:24:06.760Z", "content": "I did everything. I pushed rule-based as far as it would go. Afterward, i used word2vec and matrix factorization to create features for my GBT reranker. Both helped.
", "rawMarkdown": "I did everything. I pushed rule-based as far as it would go. Afterward, i used word2vec and matrix factorization to create features for my GBT reranker. Both helped.", "votes": 1, "replies": [ { "id": 2126080, "postDate": "2023-02-02T04:52:08.560Z", "content": "Do you think LSTM Embeddings would help more than Word2Vec given that we have aids sorted by time? Just want to know if you made a similar experiment?
", "rawMarkdown": "Do you think LSTM Embeddings would help more than Word2Vec given that we have aids sorted by time? Just want to know if you made a similar experiment? ", "votes": 1 }, { "id": 2126096, "postDate": "2023-02-02T05:08:53.167Z", "content": "I think all item embeddings would help. The trick was to first train some NN such as LSTM, Word2Vec, Matrix Factorization etc.
\nThen we take the item embeddings and for each row in our candidate dataframe, we compute the cosine similarity between user history last item and candidate. Then user history 2nd to last, 3rd, 4th 5th. Then user history first item, 2nd, 3rd, 4th, 5th. Then user history last buy, 2nd to last buy, 3rd, 4th, 5th. Then finally we take aggregate means and standard deviations of groups of these values. These features were very helpful.
", "rawMarkdown": "I think all item embeddings would help. The trick was to first train some NN such as LSTM, Word2Vec, Matrix Factorization etc.\n\nThen we take the item embeddings and for each row in our candidate dataframe, we compute the cosine similarity between user history last item and candidate. Then user history 2nd to last, 3rd, 4th 5th. Then user history first item, 2nd, 3rd, 4th, 5th. Then user history last buy, 2nd to last buy, 3rd, 4th, 5th. Then finally we take aggregate means and standard deviations of groups of these values. These features were very helpful.", "votes": 3 }, { "id": 2126106, "postDate": "2023-02-02T05:15:13.997Z", "content": "Nice! I tried a similar idea. I used Word2Vec, but calculated the cosine similarity between last, 2nd to last and 3rd to last interacted items and the last, 2nd to last and 3rd to last clicked/carted/ordered items. Honestly, I wanted to try the similarity between last clicked/carted/ordered and candidates but couldn't find enough time for that as that it was already the last day of the competition. Kudos to @tezdhar for our great ideas!
", "rawMarkdown": "Nice! I tried a similar idea. I used Word2Vec, but calculated the cosine similarity between last, 2nd to last and 3rd to last interacted items and the last, 2nd to last and 3rd to last clicked/carted/ordered items. Honestly, I wanted to try the similarity between last clicked/carted/ordered and candidates but couldn't find enough time for that as that it was already the last day of the competition. Kudos to @tezdhar for our great ideas!", "votes": 1 }, { "id": 2126108, "postDate": "2023-02-02T05:17:55.547Z", "content": "as always thanks for sharing clear and details tricks
\nI am curious about how do you compute multiple item comparisons in fast way?
\nI am stuck with for loop iteration with following steps
it's functional but it takes a lot of time to complete
", "rawMarkdown": "as always thanks for sharing clear and details tricks\n\nI am curious about how do you compute multiple item comparisons in fast way?\nI am stuck with for loop iteration with following steps\n1. in each iteration, i take vectors of item1 & item2 and append it to list items1, items2\n2. at the end of iterations, convert items1 and items2 to numpy array\n3. compute the similarity feature with vectorized cosine similarity function\n\nit's functional but it takes a lot of time to complete", "votes": 1 }, { "id": 2126193, "postDate": "2023-02-02T05:59:29.677Z", "content": "I'm not sure what you're asking. I use cupy and GPU to perform the vector multiplication of user history item with candidate item embedding. It takes a few seconds to create a new column of cosine similarity. I create dozens of features then train XGB reranker. I do not worry about making too many features. My pipeline can handle 1000+ features. (Because for XGB reranker i remove all users where no candidate has a positive target).
\n\nI do not worry about making too many features. My pipeline can handle 1000+ features. (Because for XGB reranker i remove all users where no candidate has a positive target).
\n
For us working with kaggle notebooks this is not an option. As for clicks, most sessions do have a positive target, and I struggled to fit 18 features for click model into available memory. Downsampled negatives, converted all the float features into float16 after loading from parquet, and felt I was close to maximum available features.
\nFor carts/orders kaggle notebook with GPU support could probably handle about 100-150 features, but again no chance for 1000+.
P.S. congratulations and thank you for your usefull posts!
", "rawMarkdown": ">I do not worry about making too many features. My pipeline can handle 1000+ features. (Because for XGB reranker i remove all users where no candidate has a positive target).\n\nFor us working with kaggle notebooks this is not an option. As for clicks, most sessions do have a positive target, and I struggled to fit 18 features for click model into available memory. Downsampled negatives, converted all the float features into float16 after loading from parquet, and felt I was close to maximum available features.\nFor carts/orders kaggle notebook with GPU support could probably handle about 100-150 features, but again no chance for 1000+.\n\nP.S. congratulations and thank you for your usefull posts!", "votes": 1 } ] } ] }, { "id": 2125893, "postDate": "2023-02-02T01:19:20.587Z", "content": "Amazing ！0.590 without reranker. Thanks so much for your all tutorial otherwise I didn't know how to play a recommend game.
", "rawMarkdown": "Amazing ！0.590 without reranker. Thanks so much for your all tutorial otherwise I didn't know how to play a recommend game.", "votes": 2, "replies": [ { "id": 2125903, "postDate": "2023-02-02T01:35:32.117Z", "content": "Thanks EeyoreLee. Congratulations on your solo Bronze medal finish. Your LB 0.593 was very close to Silver 0.594. You did great !
", "rawMarkdown": "Thanks EeyoreLee. Congratulations on your solo Bronze medal finish. Your LB 0.593 was very close to Silver 0.594. You did great !", "votes": 2 } ] }, { "id": 3001255, "postDate": "2024-09-28T16:12:23.177Z", "content": "Hi，thanks for your contribution！I have a question：Why not directly read train.jsonl line by line and then determine whether each line should be added to the training set or validation set? What is the purpose of constructing session_chunks?Sincerely hope for your answer！
", "rawMarkdown": "Hi，thanks for your contribution！I have a question：Why not directly read train.jsonl line by line and then determine whether each line should be added to the training set or validation set? What is the purpose of constructing session_chunks?Sincerely hope for your answer！" }, { "id": 2165553, "postDate": "2023-03-02T09:16:21.490Z", "content": "Congratulations !!!
", "rawMarkdown": "Congratulations !!! " }, { "id": 2129389, "postDate": "2023-02-04T15:39:46.547Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2129119, "postDate": "2023-02-04T10:47:07.627Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 2131517, "postDate": "2023-02-06T07:13:13.363Z", "content": "Thanks for your sharing.
", "rawMarkdown": "Thanks for your sharing.", "votes": 1 }, { "id": 3235131, "postDate": "2025-06-28T18:26:54.350Z", "content": "Thank you for the detail writeup!
", "rawMarkdown": "Thank you for the detail writeup!" } ], "comments": [ { "id": 2144158, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-02-14T19:15:12.750000", "content": "UPDATE: I published all 261 jupyter notebooks to run my full single model LB 0.601 solution at GitHub here: https://github.com/cdeotte/Kaggle-OTTO-Comp
", "votes": 7, "replies": [] }, { "id": 2127112, "author_name": "Artem Fedorov", "author_url": "", "post_date": "2023-02-02T17:23:28.827000", "content": "Everything looks so easy when reading the describtion of your solution. I've even thought for a moment \"how could it happen that I didn't try it myself?\"
\nWith just kaggle notebooks running all the possible experiments is more tricky. And anyway, I didn't even think about some of your matrices, like before/after 2PM, only users with short history, only last week. I did make a matrice for exact next aid, but only used it for clicks and didn't even try to apply it to carts/orders.
And again, thank you for you clear and insightfull posts, both during and after the competition.
", "votes": 3, "replies": [] }, { "id": 2127417, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2023-02-02T21:45:54.857000", "content": "Great job @cdeotte and the rest of the team! 🥳 Amazing performance!
\nThis write up reads like a good detective story, the plot is so clearly explained! So cool to understand what you did to achieve this phenomenal result. Love the progression from the public kernel with the co-visitation matrices there to the full solution using the reranker!
\nCongrats again and thank you for this great write-up!
", "votes": 4, "replies": [ { "id": 2128306, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-02-03T15:59:16.260000", "content": "Thanks Radek. Thanks for all your sharing during the competition!
", "votes": 0, "replies": [] } ] }, { "id": 2186719, "author_name": "马甲线来来来", "author_url": "", "post_date": "2023-03-18T03:42:24.003000", "content": "thx for sharing, really learn a lot from that
", "votes": 1, "replies": [] }, { "id": 2146750, "author_name": "Girijesh", "author_url": "", "post_date": "2023-02-16T05:38:22.827000", "content": "Congratulations @cdeotte excellent work.
", "votes": 1, "replies": [] }, { "id": 2141778, "author_name": "Ryan", "author_url": "", "post_date": "2023-02-13T05:00:00.417000", "content": "Wow, congratulations on taking 3rd place in the competition! Your solution is impressive OAO
\nI learnt your approach to using RAPID cuDF to quickly compute hundreds of covisit matrices and then compute the local CV score is clever. I can imagine that this saved a lot of time and allowed me to experiment more efficiently in the future!
\nThank for sharing!
", "votes": 1, "replies": [] }, { "id": 2139626, "author_name": "Mayuri Deshpande", "author_url": "", "post_date": "2023-02-10T23:35:14.827000", "content": "Congratulations! Great read!
", "votes": 1, "replies": [] }, { "id": 2138097, "author_name": "Apa", "author_url": "", "post_date": "2023-02-10T14:55:52.430000", "content": "I saw that in your open source code, the saved file name of the feature is gpu-116
\n;gpu-220 and so on, why are they named so?
This was experiment 116 and experiment 220. I made hundreds of different co-visit matrices and computed validation score to see which were the best.
", "votes": 1, "replies": [] } ] }, { "id": 2131421, "author_name": "Nick Liu", "author_url": "", "post_date": "2023-02-06T05:14:31.107000", "content": "Hello Chris,
\nGreat work and Congratulations.
\nI try to implement your idea but feel a little confused in some covisit items.
\nCould you slightly talk about if you don't mind:
Thanks ChickenBoy,
\ntop_20_new is clicks/carts/orders to carts/orders without time decay. And top_20_new2 is clicks/carts/orders to clicks/carts/orders with time decay.
Cold start is user_first_action - dataset_start_time > 2 weeks
df['x'] = df.groupby('session').ts.transform('min')
\n df = df.loc[df.x > train.ts.min() + TWO_WEEKS ]
\n df['n'] = df.groupby('session').cumcount()
\n df2 = df.loc[df['n']==0]
\n df3 = df.loc[df['type'].isin(['carts','orders'])]
\n df = df2.merge(df3, on='session')
In top_40_more/less, we filter users
\ndf = df.loc[df[COUNT]<THRESHOLD]
\n df = df.merge(df, on='session')
And in top_40_more2/less2, we filter first item.
\ndf2 = df.loc[df[HOUR]<14]\ndf = df2.merge(df, on='session')\nThanks! Really helpful👍👍👍
", "votes": 2, "replies": [] } ] } ] }, { "id": 2129831, "author_name": "ibrahim karatas", "author_url": "", "post_date": "2023-02-04T22:50:36.573000", "content": "A useful share. Thanks I will use this in my project
", "votes": 1, "replies": [] }, { "id": 2129759, "author_name": "ToriGlori", "author_url": "", "post_date": "2023-02-04T20:51:34.087000", "content": "Nice write down
", "votes": 1, "replies": [] }, { "id": 2128550, "author_name": "andrey bobrik", "author_url": "", "post_date": "2023-02-03T19:47:20.930000", "content": "Great breakdown
", "votes": 1, "replies": [] }, { "id": 2127381, "author_name": "S Victor Kumar", "author_url": "", "post_date": "2023-02-02T21:03:21.687000", "content": "Congratulations! @cdeotte
\nAnd Thanks it helped me to understand about Competition ranking in Kaggle.
Thank you for sharing. It helped me a lot.
\nCould you share the code for building a reank model using your CV notebook and LB notebook?
\nI had a hard time making a reank model using candidates.
\nThank you.
", "votes": 1, "replies": [] }, { "id": 2125980, "author_name": "Hoang Nguyen", "author_url": "", "post_date": "2023-02-02T03:27:05.887000", "content": "Congratulations on your gold medal, and thanks for your great help - your code and suggestions were very influential (and have helped me get my first medal!)
\nYour work actually introduced me to cuDF, which I'm still new with and have a question I hope you could help. Beside reducing dtype, to preserve GPU memory I also remove unused stuffs, but from my experience del and gc.collect() do not always help with lowering GPU memory. Take the example code snippet below - the memory usage doubles after the merge, and does not reduce after del and gc.collect(). Is it because of some caching property of cuDF?
\ndf1 = cudf.from_pandas(pd.load_parquet(path_1)) \ndf2 = cudf.from_pandas(pd.load_parquet(path_2)) \ndf1 = df1.merge(df2, on=, how=) \n df1, df2 \n_ = gc.collect() \nThank you!
", "votes": 1, "replies": [ { "id": 2126051, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-02-02T04:22:58.530000", "content": "yes, sometimes memory usage does not decrease. In this situation, i save dataframe to disk as parquet. Then i shut down notebook and start a new notebook and read in the parquet. That will clear memory.
", "votes": 1, "replies": [ { "id": 2126144, "author_name": "Hoang Nguyen", "author_url": "", "post_date": "2023-02-02T05:42:19.227000", "content": "That's what I had to do to too! Also I google-ed around for an option to release unused GPU memory and found nothing related to cuDF but a few options for PyTorch (torch.cuda.empty_cache, if I'm correct?). Would be nice if cuDF has something similar.
\nThanks Chris!
I think this can help:
\n#Free The GPU Memory
\nfrom numba import cuda
\ncuda.select_device(0)
\ncuda.close()
\ncuda.select_device(0)
Thanks Mohamed! I have not tried this, but based on numba documentation this seems equivalent to restarting the kernel. Have you tried it?
", "votes": 1, "replies": [] }, { "id": 2126678, "author_name": "Mohamed Eltayeb", "author_url": "", "post_date": "2023-02-02T11:44:11.280000", "content": "I used it several times in \"American Express Competition\". Sometimes I encounter a permission issue (Not sure honestly why it appears sometimes), but other than that, it works perfectly.
", "votes": 1, "replies": [] }, { "id": 2126917, "author_name": "Hoang Nguyen", "author_url": "", "post_date": "2023-02-02T15:04:20.790000", "content": "Sounds like what I'm looking for - will make sure to try it next time. Thanks again!
", "votes": 1, "replies": [] } ] }, { "id": 2129883, "author_name": "Giba", "author_url": "", "post_date": "2023-02-05T01:20:17.170000", "content": "Put your cudf operations inside a function. Then python releases GPU memory used inside the function after the return.
", "votes": 4, "replies": [] } ] }, { "id": 2125898, "author_name": "ADAM.", "author_url": "", "post_date": "2023-02-02T01:24:28.653000", "content": "super cool!
", "votes": 1, "replies": [ { "id": 2125916, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-02-02T01:50:43.040000", "content": "Thanks Adam
", "votes": 0, "replies": [] } ] }, { "id": 2125886, "author_name": "cooperation", "author_url": "", "post_date": "2023-02-02T00:58:10.020000", "content": "Thank you Chris, I won't get current ranking wihtout your public notebook。
\nI have some doubts in the your gbdt public notebook ：
\nI would like to know why there is no need to specify a group in Inference?
\nHow does the model judge whether the data is in the same group during inference?
\nDoes this mean that the inference group must be the same as the training group?
\nI looked at the docs but didn't find what I want to know, am I missing something?
Please forgive me for asking the question here again 😂because I really want to know the answer
", "votes": 1, "replies": [ { "id": 2125888, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-02-02T01:08:33.187000", "content": "Group is not needed during inference. The GBT will rank all rows during inference. So when we just look at each user during inference, the ranking will be correct.
\n(Group is only needed during training)
", "votes": 1, "replies": [ { "id": 2126217, "author_name": "cooperation", "author_url": "", "post_date": "2023-02-02T06:10:49.377000", "content": "Thanks Chris!
", "votes": 1, "replies": [] } ] } ] }, { "id": 2130617, "author_name": "Andrei Kavalenka", "author_url": "", "post_date": "2023-02-05T15:20:20.370000", "content": "Nice work, it was very interesting to read!
", "votes": 2, "replies": [ { "id": 2136039, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-02-09T04:09:49.487000", "content": "Thanks Andrew!
", "votes": 1, "replies": [] } ] }, { "id": 2129630, "author_name": "ibrahim karatas", "author_url": "", "post_date": "2023-02-04T19:01:51.847000", "content": "The information you share is very valuable. Thanks for sharing. I will use it in my next rule based project
", "votes": 2, "replies": [] }, { "id": 2128312, "author_name": "Taha Muhammad Haider", "author_url": "", "post_date": "2023-02-03T16:05:20.163000", "content": "Congratulations, @cdeotte on your outstanding results! Your exceptional problem-solving skills and strong intuition have truly inspired me. I have gained so much valuable knowledge from you. I am grateful for all the lessons and experiences you have shared with us. Thank you!
", "votes": 2, "replies": [] }, { "id": 2127634, "author_name": "KhanhVD", "author_url": "", "post_date": "2023-02-03T02:24:13.223000", "content": "Congrats @cdeotte and team on great results. Thanks for sharing throughout the competition, again learn alot from you!
", "votes": 2, "replies": [] }, { "id": 2126779, "author_name": "Pietro Maldini", "author_url": "", "post_date": "2023-02-02T13:01:10.733000", "content": "Congratulations! It's incredible how the rule based models could be pushed. My limit was 0.581 LB with rule based.
\nNext time I'll try out some of the ideas you presented.
Again congratulations!
", "votes": 2, "replies": [] }, { "id": 2126498, "author_name": "Wonho Song", "author_url": "", "post_date": "2023-02-02T09:08:12.020000", "content": "I love your way to explain solution starting from baseline notebook.
\nI really learned a lot from your notebook and discusssions.
\nCongrats!
This is epic! Thanks for sharing @cdeotte!
\nOut of curiosity, what makes you stick with rule-based and creating various different covisit matrix while there's other techniques such as CF, MF and word2vec?
and given all the count features, did you tune the ranker?
", "votes": 2, "replies": [ { "id": 2126053, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-02-02T04:24:06.760000", "content": "I did everything. I pushed rule-based as far as it would go. Afterward, i used word2vec and matrix factorization to create features for my GBT reranker. Both helped.
", "votes": 1, "replies": [ { "id": 2126080, "author_name": "Mohamed Eltayeb", "author_url": "", "post_date": "2023-02-02T04:52:08.560000", "content": "Do you think LSTM Embeddings would help more than Word2Vec given that we have aids sorted by time? Just want to know if you made a similar experiment?
", "votes": 1, "replies": [] }, { "id": 2126096, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-02-02T05:08:53.167000", "content": "I think all item embeddings would help. The trick was to first train some NN such as LSTM, Word2Vec, Matrix Factorization etc.
\nThen we take the item embeddings and for each row in our candidate dataframe, we compute the cosine similarity between user history last item and candidate. Then user history 2nd to last, 3rd, 4th 5th. Then user history first item, 2nd, 3rd, 4th, 5th. Then user history last buy, 2nd to last buy, 3rd, 4th, 5th. Then finally we take aggregate means and standard deviations of groups of these values. These features were very helpful.
", "votes": 3, "replies": [] }, { "id": 2126106, "author_name": "Mohamed Eltayeb", "author_url": "", "post_date": "2023-02-02T05:15:13.997000", "content": "Nice! I tried a similar idea. I used Word2Vec, but calculated the cosine similarity between last, 2nd to last and 3rd to last interacted items and the last, 2nd to last and 3rd to last clicked/carted/ordered items. Honestly, I wanted to try the similarity between last clicked/carted/ordered and candidates but couldn't find enough time for that as that it was already the last day of the competition. Kudos to @tezdhar for our great ideas!
", "votes": 1, "replies": [] }, { "id": 2126108, "author_name": "Aji Samudra", "author_url": "", "post_date": "2023-02-02T05:17:55.547000", "content": "as always thanks for sharing clear and details tricks
\nI am curious about how do you compute multiple item comparisons in fast way?
\nI am stuck with for loop iteration with following steps
it's functional but it takes a lot of time to complete
", "votes": 1, "replies": [] }, { "id": 2126193, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-02-02T05:59:29.677000", "content": "I'm not sure what you're asking. I use cupy and GPU to perform the vector multiplication of user history item with candidate item embedding. It takes a few seconds to create a new column of cosine similarity. I create dozens of features then train XGB reranker. I do not worry about making too many features. My pipeline can handle 1000+ features. (Because for XGB reranker i remove all users where no candidate has a positive target).
\n\nI do not worry about making too many features. My pipeline can handle 1000+ features. (Because for XGB reranker i remove all users where no candidate has a positive target).
\n
For us working with kaggle notebooks this is not an option. As for clicks, most sessions do have a positive target, and I struggled to fit 18 features for click model into available memory. Downsampled negatives, converted all the float features into float16 after loading from parquet, and felt I was close to maximum available features.
\nFor carts/orders kaggle notebook with GPU support could probably handle about 100-150 features, but again no chance for 1000+.
P.S. congratulations and thank you for your usefull posts!
", "votes": 1, "replies": [] } ] } ] }, { "id": 2125893, "author_name": "EeyoreLee", "author_url": "", "post_date": "2023-02-02T01:19:20.587000", "content": "Amazing ！0.590 without reranker. Thanks so much for your all tutorial otherwise I didn't know how to play a recommend game.
", "votes": 2, "replies": [ { "id": 2125903, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-02-02T01:35:32.117000", "content": "Thanks EeyoreLee. Congratulations on your solo Bronze medal finish. Your LB 0.593 was very close to Silver 0.594. You did great !
", "votes": 2, "replies": [] } ] }, { "id": 3001255, "author_name": "YuhaoWu123", "author_url": "", "post_date": "2024-09-28T16:12:23.177000", "content": "Hi，thanks for your contribution！I have a question：Why not directly read train.jsonl line by line and then determine whether each line should be added to the training set or validation set? What is the purpose of constructing session_chunks?Sincerely hope for your answer！
", "votes": 0, "replies": [] }, { "id": 2165553, "author_name": "Siliva1799", "author_url": "", "post_date": "2023-03-02T09:16:21.490000", "content": "Congratulations !!!
", "votes": 0, "replies": [] }, { "id": 2129389, "author_name": "", "author_url": "", "post_date": "2023-02-04T15:39:46.547000", "content": "", "votes": 0, "replies": [] }, { "id": 2129119, "author_name": "", "author_url": "", "post_date": "2023-02-04T10:47:07.627000", "content": "", "votes": 1, "replies": [] }, { "id": 2131517, "author_name": "", "author_url": "", "post_date": "2023-02-06T07:13:13.363000", "content": "", "votes": 1, "replies": [] }, { "id": 3235131, "author_name": "", "author_url": "", "post_date": "2025-06-28T18:26:54.350000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2125843": "# Team G & B & D & T\nIt was a pleasure to work with @titericz @benediktschifferer @theoviel . We each made individual models and then ensembled all our work together by adding the ranks of each of our predictions per user target type. Below I describe my individual single model. My teammates will describe their work in their own discussion posts. You can read about Theo's LB 0.6029 model [here][5]! You can read about Benny's LB 0.601 model [here][8]!\n\nMost of my solution was made public in my notebook [here][3] and discussion [here][2] during the competition. There were only 3 significant ideas missing from my public work. Let's discuss how to boost my public work to LB 0.601 single model. (UPDATE: All code published to GitHub [here][6])\n\n# How To Score LB 0.601 Single Model\nIt was explained [here][1] and [here][2] that the best approach was \"candidate rerank\" model. My public notebook [here][3] shows how to achieve LB 0.575. And my discussion post [here][2] explains how to improve my public notebook by adding a GBT ranker model. Below are the three missing pieces labeled (1), (2), and (3) to achieve LB 0.601\n\n# (1) Choosing Candidates\nTo build a \"candidate rerank\", we need candidates. Where do we get candidates? The easiest way is to get them from my public notebook. In the `def suggest_buys(df)` and `def suggest_clicks(df)` function, the last lines are\n\n top_aids = [aid for aid, ct in aids_counter.most_common(20)] \n return top_aids\n\nTo generate 50 candidates, we change 20 to 50 as in\n\n top_aids = [aid for aid, ct in aids_counter.most_common(50)] \n return top_aids\n\n# (2) Choosing Interaction Features\nWe now have 50 candidates per user from our public notebook above. Next we need to make features for our reranker model. The strongest and easiest way to make features is extract our co-visit counts by changing the last two lines of my public notebook to the following:\n\n top_counts = [ct for aid, ct in aids_counter.most_common(50)] \n return top_counts\n\nWhen we merge these counts to our candidates, we now have an interaction feature for each user item pair. To make more interaction features, we can extract the counts for each co-visit matrix individually. For example, imagine that we have 3 co-visit matrices named covisit2, covisit3, and covist4. Then one by one, we extract each covisit's counts:\n\n EXTRACT = ['covisit2']\n aids_counter = Counter()\n if 'covisit2' in EXTRACT:\n aids = list(itertools.chain(*[covisit2[aid] for aid in unique_aids if aid in covisit2]))\n for a in aids: aids_counter[a] += 1\n if 'covisit3' in EXTRACT:\n aids = list(itertools.chain(*[covisit3[aid] for aid in unique_aids if aid in covisit3]))\n for a in aids: aids_counter[a] += 1\n if 'covisit4' in EXTRACT:\n aids = list(itertools.chain(*[covisit4[aid] for aid in unique_aids if aid in covisit4]))\n for a in aids: aids_counter[a] += 1\n top_counts = [ct for aid, ct in aids_counter.most_common(50)] \n return top_counts\n\n# Reranker Boost CV and LB +0.011\nFirst we use the technique above to generate candidates and the technique above to extract covisit counts. Next we add some simple item and user features like counting the number of times an item is click cart or order. When we apply the XGB reranker described [here][2], our CV and LB will boost by `+0.011`. For example, the public notebook will boost to LB 0.586.\n\n# Using Rules Only (without reranker) Scores LB 0.590\nTo score over LB 0.600, we create more co-visit matrices to boost the original notebook's LB score. My public notebook uses 3 covisit matrices and achieves LB 0.575. If we make 17 more covisit matrices, we can boost my public \"rules only\" notebook to LB 0.590 (new notebook published [here][4]). Then when we extract the covisit counts explained above, the XGB reranker will boost +0.011 and achieve LB 0.601\n\n# (3) Twenty Covisit Matrices\nBelow are a description of my 20 covisit matrices. These covisit matrices are the secret sauce enabling my single XGB ranker model to achieve LB 0.601. The following variable names are from my new LB 0.590 notebook posted [here][4]. (Example code showing how to compute covisit matrices on GPU is [here][3])\n* **top_20** - this covisit matrix is in my original notebook\n* **top_20b** - all covisit pair counts are consecutive items. See code below.\n `df['k'] = np.arange(len(df))`\n `df = df.merge(df, on=['session'])`\n `df = df.loc[ (df.k_y - df.k_x).abs()==1 ]`\n* **top_20c** - all covisit pair counts are `(df.k_y - df.k_x).abs()<=2`\n* **top_20d** - all covisit pairs are carts/orders and forward at most 3 consecutive\n `df = df.loc[df['type'].isin(['carts','orders'])]`\n `df = df.merge(df, on=['session'])`\n `df = df.loc[ (df.k_y - df.k_x > 0) & (df.k_y - df.k_x <= 3) ]`\n* **top_20e** - all covisit pairs are `(df.k_y - df.k_x).abs()<=3` and have time decay with\n `df['wgt'] = (1/2)**( (df.ts_x - df.ts_y).abs() /60/60)`\n* **top_20f** - same as above but `(df.k_y - df.k_x).abs()<=6`\n* **top_20_orders** - this covisit matrix is in my original notebook\n* **top_20_buy2buy** - this covisit matrix is in my original notebook\n* **top_20_buy2buy2** - use most recent 3 weeks data and only carts/orders. Apply time decay shown above.\n* **top_20_test** - use most recent 3 weeks data. Only forward in time pairs. Use clicks/carts/orders to carts/orders. Add time decay\n `df = df.loc[df.ts >= LAST_3_WEEKS ]`\n `df2 = df.loc[df['type'].isin(['carts','orders'])]`\n `df = df.merge(df2, on=['session'])`\n `df = df.loc[ df.ts_y - df.ts_x > 0 ]`\n `df['wgt'] = (1/2)**( (df.ts_x - df.ts_y).abs() /60/60)`\n* **top_20_test2** - use most recent 2 weeks data with time decay.\n* **top_20_buy** - Limit to forward 2 hours. Use clicks/carts/orders to carts/orders. Apply time decay.\n* **top_20_new** - Find cold start users in train. Pairs using only their first history item. Use clicks/carts/orders to carts/orders.\n `df['x'] = df.groupby('session').ts.transform('min')`\n `df = df.loc[df.x > train.ts.min() + TWO_WEEKS ]`\n `df['n'] = df.groupby('session').cumcount()`\n `df2 = df.loc[df['n']==0]`\n `df3 = df.loc[df['type'].isin(['carts','orders'])]`\n `df = df2.merge(df3, on='session')`\n* **top_20_new2** - Find cold start users in train. Pairs using only their first history item. Use clicks/carts/orders to clicks/carts/orders. Apply time decay.\n* **top_40_day** - Use only last week data. Forward in time. Clicks/carts/orders to carts/orders. Time decay\n* **top_40_day2** - Use only last week data. Time decay\n* **top_40_less** - Train users with less than 6 history and test users with less than 3\n `df = df.loc[df[COUNT]