![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F83267%2F15ce231219e39c8f2f78d4b97fa8df60%2Fco_visitation_matrix.png?generation=1668155980264714&alt=media\")
I published a notebook here that uses RAPIDS cuDF to compute 30x faster co-visitation matrices. Using co-visitation matrices helps provide our models with \"candidates\". Then our models (and/or human logic) and \"rerank\" these \"candidates\" and select our final 20 predictions for submission CSV.
\nUPDATE π₯ latest version computes co-visitation matrices in 3min each π₯
\nFor more information about \"candidate rerank\" models, see Ravi's discussion here
", "messages": [ { "id": 2025071, "postDate": "2022-11-11T00:47:57.853Z", "content": "I published a notebook here that uses RAPIDS cuDF to compute 30x faster co-visitation matrices. Using co-visitation matrices helps provide our models with \"candidates\". Then our models (and/or human logic) and \"rerank\" these \"candidates\" and select our final 20 predictions for submission CSV.
\nUPDATE π₯ latest version computes co-visitation matrices in 3min each π₯
\nFor more information about \"candidate rerank\" models, see Ravi's discussion here
", "rawMarkdown": "I published a notebook [here][1] that uses RAPIDS cuDF to compute 30x faster co-visitation matrices. Using co-visitation matrices helps provide our models with \"candidates\". Then our models (and/or human logic) and \"rerank\" these \"candidates\" and select our final 20 predictions for submission CSV. \n\n**UPDATE** π₯ latest version computes co-visitation matrices in **3min** each π₯\n\nFor more information about \"candidate rerank\" models, see Ravi's discussion [here][2]\n\n[1]: https://www.kaggle.com/code/cdeotte/candidate-rerank-model-lb-0-573\n[2]: https://www.kaggle.com/competitions/otto-recommender-system/discussion/364721", "votes": 88 }, { "id": 2028397, "postDate": "2022-11-13T21:32:00.943Z", "content": "UPDATE: I updated my code to create co-visitation matrices 1.5x faster. It now creates co-visitation matrices in Kaggle notebooks at 6 minutes! (And offline using 1xV100 in 1.5 minutes!)
", "rawMarkdown": "UPDATE: I updated my code to create co-visitation matrices 1.5x faster. It now creates co-visitation matrices in Kaggle notebooks at 6 minutes! (And offline using 1xV100 in 1.5 minutes!)", "votes": 7, "replies": [ { "id": 2032873, "postDate": "2022-11-16T21:01:18.247Z", "content": "UPDATE: π₯ I updated my code again to create co-visitation matrices 2x faster. It now creates co-visitation matrices in Kaggle notebooks at 3 minutes! (This is 3x faster than version 1 of the notebook. And 30x faster than public notebooks using Pandas CPU) π₯
", "rawMarkdown": "UPDATE: π₯ I updated my code again to create co-visitation matrices 2x faster. It now creates co-visitation matrices in Kaggle notebooks at **3 minutes**! (This is **3x** faster than version 1 of the notebook. And **30x** faster than public notebooks using Pandas CPU) π₯", "votes": 4 } ] }, { "id": 2035237, "postDate": "2022-11-18T17:59:08.493Z", "content": "Hi Chris,
\nThanks for showing us this wonderful solution π.
\nI find it hard to improve on this simple heuristic approach combining information on IN_SESSION flag, TOP most popular items for given item in the session and TOP most popular items overall.
\nI was wondering, how did you come up with this heuristic? Is it something that you tried out before and it worked, or you tried out bunch of different techniques and this one worked the best or simply you had a lucky/intuitive guess?
\nBest
\nAndrej
", "rawMarkdown": "Hi Chris,\n\nThanks for showing us this wonderful solution π. \n\nI find it hard to improve on this simple heuristic approach combining information on IN_SESSION flag, TOP most popular items for given item in the session and TOP most popular items overall.\n\nI was wondering, how did you come up with this heuristic? Is it something that you tried out before and it worked, or you tried out bunch of different techniques and this one worked the best or simply you had a lucky/intuitive guess?\n\nBest \n\nAndrej", "votes": 1, "replies": [ { "id": 2036565, "postDate": "2022-11-19T22:22:38.557Z", "content": "There are two ways to improve CV and LB. Find more candidates and find better rerank logic. I discover both by using intuition and thinking about how users behave. Then i evaluate my ideas by observing if local validation score increases or not.
\nThe easiest way to improve my public notebook is to create more co-visitation matrices. It is possible to improve my notebook by changing the rerank logic, but it will be easier to just make more co-visitation matrices and add their weight in.
\nI suggest that you focus on the case of suggest_buys for len(unique_aids)<20. This has the largest effect on CV score and LB score. Change the code to allow for weights. The current code is
# USE \"CART ORDER\" CO-VISITATION MATRIX\naids2 = list(itertools.chain(*[top_20_buys[aid] for aid in unique_aids if aid in top_20_buys]))\n# USE \"BUY2BUY\" CO-VISITATION MATRIX\naids3 = list(itertools.chain(*[top_20_buy2buy[aid] for aid in unique_buys if aid in top_20_buy2buy]))\n# RERANK CANDIDATES\ntop_aids2 = [aid2 for aid2, cnt in Counter(aids2+aids3).most_common(20) if aid2 not in unique_aids] \nChange this to
\nfrom collections import Counter\naids_temp = Counter() \n# USE \"CART ORDER\" CO-VISITATION MATRIX\naids2 = list(itertools.chain(*[top_20_buys[aid] for aid in unique_aids if aid in top_20_buys]))\nfor aid in aids2: aids_temp[aid] += 1\n# USE \"BUY2BUY\" CO-VISITATION MATRIX\naids3 = list(itertools.chain(*[top_20_buy2buy[aid] for aid in unique_buys if aid in top_20_buy2buy]))\nfor aid in aids3: aids_temp[aid] += 1\n# RERANK CANDIDATES\ntop_aids2 = [k for k,v in aids_temp.most_common(20) if k not in unique_aids]\nNow we can create new co-visitation matrices and include them with
\naids4 = list(itertools.chain(*[NEW_MATRIX[aid] for aid in SUBSET_OF_USER_HISTORY_ITEMS if aid in NEW_MATRIX]))\nfor aid in aids4: aids_temp[aid] += 1\nYou need to be creative when making co-visitation matrices. The matrices makes pairs of items. If a certain type of user interacts in a certain type of way with item. Then we predict they will click and order a certain item in the future. Then when you include this new co-visitation matrix in your inference, we only apply it to the same certain type of user when they interact with an item in a certain type of way.
\nAlso note that once you change the code to include for aid in aids2: aids_temp[aid] += 1, you can now assign different situations different amount of weights during inference. This is how we change the rerank logic.
weights = LOGIC_TO_MAKES_WEIGHTS\nfor aid,w in zip(aids2,weights):\n aids_temp[aid] += w\nthis is so valuable and lets us experiment with different weighting approaches in an acceptable time! tnx chris!
", "rawMarkdown": "this is so valuable and lets us experiment with different weighting approaches in an acceptable time! tnx chris!", "votes": 1 }, { "id": 2027772, "postDate": "2022-11-13T07:20:54.380Z", "content": "Very awesome @cdeotte, not sure how to implement but curious if the session was treated as a corpus where each paring is a word and the cart and order events are treated as words in the sequence then a generator can just simulate the next 20 words which can include a cart or order event in the sequence, guessing they are not too accurate yet as they are randomized for creativity but that can also add some flavor for the buyers although in this simulation we also have to predict the behavior of current suggestion models. Back to your model, great work!
", "rawMarkdown": "Very awesome @cdeotte, not sure how to implement but curious if the session was treated as a corpus where each paring is a word and the cart and order events are treated as words in the sequence then a generator can just simulate the next 20 words which can include a cart or order event in the sequence, guessing they are not too accurate yet as they are randomized for creativity but that can also add some flavor for the buyers although in this simulation we also have to predict the behavior of current suggestion models. Back to your model, great work!", "votes": 1 }, { "id": 2026387, "postDate": "2022-11-12T03:02:50.447Z", "content": "Great kernel, as always @cdeotte.
", "rawMarkdown": "Great kernel, as always @cdeotte.\n\n", "votes": 1 }, { "id": 2030572, "postDate": "2022-11-15T14:23:26.880Z", "content": "How about the modin library it is bit faster in some areas
", "rawMarkdown": "How about the modin library it is bit faster in some areas", "votes": 2, "replies": [ { "id": 2030951, "postDate": "2022-11-15T18:29:55.410Z", "content": "Thanks @satyaprakashshukl for the suggestion. I was not aware of Modin library. I'll check it out. Another option to accelerate Pandas (groupby functions) on CPU is pandarallel here. I tried out pandarallel on my public notebook inference and it boosted inference offline Kaggle from 30 minutes to 5 minutes using 20 CPU! (discussion here)
Eventually i will rewrite my inference using RAPIDS cuDF and expect even more speed boost than pandarallel.
", "rawMarkdown": "Thanks @satyaprakashshukl for the suggestion. I was not aware of Modin library. I'll check it out. Another option to accelerate Pandas (groupby functions) on CPU is `pandarallel` [here][1]. I tried out `pandarallel` on my public notebook inference and it boosted inference offline Kaggle from 30 minutes to 5 minutes using 20 CPU! (discussion [here][2])\n\nEventually i will rewrite my inference using RAPIDS cuDF and expect even more speed boost than pandarallel.\n\n[1]: https://pypi.org/project/pandarallel/\n[2]: https://www.kaggle.com/code/cdeotte/candidate-rerank-model-lb-0-575/comments#2026011", "votes": 5 }, { "id": 2031356, "postDate": "2022-11-16T03:05:34.503Z", "content": "Thank you chris,i will explore too these option in my kernel ππ½π
", "rawMarkdown": "Thank you chris,i will explore too these option in my kernel ππ½π" }, { "id": 2032901, "postDate": "2022-11-16T21:27:09.273Z", "content": "that would be great chris! than nothing is stopping us from trying even more experiments!
", "rawMarkdown": "that would be great chris! than nothing is stopping us from trying even more experiments!" } ] }, { "id": 2025082, "postDate": "2022-11-11T01:00:44.913Z", "content": "Those speed gains are insane! π Didn't assume this workload could be transformed to run fast on the GPUβ¦ and yet here we are! π
\nHave to look more closely at how you are creating the co-visitation matrix dicts. That is some real magic happening there and something that has been an extremely slow part of the process thus far (when we went matrix[aid_x][aid_y] + w)!
Thanks Radek. To run many experiments, we need to increase the speed of our entire pipeline. Using 1xV100 32GB, we can compute co-visitation matrices in 2 minutes using that code. Now we need to speed up the test dataframe inference code. Then we can perform 100's of experiments quickly (in a few hours)!
", "rawMarkdown": "Thanks Radek. To run many experiments, we need to increase the speed of our entire pipeline. Using 1xV100 32GB, we can compute co-visitation matrices in 2 minutes using that code. Now we need to speed up the test dataframe inference code. Then we can perform 100's of experiments quickly (in a few hours)!", "votes": 3 }, { "id": 2025475, "postDate": "2022-11-11T08:41:59.943Z", "content": "That sounds really awesome, on a completely another level! π Wonder what hyperparameters you'll be able to find with this approach! π
\nI didn't have very high hopes cudf could do well here, but I was mistaken π GPUs strike again! Learned a lot from your code, some really awesome use of cumcount and sorting to emulate head and tail! And of course that trick where you are using series to replace the Counter/defaultdict combo! Really awesome stuff π
I have not tested it fully yet, still have a bit more code to change, but I wanted to operate on complete files instead of chunks. Was able to speed up calculating a single co-visitation matrix to 1 min 8 seconds π
\n
Here is the code in case it can be of use:
\n%%time\n\nchunk_size = 2_000_000\n\ntmp = None\nfor i in range(0, sessions.shape[0], chunk_size):\n df = data.loc[sessions[i]:sessions[min(sessions.shape[0]-1, i+chunk_size-1)]].reset_index()\n\n df = df.sort_values(['session','ts'],ascending=[True,False])\n\n df = df.reset_index(drop=True)\n df['n'] = df.groupby('session').cumcount()\n df = df.loc[df.n<30].drop('n',axis=1)\n df = df.merge(df,on='session')\n df = df.loc[ ((df.ts_x - df.ts_y).abs()< 24 * 60 * 60) & (df.aid_x != df.aid_y) ]\n\n # ASSIGN WEIGHTS\n df = df[['session', 'aid_x', 'aid_y','type_y']].drop_duplicates(['session', 'aid_x', 'aid_y'])\n df['wgt'] = df.type_y.map(type_weight)\n df = df[['aid_x','aid_y','wgt']]\n df.wgt = df.wgt.astype('float32')\n df = df.groupby(['aid_x','aid_y']).wgt.sum()\n\n if tmp is None: tmp = df\n else: tmp.add(df, fill_value=0)\n\n\n# CONVERT MATRIX TO DICTIONARY\ntmp = tmp.reset_index()\ntmp = tmp.sort_values(['aid_x','wgt'],ascending=[True,False])\n# SAVE TOP 40\ntmp = tmp.reset_index(drop=True)\ntmp['n'] = tmp.groupby('aid_x').aid_y.cumcount()\ntmp = tmp.loc[tmp.n<40].drop('n',axis=1)\n# SAVE TO DISK\ndf = tmp.to_pandas().groupby('aid_x').aid_y.apply(list)\nwith open(f'{PATH}/top_40_carts_orders_v{VER}.pkl', 'wb') as f:\n pickle.dump(df.to_dict(), f)\nI ran this on a Quadro with 48GB of RAM, but you lose only a couple of seconds if you make the chunk_size much smaller.
Anyhow, will only know if I didn't screw anything up once I change the rest of the notebook and run it all π
", "rawMarkdown": "That sounds really awesome, on a completely another level! π Wonder what hyperparameters you'll be able to find with this approach! π\n\nI didn't have very high hopes `cudf` could do well here, but I was mistaken π GPUs strike again! Learned a lot from your code, some really awesome use of `cumcount` and sorting to emulate `head` and `tail`! And of course that trick where you are using series to replace the Counter/defaultdict combo! Really awesome stuff π\n\nI have not tested it fully yet, still have a bit more code to change, but I wanted to operate on complete files instead of chunks. Was able to speed up calculating a single co-visitation matrix to 1 min 8 seconds π\n\n \n\nHere is the code in case it can be of use:\n\n```\n%%time\n\nchunk_size = 2_000_000\n\ntmp = None\nfor i in range(0, sessions.shape[0], chunk_size):\n df = data.loc[sessions[i]:sessions[min(sessions.shape[0]-1, i+chunk_size-1)]].reset_index()\n\n df = df.sort_values(['session','ts'],ascending=[True,False])\n\n df = df.reset_index(drop=True)\n df['n'] = df.groupby('session').cumcount()\n df = df.loc[df.n<30].drop('n',axis=1)\n df = df.merge(df,on='session')\n df = df.loc[ ((df.ts_x - df.ts_y).abs()< 24 * 60 * 60) & (df.aid_x != df.aid_y) ]\n\n # ASSIGN WEIGHTS\n df = df[['session', 'aid_x', 'aid_y','type_y']].drop_duplicates(['session', 'aid_x', 'aid_y'])\n df['wgt'] = df.type_y.map(type_weight)\n df = df[['aid_x','aid_y','wgt']]\n df.wgt = df.wgt.astype('float32')\n df = df.groupby(['aid_x','aid_y']).wgt.sum()\n \n if tmp is None: tmp = df\n else: tmp.add(df, fill_value=0)\n \n \n# CONVERT MATRIX TO DICTIONARY\ntmp = tmp.reset_index()\ntmp = tmp.sort_values(['aid_x','wgt'],ascending=[True,False])\n# SAVE TOP 40\ntmp = tmp.reset_index(drop=True)\ntmp['n'] = tmp.groupby('aid_x').aid_y.cumcount()\ntmp = tmp.loc[tmp.n<40].drop('n',axis=1)\n# SAVE TO DISK\ndf = tmp.to_pandas().groupby('aid_x').aid_y.apply(list)\nwith open(f'{PATH}/top_40_carts_orders_v{VER}.pkl', 'wb') as f:\n pickle.dump(df.to_dict(), f)\n```\n\nI ran this on a Quadro with 48GB of RAM, but you lose only a couple of seconds if you make the `chunk_size` much smaller.\n\nAnyhow, will only know if I didn't screw anything up once I change the rest of the notebook and run it all π", "votes": 1 }, { "id": 2025491, "postDate": "2022-11-11T08:49:26.860Z", "content": "π
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F83267%2F8e2f2a1ad7eaf8aafb31b8f3035f0bfd%2Ffast.png?generation=1668156556627991&alt=media\")
I just realized I am running this on my validation set, so that is nearly a quarter less of dataβ¦ that would also contribute to this being quite a bit faster π€¦ββοΈ
", "rawMarkdown": "I just realized I am running this on my validation set, so that is nearly a quarter less of data... that would also contribute to this being quite a bit faster π€¦ββοΈ" }, { "id": 2025901, "postDate": "2022-11-11T15:50:17.677Z", "content": "@Radek, i notice that your code removes the inner and outer loops. Using nested loops is for improved speed not memory management. First we pick a comfortable chunk size that the GPU can handle. Afterward say that we have 12 chunks (i.e. first level in diagram below). Instead of merging all the chunks sequentially, it is faster to merge the chunks into 3 larger pieces then merge those 3 pieces into our final dataframe. Only the first level is using groupby and processing, so the GPU can handle the larger second level pieces for the simple operation of merge. To maximize speed, you need to experiment with level 1 and level 2 sizes on your GPU:
\n![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Nov-2022/merge.png\")
wow, thank you very much for this, @cdeotte!!! I completely didn't realize this, that this reasoning would apply here. I thought it was all for memory management, but it all makes sense π
\nI have encountered something similar with DL models where beyond a certain batch size the computation is slower, time per epoch grows. But I have been wondering to what extent it is because of memory transfer issues vs how much work the GPU needs to do with bigger chunks. But here this is a clear example where operating on smaller pieces of data is faster! Amazing! π
\nThank you very much for your explanation π
", "rawMarkdown": "wow, thank you very much for this, @cdeotte!!! I completely didn't realize this, that this reasoning would apply here. I thought it was all for memory management, but it all makes sense π\n\nI have encountered something similar with DL models where beyond a certain batch size the computation is slower, time per epoch grows. But I have been wondering to what extent it is because of memory transfer issues vs how much work the GPU needs to do with bigger chunks. But here this is a clear example where operating on smaller pieces of data is faster! Amazing! π\n\nThank you very much for your explanation π" }, { "id": 2026343, "postDate": "2022-11-12T01:00:13.760Z", "content": "You two deserve more upvotes π
", "rawMarkdown": "You two deserve more upvotes π", "votes": 2 }, { "id": 2026350, "postDate": "2022-11-12T01:19:08.673Z", "content": "@radek1 I get an error that this method doesn't exist for DataFrames .to_pandas(), since tmp is already a DataFrame. Is that expected?
I didn't get any errors when running this code. Not at my computer ATM, can't check, but the df here in this code are cudf dataframes (not pandas) and IIRC tmp is a cudf series, maybe this can help
", "rawMarkdown": "I didn't get any errors when running this code. Not at my computer ATM, can't check, but the df here in this code are cudf dataframes (not pandas) and IIRC tmp is a cudf series, maybe this can help", "votes": 1 }, { "id": 2027353, "postDate": "2022-11-12T18:47:53.953Z", "content": "Radek, I have some questions about your code, if you don't mind helping me.
\nFor example, what is the variable \"sessions\"? The dataframe \"data\" is indexed by sessions? If now wouldn't some sessions divided in different chunks?
\nAnd the \"tmp.add\" code, this tmp has a fixed shape, so I don't understand how it is getting the information of all aid pairs.
\nThanks!
", "rawMarkdown": "Radek, I have some questions about your code, if you don't mind helping me.\n\nFor example, what is the variable \"sessions\"? The dataframe \"data\" is indexed by sessions? If now wouldn't some sessions divided in different chunks?\n\nAnd the \"tmp.add\" code, this tmp has a fixed shape, so I don't understand how it is getting the information of all aid pairs.\n\nThanks! " }, { "id": 2027537, "postDate": "2022-11-12T22:31:50.797Z", "content": "Hey @gabrielmoraesbarros!
\nI am including the rest of the code here, might be easiest for you to step through it and see how things work π
\nVER = 1\n\nimport pandas as pd, numpy as np\nfrom tqdm.notebook import tqdm\nimport os, sys, pickle, glob, gc\nfrom collections import Counter\nimport cudf, itertools\nprint('We will use RAPIDS version',cudf.__version__)\n\nPATH='valid'\n\ntrain = cudf.read_parquet(f'{PATH}/train.parquet')\ntest = cudf.read_parquet(f'{PATH}/test.parquet')\n\ndata = cudf.concat([train, test])\n\ntype_labels = {'clicks':0, 'carts':1, 'orders':2}\ntype_weight = {0:1, 1:6, 2:3}\n\ndata = data.set_index('session')\nsessions = data.index.unique()\n\nchunk_size = 2_000_000\n\ntmp = None\nfor i in range(0, sessions.shape[0], chunk_size):\n df = data.loc[sessions[i]:sessions[min(sessions.shape[0]-1, i+chunk_size-1)]].reset_index()\n\n df = df.sort_values(['session','ts'],ascending=[True,False])\n\n # USE TAIL OF SESSION\n df = df.reset_index(drop=True)\n df['n'] = df.groupby('session').cumcount()\n df = df.loc[df.n<30].drop('n',axis=1)\n\n # CREATE PAIRS\n df = df.merge(df,on='session')\n df = df.loc[ ((df.ts_x - df.ts_y).abs()< 24 * 60 * 60) & (df.aid_x != df.aid_y) ]\n\n # ASSIGN WEIGHTS\n df = df[['session', 'aid_x', 'aid_y','type_y']].drop_duplicates(['session', 'aid_x', 'aid_y'])\n df['wgt'] = df.type_y.map(type_weight)\n df = df[['aid_x','aid_y','wgt']]\n df.wgt = df.wgt.astype('float32')\n df = df.groupby(['aid_x','aid_y']).wgt.sum()\n\n if tmp is None: tmp = df\n else: tmp.add(df, fill_value=0)\n\n\n# CONVERT MATRIX TO DICTIONARY\ntmp = tmp.reset_index()\ntmp = tmp.sort_values(['aid_x','wgt'],ascending=[True,False])\n# SAVE TOP 40\ntmp = tmp.reset_index(drop=True)\ntmp['n'] = tmp.groupby('aid_x').aid_y.cumcount()\ntmp = tmp.loc[tmp.n<40].drop('n',axis=1)\n# SAVE TO DISK\ndf = tmp.to_pandas().groupby('aid_x').aid_y.apply(list)\nwith open(f'{PATH}/top_40_carts_orders_v{VER}.pkl', 'wb') as f:\n pickle.dump(df.to_dict(), f)\nThis is the unoptimized code, without the two loops from Chris's code. Meaning, it is not going to be as fast as it could be, but is possibly easier to reason about.
\nHope this helps! π
", "rawMarkdown": "Hey @gabrielmoraesbarros!\n\nI am including the rest of the code here, might be easiest for you to step through it and see how things work π\n\n```\nVER = 1\n\nimport pandas as pd, numpy as np\nfrom tqdm.notebook import tqdm\nimport os, sys, pickle, glob, gc\nfrom collections import Counter\nimport cudf, itertools\nprint('We will use RAPIDS version',cudf.__version__)\n\nPATH='valid'\n\ntrain = cudf.read_parquet(f'{PATH}/train.parquet')\ntest = cudf.read_parquet(f'{PATH}/test.parquet')\n\ndata = cudf.concat([train, test])\n\ntype_labels = {'clicks':0, 'carts':1, 'orders':2}\ntype_weight = {0:1, 1:6, 2:3}\n\ndata = data.set_index('session')\nsessions = data.index.unique()\n\nchunk_size = 2_000_000\n\ntmp = None\nfor i in range(0, sessions.shape[0], chunk_size):\n df = data.loc[sessions[i]:sessions[min(sessions.shape[0]-1, i+chunk_size-1)]].reset_index()\n\n df = df.sort_values(['session','ts'],ascending=[True,False])\n \n # USE TAIL OF SESSION\n df = df.reset_index(drop=True)\n df['n'] = df.groupby('session').cumcount()\n df = df.loc[df.n<30].drop('n',axis=1)\n \n # CREATE PAIRS\n df = df.merge(df,on='session')\n df = df.loc[ ((df.ts_x - df.ts_y).abs()< 24 * 60 * 60) & (df.aid_x != df.aid_y) ]\n\n # ASSIGN WEIGHTS\n df = df[['session', 'aid_x', 'aid_y','type_y']].drop_duplicates(['session', 'aid_x', 'aid_y'])\n df['wgt'] = df.type_y.map(type_weight)\n df = df[['aid_x','aid_y','wgt']]\n df.wgt = df.wgt.astype('float32')\n df = df.groupby(['aid_x','aid_y']).wgt.sum()\n \n if tmp is None: tmp = df\n else: tmp.add(df, fill_value=0)\n \n \n# CONVERT MATRIX TO DICTIONARY\ntmp = tmp.reset_index()\ntmp = tmp.sort_values(['aid_x','wgt'],ascending=[True,False])\n# SAVE TOP 40\ntmp = tmp.reset_index(drop=True)\ntmp['n'] = tmp.groupby('aid_x').aid_y.cumcount()\ntmp = tmp.loc[tmp.n<40].drop('n',axis=1)\n# SAVE TO DISK\ndf = tmp.to_pandas().groupby('aid_x').aid_y.apply(list)\nwith open(f'{PATH}/top_40_carts_orders_v{VER}.pkl', 'wb') as f:\n pickle.dump(df.to_dict(), f)\n```\n\nThis is the unoptimized code, without the two loops from Chris's code. Meaning, it is not going to be as fast as it could be, but is possibly easier to reason about.\n\nHope this helps! π" }, { "id": 2027568, "postDate": "2022-11-13T01:12:08.027Z", "content": "Thanks, man, appreciated.
\nThe \"sessions\" part was exactly what I was thinking, but I still suspicious about the \"tmp.add\" thing.
\nHere is some snippet with your parquets otto-full-optimized-memory-footprint.
\nVER = 1\n\nimport pandas as pd, numpy as np\nfrom tqdm.notebook import tqdm\nimport os, sys, pickle, glob, gc\nfrom collections import Counter\nimport cudf, itertools\nprint('We will use RAPIDS version',cudf.__version__)\n\nPATH='valid'\n\n# train = cudf.read_parquet(f'/kaggle/input/train.parquet')\ntest = cudf.read_parquet(f'/kaggle/input/otto-full-optimized-memory-footprint/test.parquet')\n\n# data = cudf.concat([train, test])\ndata = test\n\n\ntype_labels = {'clicks':0, 'carts':1, 'orders':2}\ntype_weight = {0:1, 1:6, 2:3}\n\ndata = data.set_index('session')\nsessions = data.index.unique()\n\nchunk_size = 10_000\n\nprint(data.shape)\n\ntmp = None\nlist_of_dataframes = []\nfor it, i in enumerate(range(0, sessions.shape[0], chunk_size)):\n df = data.loc[sessions[i]:sessions[min(sessions.shape[0]-1, i+chunk_size-1)]].reset_index()\n\n df = df.sort_values(['session','ts'],ascending=[True,False])\n\n # USE TAIL OF SESSION\n df = df.reset_index(drop=True)\n df['n'] = df.groupby('session').cumcount()\n df = df.loc[df.n<30].drop('n',axis=1)\n\n # CREATE PAIRS\n df = df.merge(df,on='session')\n df = df.loc[ ((df.ts_x - df.ts_y).abs()< 24 * 60 * 60) & (df.aid_x != df.aid_y) ]\n\n # ASSIGN WEIGHTS\n df = df[['session', 'aid_x', 'aid_y','type_y']].drop_duplicates(['session', 'aid_x', 'aid_y'])\n df['wgt'] = df.type_y.map(type_weight)\n df = df[['aid_x','aid_y','wgt']]\n df.wgt = df.wgt.astype('float32')\n df = df.groupby(['aid_x','aid_y']).wgt.sum()\n\n if tmp is None: tmp = df\n else: tmp.add(df, fill_value=0)\n list_of_dataframes.append(df)\n print('Iteration = {1}, tmp shape = {1} and pd.concat shape {2}'.format(it,tmp.shape,cudf.concat(list_of_dataframes, axis=0).shape))\n print()\n if it > 10:\n break\nThe command tmp.add() uses both dataframe indexes to perform the add (and the column being added is wgt). The previous command is df = df.groupby(['aid_x','aid_y']).wgt.sum() this makes the index a multilevel index with ['aid_x','aid_y']. So when we tmp.add() it adds to the weights of existing pairs of aid_x and aid_y. And if the pair does not exist, it makes a new pair.
Now I understand my confusion:
\n if tmp is None:\n tmp = df\nelse:\n tmp.add(df, fill_value=0)\nThe last line should be:
\ntmp = tmp.add(df, fill_value=0)\nWhich is equivalent to:
\ntmp = (tmp + df).fillna(0)
Thanks for the replies.
", "rawMarkdown": "@cdeotte and @radek1 \n\nNow I understand my confusion:\n\n```\n if tmp is None:\n tmp = df\nelse:\n tmp.add(df, fill_value=0)\n```\n\nThe last line should be: \n\n```\ntmp = tmp.add(df, fill_value=0)\n```\n\nWhich is equivalent to:\n`tmp = (tmp + df).fillna(0)`\n\nThanks for the replies.", "votes": 3 }, { "id": 2028426, "postDate": "2022-11-13T22:39:51.030Z", "content": "Ah so there is a bug! Well spotted @gabrielmoraesbarros! π
", "rawMarkdown": "Ah so there is a bug! Well spotted @gabrielmoraesbarros! π", "votes": 1 } ] }, { "id": 2056638, "postDate": "2022-12-06T10:17:03.190Z", "content": "Hey Chris!
\nHave you ever encountered this error: MemoryError: std::bad_alloc: CUDA error at: /opt/conda/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory?
\nWhat would you do then?
\nThanks. Cheers!
", "rawMarkdown": "Hey Chris!\n\nHave you ever encountered this error: MemoryError: std::bad_alloc: CUDA error at: /opt/conda/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory?\n\nWhat would you do then?\n\nThanks. Cheers!", "replies": [ { "id": 2056813, "postDate": "2022-12-06T13:05:46.847Z", "content": "In this case, increase the variable DISK_PIECES. This will break the processing into more chunks with each chunk using less memory. And avoid memory error.
Just start learning RAPIDS, thanks!
", "rawMarkdown": "Just start learning RAPIDS, thanks!", "votes": 1 } ], "comments": [ { "id": 2028397, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-11-13T21:32:00.943000", "content": "UPDATE: I updated my code to create co-visitation matrices 1.5x faster. It now creates co-visitation matrices in Kaggle notebooks at 6 minutes! (And offline using 1xV100 in 1.5 minutes!)
", "votes": 7, "replies": [ { "id": 2032873, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-11-16T21:01:18.247000", "content": "UPDATE: π₯ I updated my code again to create co-visitation matrices 2x faster. It now creates co-visitation matrices in Kaggle notebooks at 3 minutes! (This is 3x faster than version 1 of the notebook. And 30x faster than public notebooks using Pandas CPU) π₯
", "votes": 4, "replies": [] } ] }, { "id": 2035237, "author_name": "Andrej ZubaΔΎ", "author_url": "", "post_date": "2022-11-18T17:59:08.493000", "content": "Hi Chris,
\nThanks for showing us this wonderful solution π.
\nI find it hard to improve on this simple heuristic approach combining information on IN_SESSION flag, TOP most popular items for given item in the session and TOP most popular items overall.
\nI was wondering, how did you come up with this heuristic? Is it something that you tried out before and it worked, or you tried out bunch of different techniques and this one worked the best or simply you had a lucky/intuitive guess?
\nBest
\nAndrej
", "votes": 1, "replies": [ { "id": 2036565, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-11-19T22:22:38.557000", "content": "There are two ways to improve CV and LB. Find more candidates and find better rerank logic. I discover both by using intuition and thinking about how users behave. Then i evaluate my ideas by observing if local validation score increases or not.
\nThe easiest way to improve my public notebook is to create more co-visitation matrices. It is possible to improve my notebook by changing the rerank logic, but it will be easier to just make more co-visitation matrices and add their weight in.
\nI suggest that you focus on the case of suggest_buys for len(unique_aids)<20. This has the largest effect on CV score and LB score. Change the code to allow for weights. The current code is
# USE \"CART ORDER\" CO-VISITATION MATRIX\naids2 = list(itertools.chain(*[top_20_buys[aid] for aid in unique_aids if aid in top_20_buys]))\n# USE \"BUY2BUY\" CO-VISITATION MATRIX\naids3 = list(itertools.chain(*[top_20_buy2buy[aid] for aid in unique_buys if aid in top_20_buy2buy]))\n# RERANK CANDIDATES\ntop_aids2 = [aid2 for aid2, cnt in Counter(aids2+aids3).most_common(20) if aid2 not in unique_aids] \nChange this to
\nfrom collections import Counter\naids_temp = Counter() \n# USE \"CART ORDER\" CO-VISITATION MATRIX\naids2 = list(itertools.chain(*[top_20_buys[aid] for aid in unique_aids if aid in top_20_buys]))\nfor aid in aids2: aids_temp[aid] += 1\n# USE \"BUY2BUY\" CO-VISITATION MATRIX\naids3 = list(itertools.chain(*[top_20_buy2buy[aid] for aid in unique_buys if aid in top_20_buy2buy]))\nfor aid in aids3: aids_temp[aid] += 1\n# RERANK CANDIDATES\ntop_aids2 = [k for k,v in aids_temp.most_common(20) if k not in unique_aids]\nNow we can create new co-visitation matrices and include them with
\naids4 = list(itertools.chain(*[NEW_MATRIX[aid] for aid in SUBSET_OF_USER_HISTORY_ITEMS if aid in NEW_MATRIX]))\nfor aid in aids4: aids_temp[aid] += 1\nYou need to be creative when making co-visitation matrices. The matrices makes pairs of items. If a certain type of user interacts in a certain type of way with item. Then we predict they will click and order a certain item in the future. Then when you include this new co-visitation matrix in your inference, we only apply it to the same certain type of user when they interact with an item in a certain type of way.
\nAlso note that once you change the code to include for aid in aids2: aids_temp[aid] += 1, you can now assign different situations different amount of weights during inference. This is how we change the rerank logic.
weights = LOGIC_TO_MAKES_WEIGHTS\nfor aid,w in zip(aids2,weights):\n aids_temp[aid] += w\nthis is so valuable and lets us experiment with different weighting approaches in an acceptable time! tnx chris!
", "votes": 1, "replies": [] }, { "id": 2027772, "author_name": "jazivxt", "author_url": "", "post_date": "2022-11-13T07:20:54.380000", "content": "Very awesome @cdeotte, not sure how to implement but curious if the session was treated as a corpus where each paring is a word and the cart and order events are treated as words in the sequence then a generator can just simulate the next 20 words which can include a cart or order event in the sequence, guessing they are not too accurate yet as they are randomized for creativity but that can also add some flavor for the buyers although in this simulation we also have to predict the behavior of current suggestion models. Back to your model, great work!
", "votes": 1, "replies": [] }, { "id": 2026387, "author_name": "GabrielMoraesBarros", "author_url": "", "post_date": "2022-11-12T03:02:50.447000", "content": "Great kernel, as always @cdeotte.
", "votes": 1, "replies": [] }, { "id": 2030572, "author_name": "Satya", "author_url": "", "post_date": "2022-11-15T14:23:26.880000", "content": "How about the modin library it is bit faster in some areas
", "votes": 2, "replies": [ { "id": 2030951, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-11-15T18:29:55.410000", "content": "Thanks @satyaprakashshukl for the suggestion. I was not aware of Modin library. I'll check it out. Another option to accelerate Pandas (groupby functions) on CPU is pandarallel here. I tried out pandarallel on my public notebook inference and it boosted inference offline Kaggle from 30 minutes to 5 minutes using 20 CPU! (discussion here)
Eventually i will rewrite my inference using RAPIDS cuDF and expect even more speed boost than pandarallel.
", "votes": 5, "replies": [] }, { "id": 2031356, "author_name": "Satya", "author_url": "", "post_date": "2022-11-16T03:05:34.503000", "content": "Thank you chris,i will explore too these option in my kernel ππ½π
", "votes": 0, "replies": [] }, { "id": 2032901, "author_name": "Simon Veitner", "author_url": "", "post_date": "2022-11-16T21:27:09.273000", "content": "that would be great chris! than nothing is stopping us from trying even more experiments!
", "votes": 0, "replies": [] } ] }, { "id": 2025082, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-11-11T01:00:44.913000", "content": "Those speed gains are insane! π Didn't assume this workload could be transformed to run fast on the GPUβ¦ and yet here we are! π
\nHave to look more closely at how you are creating the co-visitation matrix dicts. That is some real magic happening there and something that has been an extremely slow part of the process thus far (when we went matrix[aid_x][aid_y] + w)!
Thanks Radek. To run many experiments, we need to increase the speed of our entire pipeline. Using 1xV100 32GB, we can compute co-visitation matrices in 2 minutes using that code. Now we need to speed up the test dataframe inference code. Then we can perform 100's of experiments quickly (in a few hours)!
", "votes": 3, "replies": [] }, { "id": 2025475, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-11-11T08:41:59.943000", "content": "That sounds really awesome, on a completely another level! π Wonder what hyperparameters you'll be able to find with this approach! π
\nI didn't have very high hopes cudf could do well here, but I was mistaken π GPUs strike again! Learned a lot from your code, some really awesome use of cumcount and sorting to emulate head and tail! And of course that trick where you are using series to replace the Counter/defaultdict combo! Really awesome stuff π
I have not tested it fully yet, still have a bit more code to change, but I wanted to operate on complete files instead of chunks. Was able to speed up calculating a single co-visitation matrix to 1 min 8 seconds π
\n
Here is the code in case it can be of use:
\n%%time\n\nchunk_size = 2_000_000\n\ntmp = None\nfor i in range(0, sessions.shape[0], chunk_size):\n df = data.loc[sessions[i]:sessions[min(sessions.shape[0]-1, i+chunk_size-1)]].reset_index()\n\n df = df.sort_values(['session','ts'],ascending=[True,False])\n\n df = df.reset_index(drop=True)\n df['n'] = df.groupby('session').cumcount()\n df = df.loc[df.n<30].drop('n',axis=1)\n df = df.merge(df,on='session')\n df = df.loc[ ((df.ts_x - df.ts_y).abs()< 24 * 60 * 60) & (df.aid_x != df.aid_y) ]\n\n # ASSIGN WEIGHTS\n df = df[['session', 'aid_x', 'aid_y','type_y']].drop_duplicates(['session', 'aid_x', 'aid_y'])\n df['wgt'] = df.type_y.map(type_weight)\n df = df[['aid_x','aid_y','wgt']]\n df.wgt = df.wgt.astype('float32')\n df = df.groupby(['aid_x','aid_y']).wgt.sum()\n\n if tmp is None: tmp = df\n else: tmp.add(df, fill_value=0)\n\n\n# CONVERT MATRIX TO DICTIONARY\ntmp = tmp.reset_index()\ntmp = tmp.sort_values(['aid_x','wgt'],ascending=[True,False])\n# SAVE TOP 40\ntmp = tmp.reset_index(drop=True)\ntmp['n'] = tmp.groupby('aid_x').aid_y.cumcount()\ntmp = tmp.loc[tmp.n<40].drop('n',axis=1)\n# SAVE TO DISK\ndf = tmp.to_pandas().groupby('aid_x').aid_y.apply(list)\nwith open(f'{PATH}/top_40_carts_orders_v{VER}.pkl', 'wb') as f:\n pickle.dump(df.to_dict(), f)\nI ran this on a Quadro with 48GB of RAM, but you lose only a couple of seconds if you make the chunk_size much smaller.
Anyhow, will only know if I didn't screw anything up once I change the rest of the notebook and run it all π
", "votes": 1, "replies": [] }, { "id": 2025491, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-11-11T08:49:26.860000", "content": "π
\n
I just realized I am running this on my validation set, so that is nearly a quarter less of dataβ¦ that would also contribute to this being quite a bit faster π€¦ββοΈ
", "votes": 0, "replies": [] }, { "id": 2025901, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-11-11T15:50:17.677000", "content": "@Radek, i notice that your code removes the inner and outer loops. Using nested loops is for improved speed not memory management. First we pick a comfortable chunk size that the GPU can handle. Afterward say that we have 12 chunks (i.e. first level in diagram below). Instead of merging all the chunks sequentially, it is faster to merge the chunks into 3 larger pieces then merge those 3 pieces into our final dataframe. Only the first level is using groupby and processing, so the GPU can handle the larger second level pieces for the simple operation of merge. To maximize speed, you need to experiment with level 1 and level 2 sizes on your GPU:
\n
wow, thank you very much for this, @cdeotte!!! I completely didn't realize this, that this reasoning would apply here. I thought it was all for memory management, but it all makes sense π
\nI have encountered something similar with DL models where beyond a certain batch size the computation is slower, time per epoch grows. But I have been wondering to what extent it is because of memory transfer issues vs how much work the GPU needs to do with bigger chunks. But here this is a clear example where operating on smaller pieces of data is faster! Amazing! π
\nThank you very much for your explanation π
", "votes": 0, "replies": [] }, { "id": 2026343, "author_name": "dpalbrecht", "author_url": "", "post_date": "2022-11-12T01:00:13.760000", "content": "You two deserve more upvotes π
", "votes": 2, "replies": [] }, { "id": 2026350, "author_name": "dpalbrecht", "author_url": "", "post_date": "2022-11-12T01:19:08.673000", "content": "@radek1 I get an error that this method doesn't exist for DataFrames .to_pandas(), since tmp is already a DataFrame. Is that expected?
I didn't get any errors when running this code. Not at my computer ATM, can't check, but the df here in this code are cudf dataframes (not pandas) and IIRC tmp is a cudf series, maybe this can help
", "votes": 1, "replies": [] }, { "id": 2027353, "author_name": "GabrielMoraesBarros", "author_url": "", "post_date": "2022-11-12T18:47:53.953000", "content": "Radek, I have some questions about your code, if you don't mind helping me.
\nFor example, what is the variable \"sessions\"? The dataframe \"data\" is indexed by sessions? If now wouldn't some sessions divided in different chunks?
\nAnd the \"tmp.add\" code, this tmp has a fixed shape, so I don't understand how it is getting the information of all aid pairs.
\nThanks!
", "votes": 0, "replies": [] }, { "id": 2027537, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-11-12T22:31:50.797000", "content": "Hey @gabrielmoraesbarros!
\nI am including the rest of the code here, might be easiest for you to step through it and see how things work π
\nVER = 1\n\nimport pandas as pd, numpy as np\nfrom tqdm.notebook import tqdm\nimport os, sys, pickle, glob, gc\nfrom collections import Counter\nimport cudf, itertools\nprint('We will use RAPIDS version',cudf.__version__)\n\nPATH='valid'\n\ntrain = cudf.read_parquet(f'{PATH}/train.parquet')\ntest = cudf.read_parquet(f'{PATH}/test.parquet')\n\ndata = cudf.concat([train, test])\n\ntype_labels = {'clicks':0, 'carts':1, 'orders':2}\ntype_weight = {0:1, 1:6, 2:3}\n\ndata = data.set_index('session')\nsessions = data.index.unique()\n\nchunk_size = 2_000_000\n\ntmp = None\nfor i in range(0, sessions.shape[0], chunk_size):\n df = data.loc[sessions[i]:sessions[min(sessions.shape[0]-1, i+chunk_size-1)]].reset_index()\n\n df = df.sort_values(['session','ts'],ascending=[True,False])\n\n # USE TAIL OF SESSION\n df = df.reset_index(drop=True)\n df['n'] = df.groupby('session').cumcount()\n df = df.loc[df.n<30].drop('n',axis=1)\n\n # CREATE PAIRS\n df = df.merge(df,on='session')\n df = df.loc[ ((df.ts_x - df.ts_y).abs()< 24 * 60 * 60) & (df.aid_x != df.aid_y) ]\n\n # ASSIGN WEIGHTS\n df = df[['session', 'aid_x', 'aid_y','type_y']].drop_duplicates(['session', 'aid_x', 'aid_y'])\n df['wgt'] = df.type_y.map(type_weight)\n df = df[['aid_x','aid_y','wgt']]\n df.wgt = df.wgt.astype('float32')\n df = df.groupby(['aid_x','aid_y']).wgt.sum()\n\n if tmp is None: tmp = df\n else: tmp.add(df, fill_value=0)\n\n\n# CONVERT MATRIX TO DICTIONARY\ntmp = tmp.reset_index()\ntmp = tmp.sort_values(['aid_x','wgt'],ascending=[True,False])\n# SAVE TOP 40\ntmp = tmp.reset_index(drop=True)\ntmp['n'] = tmp.groupby('aid_x').aid_y.cumcount()\ntmp = tmp.loc[tmp.n<40].drop('n',axis=1)\n# SAVE TO DISK\ndf = tmp.to_pandas().groupby('aid_x').aid_y.apply(list)\nwith open(f'{PATH}/top_40_carts_orders_v{VER}.pkl', 'wb') as f:\n pickle.dump(df.to_dict(), f)\nThis is the unoptimized code, without the two loops from Chris's code. Meaning, it is not going to be as fast as it could be, but is possibly easier to reason about.
\nHope this helps! π
", "votes": 0, "replies": [] }, { "id": 2027568, "author_name": "GabrielMoraesBarros", "author_url": "", "post_date": "2022-11-13T01:12:08.027000", "content": "Thanks, man, appreciated.
\nThe \"sessions\" part was exactly what I was thinking, but I still suspicious about the \"tmp.add\" thing.
\nHere is some snippet with your parquets otto-full-optimized-memory-footprint.
\nVER = 1\n\nimport pandas as pd, numpy as np\nfrom tqdm.notebook import tqdm\nimport os, sys, pickle, glob, gc\nfrom collections import Counter\nimport cudf, itertools\nprint('We will use RAPIDS version',cudf.__version__)\n\nPATH='valid'\n\n# train = cudf.read_parquet(f'/kaggle/input/train.parquet')\ntest = cudf.read_parquet(f'/kaggle/input/otto-full-optimized-memory-footprint/test.parquet')\n\n# data = cudf.concat([train, test])\ndata = test\n\n\ntype_labels = {'clicks':0, 'carts':1, 'orders':2}\ntype_weight = {0:1, 1:6, 2:3}\n\ndata = data.set_index('session')\nsessions = data.index.unique()\n\nchunk_size = 10_000\n\nprint(data.shape)\n\ntmp = None\nlist_of_dataframes = []\nfor it, i in enumerate(range(0, sessions.shape[0], chunk_size)):\n df = data.loc[sessions[i]:sessions[min(sessions.shape[0]-1, i+chunk_size-1)]].reset_index()\n\n df = df.sort_values(['session','ts'],ascending=[True,False])\n\n # USE TAIL OF SESSION\n df = df.reset_index(drop=True)\n df['n'] = df.groupby('session').cumcount()\n df = df.loc[df.n<30].drop('n',axis=1)\n\n # CREATE PAIRS\n df = df.merge(df,on='session')\n df = df.loc[ ((df.ts_x - df.ts_y).abs()< 24 * 60 * 60) & (df.aid_x != df.aid_y) ]\n\n # ASSIGN WEIGHTS\n df = df[['session', 'aid_x', 'aid_y','type_y']].drop_duplicates(['session', 'aid_x', 'aid_y'])\n df['wgt'] = df.type_y.map(type_weight)\n df = df[['aid_x','aid_y','wgt']]\n df.wgt = df.wgt.astype('float32')\n df = df.groupby(['aid_x','aid_y']).wgt.sum()\n\n if tmp is None: tmp = df\n else: tmp.add(df, fill_value=0)\n list_of_dataframes.append(df)\n print('Iteration = {1}, tmp shape = {1} and pd.concat shape {2}'.format(it,tmp.shape,cudf.concat(list_of_dataframes, axis=0).shape))\n print()\n if it > 10:\n break\nThe command tmp.add() uses both dataframe indexes to perform the add (and the column being added is wgt). The previous command is df = df.groupby(['aid_x','aid_y']).wgt.sum() this makes the index a multilevel index with ['aid_x','aid_y']. So when we tmp.add() it adds to the weights of existing pairs of aid_x and aid_y. And if the pair does not exist, it makes a new pair.
Now I understand my confusion:
\n if tmp is None:\n tmp = df\nelse:\n tmp.add(df, fill_value=0)\nThe last line should be:
\ntmp = tmp.add(df, fill_value=0)\nWhich is equivalent to:
\ntmp = (tmp + df).fillna(0)
Thanks for the replies.
", "votes": 3, "replies": [] }, { "id": 2028426, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-11-13T22:39:51.030000", "content": "Ah so there is a bug! Well spotted @gabrielmoraesbarros! π
", "votes": 1, "replies": [] } ] }, { "id": 2056638, "author_name": "enizzzz", "author_url": "", "post_date": "2022-12-06T10:17:03.190000", "content": "Hey Chris!
\nHave you ever encountered this error: MemoryError: std::bad_alloc: CUDA error at: /opt/conda/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory?
\nWhat would you do then?
\nThanks. Cheers!
", "votes": 0, "replies": [ { "id": 2056813, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-12-06T13:05:46.847000", "content": "In this case, increase the variable DISK_PIECES. This will break the processing into more chunks with each chunk using less memory. And avoid memory error.
Just start learning RAPIDS, thanks!
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "2025071": "I published a notebook [here][1] that uses RAPIDS cuDF to compute 30x faster co-visitation matrices. Using co-visitation matrices helps provide our models with \"candidates\". Then our models (and/or human logic) and \"rerank\" these \"candidates\" and select our final 20 predictions for submission CSV. \n\n**UPDATE** π₯ latest version computes co-visitation matrices in **3min** each π₯\n\nFor more information about \"candidate rerank\" models, see Ravi's discussion [here][2]\n\n[1]: https://www.kaggle.com/code/cdeotte/candidate-rerank-model-lb-0-573\n[2]: https://www.kaggle.com/competitions/otto-recommender-system/discussion/364721", "2028397": "UPDATE: I updated my code to create co-visitation matrices 1.5x faster. It now creates co-visitation matrices in Kaggle notebooks at 6 minutes! (And offline using 1xV100 in 1.5 minutes!)", "2035237": "Hi Chris,\n\nThanks for showing us this wonderful solution π. \n\nI find it hard to improve on this simple heuristic approach combining information on IN_SESSION flag, TOP most popular items for given item in the session and TOP most popular items overall.\n\nI was wondering, how did you come up with this heuristic? Is it something that you tried out before and it worked, or you tried out bunch of different techniques and this one worked the best or simply you had a lucky/intuitive guess?\n\nBest \n\nAndrej", "2032899": "this is so valuable and lets us experiment with different weighting approaches in an acceptable time! tnx chris!", "2027772": "Very awesome @cdeotte, not sure how to implement but curious if the session was treated as a corpus where each paring is a word and the cart and order events are treated as words in the sequence then a generator can just simulate the next 20 words which can include a cart or order event in the sequence, guessing they are not too accurate yet as they are randomized for creativity but that can also add some flavor for the buyers although in this simulation we also have to predict the behavior of current suggestion models. Back to your model, great work!", "2026387": "Great kernel, as always @cdeotte.\n\n", "2030572": "How about the modin library it is bit faster in some areas", "2025082": "Those speed gains are insane! π Didn't assume this workload could be transformed to run fast on the GPU... and yet here we are! π \n\nHave to look more closely at how you are creating the co-visitation matrix dicts. That is some real magic happening there and something that has been an extremely slow part of the process thus far (when we went `matrix[aid_x][aid_y] + w`)!", "2056638": "Hey Chris!\n\nHave you ever encountered this error: MemoryError: std::bad_alloc: CUDA error at: /opt/conda/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory?\n\nWhat would you do then?\n\nThanks. Cheers!", "2031150": "Just start learning RAPIDS, thanks!" } }