![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F83267%2F70e2a7f1b9621cd0afb2b815a7b3e01b%2Fyou_shall_know_a_word.png?generation=1668122115555809&alt=media\")
It is very interesting to think of modern techniques in the context of their roots.
\nFor instance, when thinking about RNNs we should consider unigram, bigram, and trigram models.
\nWhat were they?
\nThey estimated the probability of a word given the words that came before.
\nWe see \"Radek is a _\".
\nIn a trigram model, we would consider the chain \"Radek\", \"is\", and \"a\" and could find the most likely word to come next.
\nThe easiest way would be to count the occurrences of \"Radek is a\" and the words that came after and pick the one most common.
\nSo why RNNs are such a great improvement?
\nWell, there are not that many \"Radek is a _\" in most text corpora!
\nWith RNNs (or word2vec) we could operate on embeddings and look at many more words in sequence.
\nInstead of looking at only \"Radek is a \" we can use \"Radek was an _\", \"Tommy is a __\" as examples to learn from, etc.
\nEmbeddings take values of a variable of high cardinality and project them to a representation that ideally captures similarities.
\n
So how does this relate to the co-visitation matrix?
\nA co-visitation matrix counts the co-occurrence of two actions in close proximity.
\nIf a user bought A and shortly after bought B, we store these values together.
\nWe calculate counts and use them to estimate the probability of future actions based on recent history.
\nIt is quite important to understand what is happening in the co-visitation matrix approachβ¦
\nSince it suffers from the same issues as our trigram example!
\nPlus what does the co-visitation matrix resemble?
\nYou are right, it is akin to doing Matrix Factorization by counting!
\nIt is really fun that this competition exposed this heuristic (the co-visitation matrix) that I have not been aware of before! π
\nIt is very interesting to think of modern techniques in the context of their roots.
\nFor instance, when thinking about RNNs we should consider unigram, bigram, and trigram models.
\nWhat were they?
\nThey estimated the probability of a word given the words that came before.
\nWe see \"Radek is a _\".
\nIn a trigram model, we would consider the chain \"Radek\", \"is\", and \"a\" and could find the most likely word to come next.
\nThe easiest way would be to count the occurrences of \"Radek is a\" and the words that came after and pick the one most common.
\nSo why RNNs are such a great improvement?
\nWell, there are not that many \"Radek is a _\" in most text corpora!
\nWith RNNs (or word2vec) we could operate on embeddings and look at many more words in sequence.
\nInstead of looking at only \"Radek is a \" we can use \"Radek was an _\", \"Tommy is a __\" as examples to learn from, etc.
\nEmbeddings take values of a variable of high cardinality and project them to a representation that ideally captures similarities.
\n
So how does this relate to the co-visitation matrix?
\nA co-visitation matrix counts the co-occurrence of two actions in close proximity.
\nIf a user bought A and shortly after bought B, we store these values together.
\nWe calculate counts and use them to estimate the probability of future actions based on recent history.
\nIt is quite important to understand what is happening in the co-visitation matrix approachβ¦
\nSince it suffers from the same issues as our trigram example!
\nPlus what does the co-visitation matrix resemble?
\nYou are right, it is akin to doing Matrix Factorization by counting!
\nIt is really fun that this competition exposed this heuristic (the co-visitation matrix) that I have not been aware of before! π
\nNice explanationβ¦.Really helpfulβ¦Thanksπ
", "rawMarkdown": "Nice explanation....Really helpful...Thanksπ", "votes": null }, { "id": "2025025", "postDate": "11/11/2022 00:03:24", "content": "How funny that I was playing with the Word2Vec and its variants and then saw this post :D
", "rawMarkdown": "How funny that I was playing with the Word2Vec and its variants and then saw this post :D", "votes": null }, { "id": "2025036", "postDate": "11/11/2022 00:15:38", "content": "Yeah, it would be really interesting to see what word2vec can do in this competition π Might be it could be quite nice for candidate generation or some sort of similarity scoring for reranking π€
\nAnyhow, cool that we were both circulating around the same topic! π
", "rawMarkdown": "Yeah, it would be really interesting to see what word2vec can do in this competition π Might be it could be quite nice for candidate generation or some sort of similarity scoring for reranking π€\n\nAnyhow, cool that we were both circulating around the same topic! π", "votes": null }, { "id": "2025045", "postDate": "11/11/2022 00:27:43", "content": "Co-visitation matrices are the natural statistical approach. It is a way to compute conditional probabilities. Given a user has interacted with item A, it computes the most likely future items that this user will interact with.
\nAnd we can make many more specific variants such as given a user has ordered item A, what is the most likely item that this user will click, cart, order etc. Or given a user has visited this item in the morning, what is the most likely item that this user will order. Etc etc.
\nOf course if we train a reranker model it will learn all these varieties of conditional probabilities on its own.
", "rawMarkdown": "Co-visitation matrices are the natural statistical approach. It is a way to compute conditional probabilities. Given a user has interacted with item A, it computes the most likely future items that this user will interact with.\n\nAnd we can make many more specific variants such as given a user has **ordered** item A, what is the most likely item that this user will **click, cart, order** etc. Or given a user has visited this item **in the morning**, what is the most likely item that this user will **order**. Etc etc.\n\nOf course if we train a reranker model it will learn all these varieties of conditional probabilities on its own.", "votes": null }, { "id": "2025050", "postDate": "11/11/2022 00:32:37", "content": "Training Word2Vec would be very helpful. That would produce embeddings such that similar items (with co-visitation) would have distance similar embeddings. It would help us understand what these anonymous item ids represent.
\nThen we could do lots of EDA like how do the items that users purchase over time change? How do items purchased in the morning differ from items purchased in the evening. Etc etc. For each of these analysis, we could plot 2D pictures with clusters of dots (using their Word2Vec embedding 2D projection)
", "rawMarkdown": "Training Word2Vec would be very helpful. That would produce embeddings such that similar items (with co-visitation) would have distance similar embeddings. It would help us understand what these anonymous item ids represent.\n\nThen we could do lots of EDA like how do the items that users purchase over time change? How do items purchased in the morning differ from items purchased in the evening. Etc etc. For each of these analysis, we could plot 2D pictures with clusters of dots (using their Word2Vec embedding 2D projection)", "votes": null }, { "id": "2025058", "postDate": "11/11/2022 00:41:37", "content": "Well explained. Thanks for sharingπ
", "rawMarkdown": "Well explained. Thanks for sharingπ", "votes": null }, { "id": "2025072", "postDate": "11/11/2022 00:49:31", "content": "my pleasure, @oscarm524! very glad you found this useful! π
", "rawMarkdown": "my pleasure, @oscarm524! very glad you found this useful! π", "votes": null }, { "id": "2025074", "postDate": "11/11/2022 00:51:14", "content": "That is some amazing information @cdeotte, as always! π Thank you so very much for sharing all this with us π
", "rawMarkdown": "That is some amazing information @cdeotte, as always! π Thank you so very much for sharing all this with us π", "votes": null }, { "id": "2025075", "postDate": "11/11/2022 00:51:40", "content": "Yes, exactly! I was also thinking of it for the candidate generation. π€π€
", "rawMarkdown": "Yes, exactly! I was also thinking of it for the candidate generation. π€π€", "votes": null }, { "id": "2025093", "postDate": "11/11/2022 01:08:02", "content": "Word2Vec would also be great for EDA. It (with 2D projection via UMAP or TSNE) would help us visualize these anonymous items.
", "rawMarkdown": "Word2Vec would also be great for EDA. It (with 2D projection via UMAP or TSNE) would help us visualize these anonymous items.", "votes": null }, { "id": "2025750", "postDate": "11/11/2022 13:07:07", "content": "Great idea(s), Chris! Also as you have mentioned in your other comments, visualizations based on time would be insightful! I have the model, let's see if I can come up with something.
", "rawMarkdown": "Great idea(s), Chris! Also as you have mentioned in your other comments, visualizations based on time would be insightful! I have the model, let's see if I can come up with something.", "votes": null }, { "id": "2060868", "postDate": "12/10/2022 14:20:39", "content": "Thank you for both Radek and Chris; nowThank you for both Radek and Chris, now I am clear with the intuition of co-visitation matrices
", "rawMarkdown": "Thank you for both Radek and Chris; nowThank you for both Radek and Chris, now I am clear with the intuition of co-visitation matrices", "votes": null }, { "id": "2061273", "postDate": "12/10/2022 22:27:19", "content": "Wonderful! that is great to hear @leiwong! π
", "rawMarkdown": "Wonderful! that is great to hear @leiwong! π", "votes": null }, { "id": "2105430", "postDate": "01/18/2023 13:52:08", "content": "Hi Radek, I didn't appreciate this post of yours enough when I read it last month, now what you said about co-visitation matrix makes more sense to me now as I was writing down my reflection of co-visitation matrix these two days.
\nHere is a question I have about comparing co-visitation matrix and word2vect. Since you said co-visitation matrix suffers the same thing as trigram, then word2vec would in theory be a better model for generating candidates right?
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F850197%2F27930a141e9e30ced711ae9a91e518ad%2FScreen%20Shot%202023-01-18%20at%2021.49.24.png?generation=1674049779499957&alt=media\")
In theory, matrix factorization (and word2vec) should do better! But it is important to check how they work in practice. Here, I only played a little bit around with candidate generation using MF and didn't get better results than using the co-visitation matrix. But there could be many things at play.
\nIn general, yes, matrix factorization and word2vec should be superior methods to the co-visitation matrix, that is my belief. But all of this is very situational dependant and depending on the dataset a heuristic such as the co-visitation matrix might work best π (for instances, the co-visitation matrix might better capture short-term trends, or can work really well if there is a long-tail of items purchased).
\nIt is fun and useful to think about these methods, what they do, how they work, etc but I wouldn't draw too definite conclusions, always best to test your hypotheses out on the dataset you are working with π
", "rawMarkdown": "In theory, matrix factorization (and word2vec) should do better! But it is important to check how they work in practice. Here, I only played a little bit around with candidate generation using MF and didn't get better results than using the co-visitation matrix. But there could be many things at play.\n\nIn general, yes, matrix factorization and word2vec should be superior methods to the co-visitation matrix, that is my belief. But all of this is very situational dependant and depending on the dataset a heuristic such as the co-visitation matrix might work best π (for instances, the co-visitation matrix might better capture short-term trends, or can work really well if there is a long-tail of items purchased).\n\nIt is fun and useful to think about these methods, what they do, how they work, etc but I wouldn't draw too definite conclusions, always best to test your hypotheses out on the dataset you are working with π", "votes": null }, { "id": "2106172", "postDate": "01/19/2023 02:13:52", "content": "Thank you so much Radek! What an illuminating and insightful answer! I am looking forward to test them out!
", "rawMarkdown": "Thank you so much Radek! What an illuminating and insightful answer! I am looking forward to test them out!", "votes": null }, { "id": "2106250", "postDate": "01/19/2023 03:42:48", "content": "My pleasure, @saha8631! π Thank you for your kind comment! π
", "rawMarkdown": "My pleasure, @saha8631! π Thank you for your kind comment! π", "votes": null }, { "id": "2173215", "postDate": "03/08/2023 07:45:01", "content": "Can the same principle of co-visitation matrix work when dealing with datasets of restaurants, for recommending a restaurant for a user?
", "rawMarkdown": "Can the same principle of co-visitation matrix work when dealing with datasets of restaurants, for recommending a restaurant for a user?", "votes": null } ], "comments": [ { "id": 2024995, "author_name": "saha8631", "author_url": "", "post_date": "11/10/2022 23:23:57", "content": "Nice explanationβ¦.Really helpfulβ¦Thanksπ
", "votes": null, "replies": [ { "id": 2106250, "author_name": "radek1", "author_url": "", "post_date": "01/19/2023 03:42:48", "content": "My pleasure, @saha8631! π Thank you for your kind comment! π
", "votes": null, "replies": [] } ] }, { "id": 2025025, "author_name": "snnclsr", "author_url": "", "post_date": "11/11/2022 00:03:24", "content": "How funny that I was playing with the Word2Vec and its variants and then saw this post :D
", "votes": null, "replies": [ { "id": 2025036, "author_name": "radek1", "author_url": "", "post_date": "11/11/2022 00:15:38", "content": "Yeah, it would be really interesting to see what word2vec can do in this competition π Might be it could be quite nice for candidate generation or some sort of similarity scoring for reranking π€
\nAnyhow, cool that we were both circulating around the same topic! π
", "votes": null, "replies": [] }, { "id": 2025075, "author_name": "snnclsr", "author_url": "", "post_date": "11/11/2022 00:51:40", "content": "Yes, exactly! I was also thinking of it for the candidate generation. π€π€
", "votes": null, "replies": [] }, { "id": 2025093, "author_name": "cdeotte", "author_url": "", "post_date": "11/11/2022 01:08:02", "content": "Word2Vec would also be great for EDA. It (with 2D projection via UMAP or TSNE) would help us visualize these anonymous items.
", "votes": null, "replies": [] }, { "id": 2025750, "author_name": "snnclsr", "author_url": "", "post_date": "11/11/2022 13:07:07", "content": "Great idea(s), Chris! Also as you have mentioned in your other comments, visualizations based on time would be insightful! I have the model, let's see if I can come up with something.
", "votes": null, "replies": [] } ] }, { "id": 2025045, "author_name": "cdeotte", "author_url": "", "post_date": "11/11/2022 00:27:43", "content": "Co-visitation matrices are the natural statistical approach. It is a way to compute conditional probabilities. Given a user has interacted with item A, it computes the most likely future items that this user will interact with.
\nAnd we can make many more specific variants such as given a user has ordered item A, what is the most likely item that this user will click, cart, order etc. Or given a user has visited this item in the morning, what is the most likely item that this user will order. Etc etc.
\nOf course if we train a reranker model it will learn all these varieties of conditional probabilities on its own.
", "votes": null, "replies": [ { "id": 2025074, "author_name": "radek1", "author_url": "", "post_date": "11/11/2022 00:51:14", "content": "That is some amazing information @cdeotte, as always! π Thank you so very much for sharing all this with us π
", "votes": null, "replies": [] }, { "id": 2060868, "author_name": "leiwong", "author_url": "", "post_date": "12/10/2022 14:20:39", "content": "Thank you for both Radek and Chris; nowThank you for both Radek and Chris, now I am clear with the intuition of co-visitation matrices
", "votes": null, "replies": [] }, { "id": 2061273, "author_name": "radek1", "author_url": "", "post_date": "12/10/2022 22:27:19", "content": "Wonderful! that is great to hear @leiwong! π
", "votes": null, "replies": [] }, { "id": 2173215, "author_name": "edoziemenyinnaya", "author_url": "", "post_date": "03/08/2023 07:45:01", "content": "Can the same principle of co-visitation matrix work when dealing with datasets of restaurants, for recommending a restaurant for a user?
", "votes": null, "replies": [] } ] }, { "id": 2025050, "author_name": "cdeotte", "author_url": "", "post_date": "11/11/2022 00:32:37", "content": "Training Word2Vec would be very helpful. That would produce embeddings such that similar items (with co-visitation) would have distance similar embeddings. It would help us understand what these anonymous item ids represent.
\nThen we could do lots of EDA like how do the items that users purchase over time change? How do items purchased in the morning differ from items purchased in the evening. Etc etc. For each of these analysis, we could plot 2D pictures with clusters of dots (using their Word2Vec embedding 2D projection)
", "votes": null, "replies": [] }, { "id": 2025058, "author_name": "oscarm524", "author_url": "", "post_date": "11/11/2022 00:41:37", "content": "Well explained. Thanks for sharingπ
", "votes": null, "replies": [ { "id": 2025072, "author_name": "radek1", "author_url": "", "post_date": "11/11/2022 00:49:31", "content": "my pleasure, @oscarm524! very glad you found this useful! π
", "votes": null, "replies": [] } ] }, { "id": 2105430, "author_name": "danielliao", "author_url": "", "post_date": "01/18/2023 13:52:08", "content": "Hi Radek, I didn't appreciate this post of yours enough when I read it last month, now what you said about co-visitation matrix makes more sense to me now as I was writing down my reflection of co-visitation matrix these two days.
\nHere is a question I have about comparing co-visitation matrix and word2vect. Since you said co-visitation matrix suffers the same thing as trigram, then word2vec would in theory be a better model for generating candidates right?
\n
In theory, matrix factorization (and word2vec) should do better! But it is important to check how they work in practice. Here, I only played a little bit around with candidate generation using MF and didn't get better results than using the co-visitation matrix. But there could be many things at play.
\nIn general, yes, matrix factorization and word2vec should be superior methods to the co-visitation matrix, that is my belief. But all of this is very situational dependant and depending on the dataset a heuristic such as the co-visitation matrix might work best π (for instances, the co-visitation matrix might better capture short-term trends, or can work really well if there is a long-tail of items purchased).
\nIt is fun and useful to think about these methods, what they do, how they work, etc but I wouldn't draw too definite conclusions, always best to test your hypotheses out on the dataset you are working with π
", "votes": null, "replies": [ { "id": 2106172, "author_name": "danielliao", "author_url": "", "post_date": "01/19/2023 02:13:52", "content": "Thank you so much Radek! What an illuminating and insightful answer! I am looking forward to test them out!
", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2024992": "It is very interesting to think of modern techniques in the context of their roots.\n\nFor instance, when thinking about RNNs we should consider unigram, bigram, and trigram models.\n\nWhat were they?\n\nThey estimated the probability of a word given the words that came before.\n\nWe see \"Radek is a _________\".\n\nIn a trigram model, we would consider the chain \"Radek\", \"is\", and \"a\" and could find the most likely word to come next.\n\nThe easiest way would be to count the occurrences of \"Radek is a\" and the words that came after and pick the one most common.\n\nSo why RNNs are such a great improvement?\n\nWell, there are not that many \"Radek is a _________\" in most text corpora!\n\nWith RNNs (or word2vec) we could operate on embeddings and look at many more words in sequence.\n\nInstead of looking at only \"Radek is a ________\" we can use \"Radek was an ___________\", \"Tommy is a __________\" as examples to learn from, etc.\n\nEmbeddings take values of a variable of high cardinality and project them to a representation that ideally captures similarities.\n\n\n\nSo how does this relate to the co-visitation matrix?\n\nA co-visitation matrix counts the co-occurrence of two actions in close proximity.\n\nIf a user bought A and shortly after bought B, we store these values together.\n\nWe calculate counts and use them to estimate the probability of future actions based on recent history.\n\nIt is quite important to understand what is happening in the co-visitation matrix approach...\n\nSince it suffers from the same issues as our trigram example!\n\nPlus what does the co-visitation matrix resemble?\n\nYou are right, it is akin to doing Matrix Factorization by counting!\n\nIt is really fun that this competition exposed this heuristic (the co-visitation matrix) that I have not been aware of before! π\n\n### Other resources you might find useful:\n\n* [π‘ [2 methods] How-to ensemble predictions π π π ](https://www.kaggle.com/code/radek1/2-methods-how-to-ensemble-predictions)\n* [local validation tracks public LB perfecty -- here is the setup](https://www.kaggle.com/competitions/otto-recommender-system/discussion/364991)\n* [π‘ For my friends from Twitter and LinkedIn -- here is how to dive into this competition π³](https://www.kaggle.com/competitions/otto-recommender-system/discussion/368560)\n* [Full dataset processed to CSV/parquet files with optimized memory footprint](https://www.kaggle.com/competitions/otto-recommender-system/discussion/363843)\n* [co-visitation matrix - simplified, imprvd logic π₯](https://www.kaggle.com/code/radek1/co-visitation-matrix-simplified-imprvd-logic)\n* [π‘ Word2Vec How-to [training and submission]πππ](https://www.kaggle.com/code/radek1/word2vec-how-to-training-and-submission)", "2024995": "Nice explanation....Really helpful...Thanksπ", "2025025": "How funny that I was playing with the Word2Vec and its variants and then saw this post :D", "2025036": "Yeah, it would be really interesting to see what word2vec can do in this competition π Might be it could be quite nice for candidate generation or some sort of similarity scoring for reranking π€\n\nAnyhow, cool that we were both circulating around the same topic! π", "2025045": "Co-visitation matrices are the natural statistical approach. It is a way to compute conditional probabilities. Given a user has interacted with item A, it computes the most likely future items that this user will interact with.\n\nAnd we can make many more specific variants such as given a user has **ordered** item A, what is the most likely item that this user will **click, cart, order** etc. Or given a user has visited this item **in the morning**, what is the most likely item that this user will **order**. Etc etc.\n\nOf course if we train a reranker model it will learn all these varieties of conditional probabilities on its own.", "2025050": "Training Word2Vec would be very helpful. That would produce embeddings such that similar items (with co-visitation) would have distance similar embeddings. It would help us understand what these anonymous item ids represent.\n\nThen we could do lots of EDA like how do the items that users purchase over time change? How do items purchased in the morning differ from items purchased in the evening. Etc etc. For each of these analysis, we could plot 2D pictures with clusters of dots (using their Word2Vec embedding 2D projection)", "2025058": "Well explained. Thanks for sharingπ", "2025072": "my pleasure, @oscarm524! very glad you found this useful! π", "2025074": "That is some amazing information @cdeotte, as always! π Thank you so very much for sharing all this with us π", "2025075": "Yes, exactly! I was also thinking of it for the candidate generation. π€π€", "2025093": "Word2Vec would also be great for EDA. It (with 2D projection via UMAP or TSNE) would help us visualize these anonymous items.", "2025750": "Great idea(s), Chris! Also as you have mentioned in your other comments, visualizations based on time would be insightful! I have the model, let's see if I can come up with something.", "2060868": "Thank you for both Radek and Chris; nowThank you for both Radek and Chris, now I am clear with the intuition of co-visitation matrices", "2061273": "Wonderful! that is great to hear @leiwong! π", "2105430": "Hi Radek, I didn't appreciate this post of yours enough when I read it last month, now what you said about co-visitation matrix makes more sense to me now as I was writing down my reflection of co-visitation matrix these two days. \n\nHere is a question I have about comparing co-visitation matrix and word2vect. Since you said co-visitation matrix suffers the same thing as trigram, then word2vec would in theory be a better model for generating candidates right? \n\n", "2105932": "In theory, matrix factorization (and word2vec) should do better! But it is important to check how they work in practice. Here, I only played a little bit around with candidate generation using MF and didn't get better results than using the co-visitation matrix. But there could be many things at play.\n\nIn general, yes, matrix factorization and word2vec should be superior methods to the co-visitation matrix, that is my belief. But all of this is very situational dependant and depending on the dataset a heuristic such as the co-visitation matrix might work best π (for instances, the co-visitation matrix might better capture short-term trends, or can work really well if there is a long-tail of items purchased).\n\nIt is fun and useful to think about these methods, what they do, how they work, etc but I wouldn't draw too definite conclusions, always best to test your hypotheses out on the dataset you are working with π", "2106172": "Thank you so much Radek! What an illuminating and insightful answer! I am looking forward to test them out!", "2106250": "My pleasure, @saha8631! π Thank you for your kind comment! π", "2173215": "Can the same principle of co-visitation matrix work when dealing with datasets of restaurants, for recommending a restaurant for a user?" }, "source": "meta" }