Hey!
\nSo the cardinality of AIDs is nearly 2 million. That is way more than traditional methods can handle if we want to predict the next item (aid) in a sequence.
Normally, when doing single-label classification, we would use softmax to output our predicitons. But softmax is known to not work that well beyond a certain size. Plus, it becomes very resource intensive to have over 1.5 million activations in a layer!
\nI haven't seen deep learning models shared just yet, but one could imagine an RNN or a Transformer trained on sequences and outputting single (or multiple) labels.
\nHow are you addressing this situation? More generally, how are you working around the limitations of Softmax?
\nOne approach would certainly be sampled softmax, though I have not come across a good PyTorch implementation. Guess implementing it on your own shouldn't be too tricky.
\nš now here is quite likely a very valuable idea: instead of using the covisitation matrix, train a matrix factorization model, I am not sure how I am doing on time this week but if time permits maybe I'll hack such a thing together and share š
\nWith embeddings, you can use nearest neighbor search and the whole idea of cardinality goes away.
\nWhat are your thoughts on this?
\nHey!
\nSo the cardinality of AIDs is nearly 2 million. That is way more than traditional methods can handle if we want to predict the next item (aid) in a sequence.
Normally, when doing single-label classification, we would use softmax to output our predicitons. But softmax is known to not work that well beyond a certain size. Plus, it becomes very resource intensive to have over 1.5 million activations in a layer!
\nI haven't seen deep learning models shared just yet, but one could imagine an RNN or a Transformer trained on sequences and outputting single (or multiple) labels.
\nHow are you addressing this situation? More generally, how are you working around the limitations of Softmax?
\nOne approach would certainly be sampled softmax, though I have not come across a good PyTorch implementation. Guess implementing it on your own shouldn't be too tricky.
\nš now here is quite likely a very valuable idea: instead of using the covisitation matrix, train a matrix factorization model, I am not sure how I am doing on time this week but if time permits maybe I'll hack such a thing together and share š
\nWith embeddings, you can use nearest neighbor search and the whole idea of cardinality goes away.
\nWhat are your thoughts on this?
\nOne technique that can handle high cardinality is \"candidate rerank\" models. For example, for each test session, we generate candidates. Then we merge features unto the candidates and finally predict final 20 via XGB rerank using the features.
", "rawMarkdown": "One technique that can handle high cardinality is \"candidate rerank\" models. For example, for each test session, we generate candidates. Then we merge features unto the candidates and finally predict final 20 via XGB rerank using the features.", "votes": null }, { "id": "2021975", "postDate": "11/08/2022 15:47:07", "content": "Ravi posted a discussion about this idea here
", "rawMarkdown": "Ravi posted a discussion about this idea [here][1]\n\n[1]: https://www.kaggle.com/competitions/otto-recommender-system/discussion/364721", "votes": null } ], "comments": [ { "id": 2021088, "author_name": "cdeotte", "author_url": "", "post_date": "11/08/2022 01:44:56", "content": "One technique that can handle high cardinality is \"candidate rerank\" models. For example, for each test session, we generate candidates. Then we merge features unto the candidates and finally predict final 20 via XGB rerank using the features.
", "votes": null, "replies": [ { "id": 2021975, "author_name": "cdeotte", "author_url": "", "post_date": "11/08/2022 15:47:07", "content": "Ravi posted a discussion about this idea here
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2021034": "Hey!\n\nSo the cardinality of `AIDs` is nearly 2 million. That is way more than traditional methods can handle if we want to predict the next item (`aid`) in a sequence.\n\nNormally, when doing single-label classification, we would use softmax to output our predicitons. But softmax is known to not work that well beyond a certain size. Plus, it becomes very resource intensive to have over 1.5 million activations in a layer!\n\nI haven't seen deep learning models shared just yet, but one could imagine an RNN or a Transformer trained on sequences and outputting single (or multiple) labels. \n\nHow are you addressing this situation? More generally, how are you working around the limitations of Softmax?\n\nOne approach would certainly be sampled softmax, though I have not come across a good PyTorch implementation. Guess implementing it on your own shouldn't be too tricky.\n\nš now here is quite likely a very valuable idea: instead of using the covisitation matrix, train a matrix factorization model, I am not sure how I am doing on time this week but if time permits maybe I'll hack such a thing together and share š\n\nWith embeddings, you can use nearest neighbor search and the whole idea of cardinality goes away.\n\nWhat are your thoughts on this?\n\n### Other resources you might find useful:\n\n* [š” [2 methods] How-to ensemble predictions š š š ](https://www.kaggle.com/code/radek1/2-methods-how-to-ensemble-predictions)\n* [local validation tracks public LB perfecty -- here is the setup](https://www.kaggle.com/competitions/otto-recommender-system/discussion/364991)\n* [š” For my friends from Twitter and LinkedIn -- here is how to dive into this competition š³](https://www.kaggle.com/competitions/otto-recommender-system/discussion/368560)\n* [Full dataset processed to CSV/parquet files with optimized memory footprint](https://www.kaggle.com/competitions/otto-recommender-system/discussion/363843)\n* [co-visitation matrix - simplified, imprvd logic š„](https://www.kaggle.com/code/radek1/co-visitation-matrix-simplified-imprvd-logic)\n* [š” Word2Vec How-to [training and submission]ššš](https://www.kaggle.com/code/radek1/word2vec-how-to-training-and-submission)", "2021088": "One technique that can handle high cardinality is \"candidate rerank\" models. For example, for each test session, we generate candidates. Then we merge features unto the candidates and finally predict final 20 via XGB rerank using the features.", "2021975": "Ravi posted a discussion about this idea [here][1]\n\n[1]: https://www.kaggle.com/competitions/otto-recommender-system/discussion/364721" }, "source": "meta" }