Do you think that Neural Nets will do a good job in this competition? Especially with this huge amount of data to fit the model!! Theoretically, what are the best algorithms to use?
\nGood luck!
", "messages": [ { "id": "1083762", "postDate": "11/19/2020 10:53:08", "content": "Do you think that Neural Nets will do a good job in this competition? Especially with this huge amount of data to fit the model!! Theoretically, what are the best algorithms to use?
\nGood luck!
", "rawMarkdown": "Do you think that *Neural Nets* will do a good job in this competition? Especially with this huge amount of data to fit the model!! Theoretically, what are the best *algorithms* to use? \n\nGood luck!", "votes": null }, { "id": "1083802", "postDate": "11/19/2020 11:54:07", "content": "I tried different NN's and my results were not good at all, overfitting one batch was hard and did not work for all of them. I'm sticking to boosted methods e.g. RandomForest, XGBoost etc. because my personal results are better.
", "rawMarkdown": "I tried different NN's and my results were not good at all, overfitting one batch was hard and did not work for all of them. I'm sticking to boosted methods e.g. RandomForest, XGBoost etc. because my personal results are better.", "votes": null }, { "id": "1087443", "postDate": "11/22/2020 18:00:00", "content": "Thank you @aliabdin1 for you prompt reply. Obviously, Boosting methods are the preferred for all competitors, but why? Is their a scientific explanation?
", "rawMarkdown": "Thank you @aliabdin1 for you prompt reply. Obviously, Boosting methods are the preferred for all competitors, but why? Is their a scientific explanation?", "votes": null }, { "id": "1092323", "postDate": "11/26/2020 17:35:02", "content": "I used a simple NN with just emb_layer(user_id, content_id)+Linear+Linear and it achieves .753 auc in hidden dataset
\nbut it takes nothing special. (onecycle lr + weightdecay(Im not sure if its necessary))
\nIm just not convinced simple NN is going to be good enough for higher scores. It seems u need something like transformers
Drop user_id and the score will drop i believe as you shouldn't embed user_id's as they are just number's and it's best to avoid it for building models even though it overlaps with the test_set.
", "rawMarkdown": "Drop user_id and the score will drop i believe as you shouldn't embed user_id's as they are just number's and it's best to avoid it for building models even though it overlaps with the test_set.", "votes": null }, { "id": "1092372", "postDate": "11/26/2020 18:37:33", "content": "You can use NNs when the data is in tabular format. The popular fast.ai library has a tabular module that works well. You also get the advantage of using Entity Embeddings which are really effective in tabular format. Here is the paper, if you are interested: Entity Embeddings of Categorical Variables.
\nThe paper was written by people who were among the top teams in a Kaggle competition which had tabular data.
\nHere is a detailed tutorial in notebook format (which you can download or run in Colab) from fast.ai which demonstrates the use of the tabular module of fastai library- https://github.com/fastai/fastbook/blob/master/09_tabular.ipynb. It also teaches a lot more about the use of NNs for tabular data with complete code examples of another Kaggle tabular competition. So, have a go at it!
Also, it is important to mention that Random Forests and Boosted Trees are often better than NNs when it comes to tabular data.
", "rawMarkdown": "You can use NNs when the data is in tabular format. The popular [fast.ai](https://fast.ai) library has a tabular module that works well. You also get the advantage of using Entity Embeddings which are really effective in tabular format. Here is the paper, if you are interested: [Entity Embeddings of Categorical Variables](https://arxiv.org/abs/1604.06737).\n\nThe paper was written by people who were among the top teams in a Kaggle competition which had tabular data.\n\nHere is a detailed tutorial in notebook format (which you can download or run in Colab) from fast.ai which demonstrates the use of the `tabular` module of `fastai` library- https://github.com/fastai/fastbook/blob/master/09_tabular.ipynb. It also teaches a lot more about the use of NNs for tabular data with complete code examples of another Kaggle tabular competition. So, have a go at it!\n\nAlso, it is important to mention that Random Forests and Boosted Trees are often better than NNs when it comes to tabular data.", "votes": null }, { "id": "1092388", "postDate": "11/26/2020 18:58:19", "content": "sry english is not my first language
\nactually it depends on model, a regular NN doesn't have any sort of mechanic to keep user history, so user embedding purpose here is coding the user characteristic. I have a null user embedding which Im using for unknown users in inference time. I used data from users with less than 5 interactions for training null user embedding (it makes it's experience so diverse, I think its a good thing for unknown users), however I think there are better approaches.
\nI'm agree with your point in a transformer, because the history (user characteristic) is part of the data sequence
BTW: I don't have a good understanding about how to use transformers here (and why transformers learn overall).. waiting for ur notebook update :D
", "rawMarkdown": "sry english is not my first language\n\nactually it depends on model, a regular NN doesn't have any sort of mechanic to keep user history, so user embedding purpose here is coding the user characteristic. I have a null user embedding which Im using for unknown users in inference time. I used data from users with less than 5 interactions for training null user embedding (it makes it's experience so diverse, I think its a good thing for unknown users), however I think there are better approaches.\nI'm agree with your point in a transformer, because the history (user characteristic) is part of the data sequence\n\nBTW: I don't have a good understanding about how to use transformers here (and why transformers learn overall).. waiting for ur notebook update :D", "votes": null } ], "comments": [ { "id": 1083802, "author_name": "aliabdin1", "author_url": "", "post_date": "11/19/2020 11:54:07", "content": "I tried different NN's and my results were not good at all, overfitting one batch was hard and did not work for all of them. I'm sticking to boosted methods e.g. RandomForest, XGBoost etc. because my personal results are better.
", "votes": null, "replies": [ { "id": 1087443, "author_name": "theexpertahmeed", "author_url": "", "post_date": "11/22/2020 18:00:00", "content": "Thank you @aliabdin1 for you prompt reply. Obviously, Boosting methods are the preferred for all competitors, but why? Is their a scientific explanation?
", "votes": null, "replies": [] } ] }, { "id": 1092323, "author_name": "feriiiiiiiiii", "author_url": "", "post_date": "11/26/2020 17:35:02", "content": "I used a simple NN with just emb_layer(user_id, content_id)+Linear+Linear and it achieves .753 auc in hidden dataset
\nbut it takes nothing special. (onecycle lr + weightdecay(Im not sure if its necessary))
\nIm just not convinced simple NN is going to be good enough for higher scores. It seems u need something like transformers
Drop user_id and the score will drop i believe as you shouldn't embed user_id's as they are just number's and it's best to avoid it for building models even though it overlaps with the test_set.
", "votes": null, "replies": [] }, { "id": 1092388, "author_name": "feriiiiiiiiii", "author_url": "", "post_date": "11/26/2020 18:58:19", "content": "sry english is not my first language
\nactually it depends on model, a regular NN doesn't have any sort of mechanic to keep user history, so user embedding purpose here is coding the user characteristic. I have a null user embedding which Im using for unknown users in inference time. I used data from users with less than 5 interactions for training null user embedding (it makes it's experience so diverse, I think its a good thing for unknown users), however I think there are better approaches.
\nI'm agree with your point in a transformer, because the history (user characteristic) is part of the data sequence
BTW: I don't have a good understanding about how to use transformers here (and why transformers learn overall).. waiting for ur notebook update :D
", "votes": null, "replies": [] } ] }, { "id": 1092372, "author_name": "truthr", "author_url": "", "post_date": "11/26/2020 18:37:33", "content": "You can use NNs when the data is in tabular format. The popular fast.ai library has a tabular module that works well. You also get the advantage of using Entity Embeddings which are really effective in tabular format. Here is the paper, if you are interested: Entity Embeddings of Categorical Variables.
\nThe paper was written by people who were among the top teams in a Kaggle competition which had tabular data.
\nHere is a detailed tutorial in notebook format (which you can download or run in Colab) from fast.ai which demonstrates the use of the tabular module of fastai library- https://github.com/fastai/fastbook/blob/master/09_tabular.ipynb. It also teaches a lot more about the use of NNs for tabular data with complete code examples of another Kaggle tabular competition. So, have a go at it!
Also, it is important to mention that Random Forests and Boosted Trees are often better than NNs when it comes to tabular data.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1083762": "Do you think that *Neural Nets* will do a good job in this competition? Especially with this huge amount of data to fit the model!! Theoretically, what are the best *algorithms* to use? \n\nGood luck!", "1083802": "I tried different NN's and my results were not good at all, overfitting one batch was hard and did not work for all of them. I'm sticking to boosted methods e.g. RandomForest, XGBoost etc. because my personal results are better.", "1087443": "Thank you @aliabdin1 for you prompt reply. Obviously, Boosting methods are the preferred for all competitors, but why? Is their a scientific explanation?", "1092323": "I used a simple NN with just emb_layer(user_id, content_id)+Linear+Linear and it achieves .753 auc in hidden dataset\nbut it takes nothing special. (onecycle lr + weightdecay(Im not sure if its necessary))\nIm just not convinced simple NN is going to be good enough for higher scores. It seems u need something like transformers", "1092356": "Drop user_id and the score will drop i believe as you shouldn't embed user_id's as they are just number's and it's best to avoid it for building models even though it overlaps with the test_set.", "1092372": "You can use NNs when the data is in tabular format. The popular [fast.ai](https://fast.ai) library has a tabular module that works well. You also get the advantage of using Entity Embeddings which are really effective in tabular format. Here is the paper, if you are interested: [Entity Embeddings of Categorical Variables](https://arxiv.org/abs/1604.06737).\n\nThe paper was written by people who were among the top teams in a Kaggle competition which had tabular data.\n\nHere is a detailed tutorial in notebook format (which you can download or run in Colab) from fast.ai which demonstrates the use of the `tabular` module of `fastai` library- https://github.com/fastai/fastbook/blob/master/09_tabular.ipynb. It also teaches a lot more about the use of NNs for tabular data with complete code examples of another Kaggle tabular competition. So, have a go at it!\n\nAlso, it is important to mention that Random Forests and Boosted Trees are often better than NNs when it comes to tabular data.", "1092388": "sry english is not my first language\n\nactually it depends on model, a regular NN doesn't have any sort of mechanic to keep user history, so user embedding purpose here is coding the user characteristic. I have a null user embedding which Im using for unknown users in inference time. I used data from users with less than 5 interactions for training null user embedding (it makes it's experience so diverse, I think its a good thing for unknown users), however I think there are better approaches.\nI'm agree with your point in a transformer, because the history (user characteristic) is part of the data sequence\n\nBTW: I don't have a good understanding about how to use transformers here (and why transformers learn overall).. waiting for ur notebook update :D" }, "source": "meta" }