I've just red about TabNet. Here is it's implementation on PyTorch https://github.com/dreamquark-ai/tabnet. Will it be useful for this competition?
", "messages": [ { "id": "1056392", "postDate": "10/21/2020 16:49:40", "content": "I've just red about TabNet. Here is it's implementation on PyTorch https://github.com/dreamquark-ai/tabnet. Will it be useful for this competition?
", "rawMarkdown": "I've just red about TabNet. Here is it's implementation on PyTorch https://github.com/dreamquark-ai/tabnet. Will it be useful for this competition?", "votes": null }, { "id": "1056921", "postDate": "10/22/2020 07:45:58", "content": "I've made a starter kernel : https://www.kaggle.com/alexj21/riiid-tabnet-starter-kernel
\nIt is much slower than trees and provides lower results. It seems not well suited for very large data
I thought it would be worth a shot as well, and when comparing it to other methods I had the same experience as @alexj21 : slower and performing worse. Perhaps later on when I have had the time to redesign the architecture and features I will try again.
", "rawMarkdown": "I thought it would be worth a shot as well, and when comparing it to other methods I had the same experience as @alexj21 : slower and performing worse. Perhaps later on when I have had the time to redesign the architecture and features I will try again.", "votes": null }, { "id": "1057695", "postDate": "10/22/2020 22:03:55", "content": "This was my experience as well, although I used a TabNetRegressor instead of a TabNetClassifier. Didn't submit to the LB because validation scores were only around ~0.72-0.73
", "rawMarkdown": "This was my experience as well, although I used a TabNetRegressor instead of a TabNetClassifier. Didn't submit to the LB because validation scores were only around ~0.72-0.73", "votes": null }, { "id": "1057958", "postDate": "10/23/2020 07:26:31", "content": "Depending on your validation scheme, local 0.73 could lead to a nice LB score.
\nAre you using trees right now ?
Yeah if I remember well I've seen a notebook in which the user redesigned the network
\nIt would be an elegant solution. I would have reached top 7% in OSIC challenge with a single tabnet if I had chosen it as final solution
Yep, just a single LightGBM model with a lot of feature engineering.
", "rawMarkdown": "Yep, just a single LightGBM model with a lot of feature engineering.", "votes": null }, { "id": "1058463", "postDate": "10/23/2020 18:33:57", "content": "Great Going and Good luck! I hope that single lgbm is not trained on whole data. In case it's, What's the peak mem requirement and time it takes to do a pass? Ty!
", "rawMarkdown": "Great Going and Good luck! I hope that single lgbm is not trained on whole data. In case it's, What's the peak mem requirement and time it takes to do a pass? Ty!", "votes": null }, { "id": "1058469", "postDate": "10/23/2020 18:43:04", "content": "Thanks! Nah it's only trained on a small subset of the data--this is how I implement the train/validation split:
\nprint(\"[1] Create Training Set\")\ntraining = combined_df.groupby(\"user_id\").tail(24)\ncombined_df = combined_df.drop(training.index)\n\nprint(\"[2] Split Training Set into Validation\")\nvalidation = training.groupby(\"user_id\").tail(6)\ntraining = training.drop(validation.index)\nRegarding memory, it is a bit difficult to keep within the limits, but there's a ton of tricks you can implement to conserve it, including gc, doing memory-intensive calculations in a separate notebook, and the like. It currently takes about two hours to train a model, and then about four hours to see results on the LB after using that saved model in a separate run to generate predictions.
", "rawMarkdown": "Thanks! Nah it's only trained on a small subset of the data--this is how I implement the train/validation split:\n\n```\nprint(\"[1] Create Training Set\")\ntraining = combined_df.groupby(\"user_id\").tail(24)\ncombined_df = combined_df.drop(training.index)\n\nprint(\"[2] Split Training Set into Validation\")\nvalidation = training.groupby(\"user_id\").tail(6)\ntraining = training.drop(validation.index)\n```\nRegarding memory, it is a bit difficult to keep within the limits, but there's a ton of tricks you can implement to conserve it, including gc, doing memory-intensive calculations in a separate notebook, and the like. It currently takes about two hours to train a model, and then about four hours to see results on the LB after using that saved model in a separate run to generate predictions.", "votes": null }, { "id": "1058475", "postDate": "10/23/2020 18:53:10", "content": "Thanks for sharing Nick! I guess, I am over thinking about the problem statement 😅!
", "rawMarkdown": "Thanks for sharing Nick! I guess, I am over thinking about the problem statement 😅!", "votes": null }, { "id": "1063663", "postDate": "10/29/2020 07:31:48", "content": "Thak you for sharing!
\n When creating features for validation, do you use combined_df or combined_df + trainig?
I've made a starter kernel : https://www.kaggle.com/alexj21/riiid-tabnet-starter-kernel
\nIt is much slower than trees and provides lower results. It seems not well suited for very large data
This was my experience as well, although I used a TabNetRegressor instead of a TabNetClassifier. Didn't submit to the LB because validation scores were only around ~0.72-0.73
", "votes": null, "replies": [] }, { "id": 1057958, "author_name": "alexj21", "author_url": "", "post_date": "10/23/2020 07:26:31", "content": "Depending on your validation scheme, local 0.73 could lead to a nice LB score.
\nAre you using trees right now ?
Yep, just a single LightGBM model with a lot of feature engineering.
", "votes": null, "replies": [] }, { "id": 1058463, "author_name": "adityaecdrid", "author_url": "", "post_date": "10/23/2020 18:33:57", "content": "Great Going and Good luck! I hope that single lgbm is not trained on whole data. In case it's, What's the peak mem requirement and time it takes to do a pass? Ty!
", "votes": null, "replies": [] }, { "id": 1058469, "author_name": "misfyre", "author_url": "", "post_date": "10/23/2020 18:43:04", "content": "Thanks! Nah it's only trained on a small subset of the data--this is how I implement the train/validation split:
\nprint(\"[1] Create Training Set\")\ntraining = combined_df.groupby(\"user_id\").tail(24)\ncombined_df = combined_df.drop(training.index)\n\nprint(\"[2] Split Training Set into Validation\")\nvalidation = training.groupby(\"user_id\").tail(6)\ntraining = training.drop(validation.index)\nRegarding memory, it is a bit difficult to keep within the limits, but there's a ton of tricks you can implement to conserve it, including gc, doing memory-intensive calculations in a separate notebook, and the like. It currently takes about two hours to train a model, and then about four hours to see results on the LB after using that saved model in a separate run to generate predictions.
", "votes": null, "replies": [] }, { "id": 1058475, "author_name": "adityaecdrid", "author_url": "", "post_date": "10/23/2020 18:53:10", "content": "Thanks for sharing Nick! I guess, I am over thinking about the problem statement 😅!
", "votes": null, "replies": [] }, { "id": 1063663, "author_name": "d1348k", "author_url": "", "post_date": "10/29/2020 07:31:48", "content": "Thak you for sharing!
\n When creating features for validation, do you use combined_df or combined_df + trainig?
I thought it would be worth a shot as well, and when comparing it to other methods I had the same experience as @alexj21 : slower and performing worse. Perhaps later on when I have had the time to redesign the architecture and features I will try again.
", "votes": null, "replies": [ { "id": 1057962, "author_name": "alexj21", "author_url": "", "post_date": "10/23/2020 07:32:08", "content": "Yeah if I remember well I've seen a notebook in which the user redesigned the network
\nIt would be an elegant solution. I would have reached top 7% in OSIC challenge with a single tabnet if I had chosen it as final solution