I had this question. Let's say one use LGBM without any feature engineering, there are many hyperparameter combinations and one finds the optimal params for best performance. Then let's say there is some feature engineering, some magic features that definitely improves score. Will it be necessary to tune hyperparameters again after adding this feature to find new optimal ones?
\nAlso similar question. Let's say we have some set of hyperparameters and we want to test a new feature, is it helpful or no. Can it be so that on this exact set of hyperparams quality will decrease, however, this feature is very helpful and on some different set of hyperparams it might increase quality significantly? If it is so, how should new features or tricks be tested?
", "messages": [ { "id": 1837818, "postDate": "2022-06-30T01:04:16.520Z", "content": "I had this question. Let's say one use LGBM without any feature engineering, there are many hyperparameter combinations and one finds the optimal params for best performance. Then let's say there is some feature engineering, some magic features that definitely improves score. Will it be necessary to tune hyperparameters again after adding this feature to find new optimal ones?
\nAlso similar question. Let's say we have some set of hyperparameters and we want to test a new feature, is it helpful or no. Can it be so that on this exact set of hyperparams quality will decrease, however, this feature is very helpful and on some different set of hyperparams it might increase quality significantly? If it is so, how should new features or tricks be tested?
", "rawMarkdown": "I had this question. Let's say one use LGBM without any feature engineering, there are many hyperparameter combinations and one finds the optimal params for best performance. Then let's say there is some feature engineering, some magic features that definitely improves score. Will it be necessary to tune hyperparameters again after adding this feature to find new optimal ones?\n\nAlso similar question. Let's say we have some set of hyperparameters and we want to test a new feature, is it helpful or no. Can it be so that on this exact set of hyperparams quality will decrease, however, this feature is very helpful and on some different set of hyperparams it might increase quality significantly? If it is so, how should new features or tricks be tested?", "votes": 12 }, { "id": 1838791, "postDate": "2022-06-30T20:18:12.500Z", "content": "Hyperparameters optimization should be the last step, if not, you might overfit your validation set (or multiple K validation sets).
\nThis is what I normally do:
\nWouldnt it be better to HPO first, so that the model is familiar with the type of data you are using? And then doing feature selection? Also is there any where we can see that HPO then FS is overfitting?
", "rawMarkdown": "Wouldnt it be better to HPO first, so that the model is familiar with the type of data you are using? And then doing feature selection? Also is there any where we can see that HPO then FS is overfitting?" }, { "id": 1838882, "postDate": "2022-06-30T23:37:25.317Z", "content": "I think hyperparams optimization might be useless, if you later add or delete features. As I understood it's easier to check features usefulness on default params and later tune them, when you choose features. About overfitting validation I also do not really understood this, I mean, if you do cross validation on same hyperparams, you'll see what are the best for all folds, not just for the one of them. And then you have public lb score to test final quality and see, if you overfitted due cross validation or not.
", "rawMarkdown": "I think hyperparams optimization might be useless, if you later add or delete features. As I understood it's easier to check features usefulness on default params and later tune them, when you choose features. About overfitting validation I also do not really understood this, I mean, if you do cross validation on same hyperparams, you'll see what are the best for all folds, not just for the one of them. And then you have public lb score to test final quality and see, if you overfitted due cross validation or not.", "votes": 5 } ] }, { "id": 1838081, "postDate": "2022-06-30T07:36:58.490Z", "content": "Changed or added features definetly affect the model directly by changing the values or adding new dimensions to the feature space, thats why you should tune the parameters after you have done your feature engineering.
", "rawMarkdown": "Changed or added features definetly affect the model directly by changing the values or adding new dimensions to the feature space, thats why you should tune the parameters after you have done your feature engineering.", "votes": 6 }, { "id": 1838910, "postDate": "2022-07-01T02:01:18.850Z", "content": "As everyone suggested the first step is FE and the last step is stratified k-fold grid search.
\nHere are two interesting papers for FE:
Automated Feature Engineering Using Neural Networks
\nWhy You Should Always Use Feature Embeddings With Structured Datasets
There's recursive feature elimination (RFE) as well.
\nGood share with the links 👍
Hello @manwithaflower, I usually reserve hyperparam tunning as the last step of my workflow; I feel that feature engineering should receive all my focus on the first iterations
", "rawMarkdown": "Hello @manwithaflower, I usually reserve hyperparam tunning as the last step of my workflow; I feel that feature engineering should receive all my focus on the first iterations ", "votes": 3 }, { "id": 1838747, "postDate": "2022-06-30T19:09:36.867Z", "content": "Hi o/
\nHyperparameter tuning is the last step in the process, so if you've added a or modified a feature, then you should redo the entire tuning from start to get the best benefit from new feature.
I usually create a baseline model with increased n_estimators to 500-4000 on CPU, or 5000-10000 on GPU - in the initial stage you want the model to train quickly so you can get the results faster.
\nIf you want to know how is your new feature performing then train the original data on a model with preset random_state, then add the feature, use same random_state and compare the results
\nmodel = LGBMRegressor(random_state=RS, n_estimators=2000)
To know how the feature compares to others you can use model.feature_importances_
\nor even better use SequentialFeatureSelector to sequentially add features and see which actually add value.
You should carry out feature engineering before model training almost always. I don't think one should tune parameters before feature engineering as this could cause improper model fitting.
", "rawMarkdown": "You should carry out feature engineering before model training almost always. I don't think one should tune parameters before feature engineering as this could cause improper model fitting. ", "votes": 2 }, { "id": 1837928, "postDate": "2022-06-30T04:06:52.997Z", "content": "I like to build a tuning process for the model - run it once per week. In general we are looking for 0.00X kind of leaps to win this competition. I think for kaggle you need to balance model tuning and feature engineering.
", "rawMarkdown": "I like to build a tuning process for the model - run it once per week. In general we are looking for 0.00X kind of leaps to win this competition. I think for kaggle you need to balance model tuning and feature engineering.", "votes": 2 }, { "id": 1845123, "postDate": "2022-07-06T04:45:07.520Z", "content": "Feature engineering first for sure!
\nHere is my rough workflow for a project like this:
\nHyperparameters optimization should be the last step, if not, you might overfit your validation set (or multiple K validation sets).
\nThis is what I normally do:
\nWouldnt it be better to HPO first, so that the model is familiar with the type of data you are using? And then doing feature selection? Also is there any where we can see that HPO then FS is overfitting?
", "votes": 0, "replies": [] }, { "id": 1838882, "author_name": "Man of the year", "author_url": "", "post_date": "2022-06-30T23:37:25.317000", "content": "I think hyperparams optimization might be useless, if you later add or delete features. As I understood it's easier to check features usefulness on default params and later tune them, when you choose features. About overfitting validation I also do not really understood this, I mean, if you do cross validation on same hyperparams, you'll see what are the best for all folds, not just for the one of them. And then you have public lb score to test final quality and see, if you overfitted due cross validation or not.
", "votes": 5, "replies": [] } ] }, { "id": 1838081, "author_name": "Ali Abdin", "author_url": "", "post_date": "2022-06-30T07:36:58.490000", "content": "Changed or added features definetly affect the model directly by changing the values or adding new dimensions to the feature space, thats why you should tune the parameters after you have done your feature engineering.
", "votes": 6, "replies": [] }, { "id": 1838910, "author_name": "1110Ra", "author_url": "", "post_date": "2022-07-01T02:01:18.850000", "content": "As everyone suggested the first step is FE and the last step is stratified k-fold grid search.
\nHere are two interesting papers for FE:
Automated Feature Engineering Using Neural Networks
\nWhy You Should Always Use Feature Embeddings With Structured Datasets
There's recursive feature elimination (RFE) as well.
\nGood share with the links 👍
Hello @manwithaflower, I usually reserve hyperparam tunning as the last step of my workflow; I feel that feature engineering should receive all my focus on the first iterations
", "votes": 3, "replies": [] }, { "id": 1838747, "author_name": "Waldemar", "author_url": "", "post_date": "2022-06-30T19:09:36.867000", "content": "Hi o/
\nHyperparameter tuning is the last step in the process, so if you've added a or modified a feature, then you should redo the entire tuning from start to get the best benefit from new feature.
I usually create a baseline model with increased n_estimators to 500-4000 on CPU, or 5000-10000 on GPU - in the initial stage you want the model to train quickly so you can get the results faster.
\nIf you want to know how is your new feature performing then train the original data on a model with preset random_state, then add the feature, use same random_state and compare the results
\nmodel = LGBMRegressor(random_state=RS, n_estimators=2000)
To know how the feature compares to others you can use model.feature_importances_
\nor even better use SequentialFeatureSelector to sequentially add features and see which actually add value.
You should carry out feature engineering before model training almost always. I don't think one should tune parameters before feature engineering as this could cause improper model fitting.
", "votes": 2, "replies": [] }, { "id": 1837928, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2022-06-30T04:06:52.997000", "content": "I like to build a tuning process for the model - run it once per week. In general we are looking for 0.00X kind of leaps to win this competition. I think for kaggle you need to balance model tuning and feature engineering.
", "votes": 2, "replies": [] }, { "id": 1845123, "author_name": "Xavier R Nogueira", "author_url": "", "post_date": "2022-07-06T04:45:07.520000", "content": "Feature engineering first for sure!
\nHere is my rough workflow for a project like this:
\n