{ "id": 344704, "title": "Features importance and hyper parameters optimization - which fisrt, which after?", "url": "/competitions/amex-default-prediction/discussion/344704", "author_name": "", "post_date": "2022-08-16T09:10:46.725270900Z", "votes": 1, "comment_count": 5, "views": 0, "content": "

Greetings!
\nI have a question )
\nWe have 2 techniques of score improvement
\nCalculation of features importance and hyper parameters optimization
\nWhat is the best way to combine them both?
\nIn which order should I use it?
\nRight now I am using permutation features importance first and selecting for train only features with positive importance, after that I am running OPTUNA to determine best values of hyper parameters
\nMay be our respective GMs know better way to use these techniques?

", "messages": [ { "id": "1900806", "postDate": "08/16/2022 09:10:46", "content": "

Greetings!
\nI have a question )
\nWe have 2 techniques of score improvement
\nCalculation of features importance and hyper parameters optimization
\nWhat is the best way to combine them both?
\nIn which order should I use it?
\nRight now I am using permutation features importance first and selecting for train only features with positive importance, after that I am running OPTUNA to determine best values of hyper parameters
\nMay be our respective GMs know better way to use these techniques?

", "rawMarkdown": "Greetings!\nI have a question )\nWe have 2 techniques of score improvement\nCalculation of features importance and hyper parameters optimization\nWhat is the best way to combine them both?\nIn which order should I use it?\nRight now I am using permutation features importance first and selecting for train only features with positive importance, after that I am running OPTUNA to determine best values of hyper parameters\nMay be our respective GMs know better way to use these techniques?", "votes": null }, { "id": "1900845", "postDate": "08/16/2022 09:40:42", "content": "

There is no point in doing hyperparameter optimization and than changing the features. The found parameters work for sure only for the features that were used during the search. They may by luck work similarly after feature selection, but that is not guaranteed. This is a long way of saying that feature selection comes first.

", "rawMarkdown": "There is no point in doing hyperparameter optimization and than changing the features. The found parameters work for sure only for the features that were used during the search. They may by luck work similarly after feature selection, but that is not guaranteed. This is a long way of saying that feature selection comes first.", "votes": null }, { "id": "1900886", "postDate": "08/16/2022 10:27:38", "content": "

Thanks a lot!
\nI am glad that my intuition was correct

", "rawMarkdown": "Thanks a lot!\nI am glad that my intuition was correct", "votes": null }, { "id": "1904568", "postDate": "08/18/2022 09:48:33", "content": "

Greetings, Tilii!
\nLet me ask one more question about hyper parameters optimization, please
\nDoes learning rate affect other parameters?
\nIf I optimized all the parameters and after that decreased learning rate sufficiently (3-5-10 times) will it make the parameters combination optimization insensible?

", "rawMarkdown": "Greetings, Tilii!\nLet me ask one more question about hyper parameters optimization, please\nDoes learning rate affect other parameters?\nIf I optimized all the parameters and after that decreased learning rate sufficiently (3-5-10 times) will it make the parameters combination optimization insensible?", "votes": null }, { "id": "1904868", "postDate": "08/18/2022 14:55:36", "content": "

Good question! I often search for hyperparameters with lr=0.1 or lf=0.05 as it speeds up convergence. After finding the parameters, I will do a proper run with lr=0.01 and always get a better score than during the search. In my experience, what you are asking about is doable.

", "rawMarkdown": "Good question! I often search for hyperparameters with `lr=0.1` or `lf=0.05` as it speeds up convergence. After finding the parameters, I will do a proper run with `lr=0.01` and always get a better score than during the search. In my experience, what you are asking about is doable.", "votes": null }, { "id": "1905463", "postDate": "08/19/2022 04:55:26", "content": "

Thank You!

", "rawMarkdown": "Thank You!", "votes": null } ], "comments": [ { "id": 1900845, "author_name": "tilii7", "author_url": "", "post_date": "08/16/2022 09:40:42", "content": "

There is no point in doing hyperparameter optimization and than changing the features. The found parameters work for sure only for the features that were used during the search. They may by luck work similarly after feature selection, but that is not guaranteed. This is a long way of saying that feature selection comes first.

", "votes": null, "replies": [ { "id": 1900886, "author_name": "kaggledummie007", "author_url": "", "post_date": "08/16/2022 10:27:38", "content": "

Thanks a lot!
\nI am glad that my intuition was correct

", "votes": null, "replies": [] }, { "id": 1904568, "author_name": "kaggledummie007", "author_url": "", "post_date": "08/18/2022 09:48:33", "content": "

Greetings, Tilii!
\nLet me ask one more question about hyper parameters optimization, please
\nDoes learning rate affect other parameters?
\nIf I optimized all the parameters and after that decreased learning rate sufficiently (3-5-10 times) will it make the parameters combination optimization insensible?

", "votes": null, "replies": [] }, { "id": 1904868, "author_name": "tilii7", "author_url": "", "post_date": "08/18/2022 14:55:36", "content": "

Good question! I often search for hyperparameters with lr=0.1 or lf=0.05 as it speeds up convergence. After finding the parameters, I will do a proper run with lr=0.01 and always get a better score than during the search. In my experience, what you are asking about is doable.

", "votes": null, "replies": [] }, { "id": 1905463, "author_name": "kaggledummie007", "author_url": "", "post_date": "08/19/2022 04:55:26", "content": "

Thank You!

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1900806": "Greetings!\nI have a question )\nWe have 2 techniques of score improvement\nCalculation of features importance and hyper parameters optimization\nWhat is the best way to combine them both?\nIn which order should I use it?\nRight now I am using permutation features importance first and selecting for train only features with positive importance, after that I am running OPTUNA to determine best values of hyper parameters\nMay be our respective GMs know better way to use these techniques?", "1900845": "There is no point in doing hyperparameter optimization and than changing the features. The found parameters work for sure only for the features that were used during the search. They may by luck work similarly after feature selection, but that is not guaranteed. This is a long way of saying that feature selection comes first.", "1900886": "Thanks a lot!\nI am glad that my intuition was correct", "1904568": "Greetings, Tilii!\nLet me ask one more question about hyper parameters optimization, please\nDoes learning rate affect other parameters?\nIf I optimized all the parameters and after that decreased learning rate sufficiently (3-5-10 times) will it make the parameters combination optimization insensible?", "1904868": "Good question! I often search for hyperparameters with `lr=0.1` or `lf=0.05` as it speeds up convergence. After finding the parameters, I will do a proper run with `lr=0.01` and always get a better score than during the search. In my experience, what you are asking about is doable.", "1905463": "Thank You!" }, "source": "meta" }