![\"\"](\"https://i.postimg.cc/2SdHN5fL/1-Pgw-A0-J423hc-JNpym8o-M9-w.png\")
This is an explanation of the Boruta-Shap Feature Selection.
\nIt is a pretty smart algorithm used for Feature Selection. (just like Permutation feature selection). It was developed as a package of R(here). There are some simple steps required for this algorithm.
\n: In Boruta the Features compete with the randomized version of themselves. For example, suppose we have a small dataset consisting of 3 features, (age, height, weight) with only 5 entries and we have to predict income.
Now we need to create another DataFrame by shuffling each feature. We will call it Shadow features and then we will concatenate this with the main DataFrame, let's call it X_baruta.
![\"\"](\"https://i.postimg.cc/7PctgcVh/1-z-Stv-S-9-Gp-DEJJFZb0-Av-G4-Q.png\")
: Now we have to fit the model(Random Forest) on X_baruta and y. After training the model, we will take the highest feature importance recorded among the Shadow features and consider it as the threshold value. When the importance of a feature(from the main data frame) is higher than this threshold, we will call it a hit. The idea is that a feature is useful only if it’s capable of doing better than the best-randomized feature. The result is,
![\"\"](\"https://i.postimg.cc/V6DfGtgf/1-bt-H1-CNb6l-Q62zak-Bx-YECYg.png\")
Here, the threshold is 14%(shadow_height). It seems that only age and height are useful for this run.
\nBut we can't trust this one run and determine the importance of the features, so we will run it multiple times.
: Result after 20 trials is,
\n![\"\"](\"https://i.postimg.cc/x8cpzk1p/1-Zly3-ZYol-DDsvn8x8-Me-Ieg.png\")
All features will have only two outcomes: hit or not hit, therefore we can perform the previous step several times and build a binomial distribution out of the features.
: We will keep the features which are in the Green region, we will drop those features in the Red region and we haven't come to a conclusion for features in the Blue region. We can run the algorithm for more trials to reach a conclusion about those indecisive features.
\nSHAP is a unified approach to explain the output of any machine learning model. SHAP connects game theory with local explanations to create the only consistent and accurate explainer. We can use it to compute feature importance for each model. More about it here.
\nWe use this feature importance technique with Boruta algorithm to get a robust feature selection technique.
\nI used Boruta-Shap(XGBOOST as the base model) and was really amazed by the results.
\nThe dataset I use consists of 1328 features. The baseline result of my XGBOOST model was 0.7897781200376643 (using all of the features). But after running this algorithm it only accepted 116 features. I trained the XGBOOST model with the accepted features and with the same hyper-parameters again and I got 0.7893407118088636!
I retained 99.9446163% of the performance by using only 8.7349398% of the features!
\nThe accepted features are here.
\nThe implementation and the results can be found in this notebook: AMEX - BorutaShap Feature Selection
", "messages": [ { "id": 1829070, "postDate": "2022-06-22T09:22:01.520Z", "content": "This is an explanation of the Boruta-Shap Feature Selection.
\nIt is a pretty smart algorithm used for Feature Selection. (just like Permutation feature selection). It was developed as a package of R(here). There are some simple steps required for this algorithm.
\n: In Boruta the Features compete with the randomized version of themselves. For example, suppose we have a small dataset consisting of 3 features, (age, height, weight) with only 5 entries and we have to predict income.
Now we need to create another DataFrame by shuffling each feature. We will call it Shadow features and then we will concatenate this with the main DataFrame, let's call it X_baruta.
: Now we have to fit the model(Random Forest) on X_baruta and y. After training the model, we will take the highest feature importance recorded among the Shadow features and consider it as the threshold value. When the importance of a feature(from the main data frame) is higher than this threshold, we will call it a hit. The idea is that a feature is useful only if it’s capable of doing better than the best-randomized feature. The result is,
Here, the threshold is 14%(shadow_height). It seems that only age and height are useful for this run.
\nBut we can't trust this one run and determine the importance of the features, so we will run it multiple times.
: Result after 20 trials is,
\n
All features will have only two outcomes: hit or not hit, therefore we can perform the previous step several times and build a binomial distribution out of the features.
: We will keep the features which are in the Green region, we will drop those features in the Red region and we haven't come to a conclusion for features in the Blue region. We can run the algorithm for more trials to reach a conclusion about those indecisive features.
\nSHAP is a unified approach to explain the output of any machine learning model. SHAP connects game theory with local explanations to create the only consistent and accurate explainer. We can use it to compute feature importance for each model. More about it here.
\nWe use this feature importance technique with Boruta algorithm to get a robust feature selection technique.
\nI used Boruta-Shap(XGBOOST as the base model) and was really amazed by the results.
\nThe dataset I use consists of 1328 features. The baseline result of my XGBOOST model was 0.7897781200376643 (using all of the features). But after running this algorithm it only accepted 116 features. I trained the XGBOOST model with the accepted features and with the same hyper-parameters again and I got 0.7893407118088636!
I retained 99.9446163% of the performance by using only 8.7349398% of the features!
\nThe accepted features are here.
\nThe implementation and the results can be found in this notebook: AMEX - BorutaShap Feature Selection
", "rawMarkdown": "**This is an explanation of the Boruta-Shap Feature Selection.**\n\n\n## Boruta\nIt is a pretty smart algorithm used for Feature Selection. (just like Permutation feature selection). It was developed as a package of R([here](https://www.jstatsoft.org/article/view/v036i11)). There are some simple steps required for this algorithm.\n\n**Step 1(Shadow Features)**: `In Boruta the Features compete with the randomized version of themselves`. For example, suppose we have a small dataset consisting of 3 features, (`age`, `height`, `weight`) with only 5 entries and we have to predict `income`.\n\n\n\nNow we need to create another DataFrame by shuffling each feature. We will call it `Shadow features` and then we will concatenate this with the main DataFrame, let's call it X_baruta.\n\n\n\n\n**Step 2(Fitting the model)**: Now we have to fit the model(Random Forest) on X_baruta and y. After training the model, we will take the highest feature importance recorded among the Shadow features and consider it as the threshold value. When the importance of a feature(from the main data frame) is higher than this threshold, we will call it a `hit`. **The idea is that a feature is useful only if it’s capable of doing better than the best-randomized feature**. The result is, \n\n\n\nHere, the threshold is 14%(shadow_height). It seems that only `age` and `height` are useful for this run.\nBut we can't trust this one run and determine the importance of the features, so we will run it multiple times. \n\n\n**Step 3(Binomial Distribution)**: Result after 20 trials is, \n\n\n\nAll features will have only two outcomes: `hit` or `not hit`, therefore we can perform the previous step several times and build a binomial distribution out of the features.\n\n\n[Image and Table Source](https://towardsdatascience.com/boruta-explained-the-way-i-wish-someone-explained-it-to-me-4489d70e154a)\n\n**Step 4(Keeping/Dropping Features)**: We will keep the features which are in the Green region, we will drop those features in the Red region and we haven't come to a conclusion for features in the Blue region. We can run the algorithm for more trials to reach a conclusion about those indecisive features.\n\n\n## SHAP (SHapley Additive exPlanations)\n\n[SHAP](https://github.com/slundberg/shap) is a unified approach to explain the output of any machine learning model. SHAP connects game theory with local explanations to create the only consistent and accurate explainer. We can use it to compute feature importance for each model. More about it [here](https://christophm.github.io/interpretable-ml-book/shap.html). \n\nWe use this feature importance technique with Boruta algorithm to get a robust feature selection technique. \n\n## My Results & Conclusion\n\nI used Boruta-Shap(XGBOOST as the base model) and was really amazed by the results. \nThe dataset I use consists of 1328 features. The baseline result of my XGBOOST model was 0.7897781200376643 (using all of the features). But after running this algorithm it only accepted 116 [features](https://www.kaggle.com/datasets/susnato/shapxgboost). I trained the XGBOOST model with the accepted features and with the same hyper-parameters again and I got 0.7893407118088636!\n\n**I retained 99.9446163% of the performance by using only 8.7349398% of the features!**\n\nThe accepted features are [here](https://www.kaggle.com/datasets/susnato/shapxgboost).\n\nThe implementation and the results can be found in this notebook: [AMEX - BorutaShap Feature Selection](https://www.kaggle.com/code/susnato/amex-borutashap-feature-selection/notebook)\n\n", "votes": 19 } ], "comments": [], "raw_markdown_by_id": { "1829070": "**This is an explanation of the Boruta-Shap Feature Selection.**\n\n\n## Boruta\nIt is a pretty smart algorithm used for Feature Selection. (just like Permutation feature selection). It was developed as a package of R([here](https://www.jstatsoft.org/article/view/v036i11)). There are some simple steps required for this algorithm.\n\n**Step 1(Shadow Features)**: `In Boruta the Features compete with the randomized version of themselves`. For example, suppose we have a small dataset consisting of 3 features, (`age`, `height`, `weight`) with only 5 entries and we have to predict `income`.\n\n\n\nNow we need to create another DataFrame by shuffling each feature. We will call it `Shadow features` and then we will concatenate this with the main DataFrame, let's call it X_baruta.\n\n\n\n\n**Step 2(Fitting the model)**: Now we have to fit the model(Random Forest) on X_baruta and y. After training the model, we will take the highest feature importance recorded among the Shadow features and consider it as the threshold value. When the importance of a feature(from the main data frame) is higher than this threshold, we will call it a `hit`. **The idea is that a feature is useful only if it’s capable of doing better than the best-randomized feature**. The result is, \n\n\n\nHere, the threshold is 14%(shadow_height). It seems that only `age` and `height` are useful for this run.\nBut we can't trust this one run and determine the importance of the features, so we will run it multiple times. \n\n\n**Step 3(Binomial Distribution)**: Result after 20 trials is, \n\n\n\nAll features will have only two outcomes: `hit` or `not hit`, therefore we can perform the previous step several times and build a binomial distribution out of the features.\n\n\n[Image and Table Source](https://towardsdatascience.com/boruta-explained-the-way-i-wish-someone-explained-it-to-me-4489d70e154a)\n\n**Step 4(Keeping/Dropping Features)**: We will keep the features which are in the Green region, we will drop those features in the Red region and we haven't come to a conclusion for features in the Blue region. We can run the algorithm for more trials to reach a conclusion about those indecisive features.\n\n\n## SHAP (SHapley Additive exPlanations)\n\n[SHAP](https://github.com/slundberg/shap) is a unified approach to explain the output of any machine learning model. SHAP connects game theory with local explanations to create the only consistent and accurate explainer. We can use it to compute feature importance for each model. More about it [here](https://christophm.github.io/interpretable-ml-book/shap.html). \n\nWe use this feature importance technique with Boruta algorithm to get a robust feature selection technique. \n\n## My Results & Conclusion\n\nI used Boruta-Shap(XGBOOST as the base model) and was really amazed by the results. \nThe dataset I use consists of 1328 features. The baseline result of my XGBOOST model was 0.7897781200376643 (using all of the features). But after running this algorithm it only accepted 116 [features](https://www.kaggle.com/datasets/susnato/shapxgboost). I trained the XGBOOST model with the accepted features and with the same hyper-parameters again and I got 0.7893407118088636!\n\n**I retained 99.9446163% of the performance by using only 8.7349398% of the features!**\n\nThe accepted features are [here](https://www.kaggle.com/datasets/susnato/shapxgboost).\n\nThe implementation and the results can be found in this notebook: [AMEX - BorutaShap Feature Selection](https://www.kaggle.com/code/susnato/amex-borutashap-feature-selection/notebook)\n\n" } }![\"\"](\"https://i.postimg.cc/qvznH8Z5/1-yq-AUl-Mt-PUi-Fyr8g-YLFag-TA.png\")