{ "id": 548096, "title": "Note that feature Importance quite different from correlations", "url": "/competitions/child-mind-institute-problematic-internet-use/discussion/548096", "author_name": "", "post_date": "2024-11-25T06:00:40.105002700Z", "votes": 2, "comment_count": 1, "views": 0, "content": "

Features in tree based models are more powerful than linear relationships as in correlations.

\n

I get these as the three most important features across lgb, xgb, and catboost:

\n
    \n
  1. SDS-SDS_Total_Raw
    2. \n
  2. PAQ_C-PAQ_C_Total
    3. \n
  3. FGC-FGC_CU
    4. \n
", "messages": [ { "id": "3054769", "postDate": "11/25/2024 06:00:40", "content": "

Features in tree based models are more powerful than linear relationships as in correlations.

\n

I get these as the three most important features across lgb, xgb, and catboost:

\n
    \n
  1. SDS-SDS_Total_Raw
    2. \n
  2. PAQ_C-PAQ_C_Total
    3. \n
  3. FGC-FGC_CU
    4. \n
", "rawMarkdown": "Features in tree based models are more powerful than linear relationships as in correlations.\n\nI get these as the three most important features across lgb, xgb, and catboost: \n1. SDS-SDS_Total_Raw\n2. PAQ_C-PAQ_C_Total\n3. FGC-FGC_CU", "votes": null }, { "id": "3062729", "postDate": "12/03/2024 21:18:33", "content": "

That's a good point about the difference between the correlation of a feature with the target, vs that feature's importance in the model -- valuable features may have low correlations.

\n

Another \"importance\" distinction can be made between the usual \"feature importance\", see the example from an XGB model below, and the \"permutation importance\", mentioned in @gkitchen 's Problematic Internet Usage notebook. Example code and permutation importance output for the same XGB model are also shown below.

\n

I don't know details of these, but the \"feature importance\" may indicate the importance to the model in making decisions -- this would include features that the model uses to overfit the data. On the other hand, the \"permutation importance\" might be a better indicator of a real relationship between the feature and target. If you are more familiar with these importances, let me/us know 🙏 🙂

\n

\"\"

\n
\nperm = PermutationImportance(xgbregr, =None, =9, =6,  # =  number\n                             =None, =).fit(X,y)\neli5.show_weights(perm, =None, feature_names = X.columns.tolist())\n
\n

\"\"🙂🙂

", "rawMarkdown": "That's a good point about the difference between the correlation of a feature with the target, vs that feature's importance in the model -- valuable features may have low correlations.\n\nAnother \"importance\" distinction can be made between the usual \"feature importance\", see the example from an XGB model below, and the [\"permutation importance\"](https://eli5.readthedocs.io/en/latest/blackbox/permutation_importance.html), mentioned in @gkitchen 's [Problematic Internet Usage](https://www.kaggle.com/code/gkitchen/problematic-internet-usage) notebook. Example code and permutation importance output for the same XGB model are also shown below.\n\nI don't know details of these, but the \"feature importance\" may indicate the importance to the model in making decisions -- this would include features that the model uses to overfit the data. On the other hand, the \"permutation importance\" might be a better indicator of a real relationship between the feature and target. If you are more familiar with these importances, let me/us know 🙏 🙂\n\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1496178%2F1ca8d0cf9f6a1c55af710690138e3fb6%2FXGB_importance.png?generation=1733258873003551&alt=media) \n\n```\n# Use CV to get \"generalization\" importance\nperm = PermutationImportance(xgbregr, scoring=None, n_iter=9, cv=6, # cv='prefit' or number\n random_state=None, refit=True).fit(X,y)\neli5.show_weights(perm, top=None, feature_names = X.columns.tolist())\n```\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1496178%2F074cc762cca03158dcc8769e5789f232%2FPermu_Import.png?generation=1733258891367467&alt=media)🙂🙂", "votes": null } ], "comments": [ { "id": 3062729, "author_name": "dan3dewey", "author_url": "", "post_date": "12/03/2024 21:18:33", "content": "

That's a good point about the difference between the correlation of a feature with the target, vs that feature's importance in the model -- valuable features may have low correlations.

\n

Another \"importance\" distinction can be made between the usual \"feature importance\", see the example from an XGB model below, and the \"permutation importance\", mentioned in @gkitchen 's Problematic Internet Usage notebook. Example code and permutation importance output for the same XGB model are also shown below.

\n

I don't know details of these, but the \"feature importance\" may indicate the importance to the model in making decisions -- this would include features that the model uses to overfit the data. On the other hand, the \"permutation importance\" might be a better indicator of a real relationship between the feature and target. If you are more familiar with these importances, let me/us know 🙏 🙂

\n

\"\"

\n
\nperm = PermutationImportance(xgbregr, =None, =9, =6,  # =  number\n                             =None, =).fit(X,y)\neli5.show_weights(perm, =None, feature_names = X.columns.tolist())\n
\n

\"\"🙂🙂

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3054769": "Features in tree based models are more powerful than linear relationships as in correlations.\n\nI get these as the three most important features across lgb, xgb, and catboost: \n1. SDS-SDS_Total_Raw\n2. PAQ_C-PAQ_C_Total\n3. FGC-FGC_CU", "3062729": "That's a good point about the difference between the correlation of a feature with the target, vs that feature's importance in the model -- valuable features may have low correlations.\n\nAnother \"importance\" distinction can be made between the usual \"feature importance\", see the example from an XGB model below, and the [\"permutation importance\"](https://eli5.readthedocs.io/en/latest/blackbox/permutation_importance.html), mentioned in @gkitchen 's [Problematic Internet Usage](https://www.kaggle.com/code/gkitchen/problematic-internet-usage) notebook. Example code and permutation importance output for the same XGB model are also shown below.\n\nI don't know details of these, but the \"feature importance\" may indicate the importance to the model in making decisions -- this would include features that the model uses to overfit the data. On the other hand, the \"permutation importance\" might be a better indicator of a real relationship between the feature and target. If you are more familiar with these importances, let me/us know 🙏 🙂\n\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1496178%2F1ca8d0cf9f6a1c55af710690138e3fb6%2FXGB_importance.png?generation=1733258873003551&alt=media) \n\n```\n# Use CV to get \"generalization\" importance\nperm = PermutationImportance(xgbregr, scoring=None, n_iter=9, cv=6, # cv='prefit' or number\n random_state=None, refit=True).fit(X,y)\neli5.show_weights(perm, top=None, feature_names = X.columns.tolist())\n```\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1496178%2F074cc762cca03158dcc8769e5789f232%2FPermu_Import.png?generation=1733258891367467&alt=media)🙂🙂" }, "source": "meta" }