![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F471225%2F2ad61716f63933662f3e1a644ed06bf3%2FPermImp.png?generation=1657550240342914&alt=media\")
I found the discussion post on feature importance by @ambrosm very interesting. So I wanted to try to implement permutation feature importance and see the results
\nI have explored this in my notebook and it looks like there is potential to improve on the baseline
\nhttps://www.kaggle.com/code/illidan7/amex-permutation-feature-importance
If we consider the XGB split feature importance from @cdeotte 's XGB Starter notebook, we see that feature 'P_3_last' is the 15th most important feature
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F471225%2Faa205a644a05e6bf8dbeb0136d908fed%2FXGBImp.png?generation=1657550262055064&alt=media\")
However, based on the results from my notebook
\n137 P_3_last 0.791860
\n367 BASELINE 0.791720
\nWe can see that the score can potentially improve by not using this variable
\nVery similar exploration to that done by @ambrosm. Just sharing my own observations here
\nPlease check out the following links:
\nhttps://www.kaggle.com/competitions/amex-default-prediction/discussion/331131
\nhttps://www.kaggle.com/code/cdeotte/xgboost-starter-0-793
\nhttps://www.kaggle.com/code/cdeotte/lstm-feature-importance
\nhttps://www.kaggle.com/datasets/raddar/amex-data-integer-dtypes-parquet-format
I found the discussion post on feature importance by @ambrosm very interesting. So I wanted to try to implement permutation feature importance and see the results
\nI have explored this in my notebook and it looks like there is potential to improve on the baseline
\nhttps://www.kaggle.com/code/illidan7/amex-permutation-feature-importance
If we consider the XGB split feature importance from @cdeotte 's XGB Starter notebook, we see that feature 'P_3_last' is the 15th most important feature
\n
However, based on the results from my notebook
\n137 P_3_last 0.791860
\n367 BASELINE 0.791720
\nWe can see that the score can potentially improve by not using this variable
\nVery similar exploration to that done by @ambrosm. Just sharing my own observations here
\nPlease check out the following links:
\nhttps://www.kaggle.com/competitions/amex-default-prediction/discussion/331131
\nhttps://www.kaggle.com/code/cdeotte/xgboost-starter-0-793
\nhttps://www.kaggle.com/code/cdeotte/lstm-feature-importance
\nhttps://www.kaggle.com/datasets/raddar/amex-data-integer-dtypes-parquet-format
This is great! thanks for sharing your work!
", "rawMarkdown": "This is great! thanks for sharing your work!", "votes": 1 }, { "id": 1850995, "postDate": "2022-07-11T01:05:03.453Z", "content": "Nice work! 2 questions:
\nYeah choosing features to eliminate/improve the model is definitely proving tricky. I have not specifically tried to retrain using these results and this set of features. I am trying different feature engineering ideas at the moment
\nI have not tried to apply it to more than one fold and compare results. But that could be useful too
\nI found this discussion interesting where @cdeotte has explained an approach to selecting features which impact by more than 2x standard deviation of CV score change based on different seeds
\nhttps://www.kaggle.com/competitions/amex-default-prediction/discussion/336145
", "rawMarkdown": "Yeah choosing features to eliminate/improve the model is definitely proving tricky. I have not specifically tried to retrain using these results and this set of features. I am trying different feature engineering ideas at the moment\n\nI have not tried to apply it to more than one fold and compare results. But that could be useful too\n\nI found this discussion interesting where @cdeotte has explained an approach to selecting features which impact by more than 2x standard deviation of CV score change based on different seeds\n\nhttps://www.kaggle.com/competitions/amex-default-prediction/discussion/336145" } ] }, { "id": 1849911, "postDate": "2022-07-09T23:41:34.997Z", "content": "Thanks for sharing your observations!
\nThe images seems broken on your post..
Thanks for pointing it out. Looks ok on my end now!
", "rawMarkdown": "Thanks for pointing it out. Looks ok on my end now!" } ] }, { "id": 1848439, "postDate": "2022-07-08T16:17:59.030Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1848850, "postDate": "2022-07-09T01:34:59.587Z", "content": "Thank you @illidan7
", "rawMarkdown": "Thank you @illidan7 ", "votes": 1 } ], "comments": [ { "id": 1851357, "author_name": "1110Ra", "author_url": "", "post_date": "2022-07-11T07:26:00.370000", "content": "This is great! thanks for sharing your work!
", "votes": 1, "replies": [] }, { "id": 1850995, "author_name": "k3vla", "author_url": "", "post_date": "2022-07-11T01:05:03.453000", "content": "Nice work! 2 questions:
\nYeah choosing features to eliminate/improve the model is definitely proving tricky. I have not specifically tried to retrain using these results and this set of features. I am trying different feature engineering ideas at the moment
\nI have not tried to apply it to more than one fold and compare results. But that could be useful too
\nI found this discussion interesting where @cdeotte has explained an approach to selecting features which impact by more than 2x standard deviation of CV score change based on different seeds
\nhttps://www.kaggle.com/competitions/amex-default-prediction/discussion/336145
", "votes": 0, "replies": [] } ] }, { "id": 1849911, "author_name": "The Devastator", "author_url": "", "post_date": "2022-07-09T23:41:34.997000", "content": "Thanks for sharing your observations!
\nThe images seems broken on your post..
Thanks for pointing it out. Looks ok on my end now!
", "votes": 0, "replies": [] } ] }, { "id": 1848439, "author_name": "", "author_url": "", "post_date": "2022-07-08T16:17:59.030000", "content": "", "votes": 0, "replies": [] }, { "id": 1848850, "author_name": "Muhammed Tausif", "author_url": "", "post_date": "2022-07-09T01:34:59.587000", "content": "Thank you @illidan7
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "1848429": "I found the discussion post on feature importance by @ambrosm very interesting. So I wanted to try to implement permutation feature importance and see the results\n\nI have explored this in my notebook and it looks like there is potential to improve on the baseline\nhttps://www.kaggle.com/code/illidan7/amex-permutation-feature-importance\n\n\n\n\n\nIf we consider the XGB split feature importance from @cdeotte 's XGB Starter notebook, we see that feature 'P_3_last' is the 15th most important feature\n\n\n\n\nHowever, based on the results from my notebook\n\n137 P_3_last 0.791860\n\n367 BASELINE 0.791720\n\nWe can see that the score can potentially improve by not using this variable\n\nVery similar exploration to that done by @ambrosm. Just sharing my own observations here\n\nPlease check out the following links:\n\nhttps://www.kaggle.com/competitions/amex-default-prediction/discussion/331131\nhttps://www.kaggle.com/code/cdeotte/xgboost-starter-0-793\nhttps://www.kaggle.com/code/cdeotte/lstm-feature-importance\nhttps://www.kaggle.com/datasets/raddar/amex-data-integer-dtypes-parquet-format", "1851357": "This is great! thanks for sharing your work!", "1850995": "Nice work! 2 questions:\n- Did you try retraining without these features? I also wonder what threshold we should use to decide to keep/reject features.\nThe best boost in your example is D_52_std: +0.000716538. It is less than the variation we get just by changing the seed so it's tricky.\n- Do you think we should apply this feature selection to each fold independently? Or get a common list to apply to each fold?", "1849911": "Thanks for sharing your observations!\nThe images seems broken on your post.. \n", "1848439": "", "1848850": "Thank you @illidan7 " } }