For each group_level part, I use corresponding data to make feature engineering and train models(without historical and future informations or any encoding). What I am confused is that lesser features I use, higher CV it is, but lower LB score as result. Some examples are below:
\n| origin features(each level) | \nCV | \nmost important features(each level) | \nCV | \nLB | \n
|---|---|---|---|---|
| 300,500,600 | \n0.6933 | \nnot reduced | \n\n | 0.693 | \n
| 300,500,600 | \n0.6935 | \n200,300,400 | \n0.6943 | \n0.692 | \n
| 500,800,1000 | \n0.6935 | \n100,110,120 | \n0.6958 | \n0.691 | \n
How could this be?
", "messages": [ { "id": "2150121", "postDate": "02/19/2023 01:07:18", "content": "For each group_level part, I use corresponding data to make feature engineering and train models(without historical and future informations or any encoding). What I am confused is that lesser features I use, higher CV it is, but lower LB score as result. Some examples are below:
\n| origin features(each level) | \nCV | \nmost important features(each level) | \nCV | \nLB | \n
|---|---|---|---|---|
| 300,500,600 | \n0.6933 | \nnot reduced | \n\n | 0.693 | \n
| 300,500,600 | \n0.6935 | \n200,300,400 | \n0.6943 | \n0.692 | \n
| 500,800,1000 | \n0.6935 | \n100,110,120 | \n0.6958 | \n0.691 | \n
How could this be?
", "rawMarkdown": "For each group_level part, I use corresponding data to make feature engineering and train models(without historical and future informations or any encoding). What I am confused is that lesser features I use, higher CV it is, but lower LB score as result. Some examples are below:\n| origin features(each level) | CV | most important features(each level) | CV | LB |\n| --- | --- | --- | --- | --- |\n| 300,500,600 | 0.6933 | not reduced | | 0.693 |\n| 300,500,600 | 0.6935 | 200,300,400 | 0.6943 | 0.692 |\n| 500,800,1000 | 0.6935 | 100,110,120 | 0.6958 | 0.691 |\n\nHow could this be?", "votes": null }, { "id": "2150127", "postDate": "02/19/2023 01:10:34", "content": "It could be a CV leak. How do you choose your features? If you use information from all folds to select features for fold 1, then you are leaking the OOF targets of fold 1 by using feature selection information from folds 2,3,4,5
", "rawMarkdown": "It could be a CV leak. How do you choose your features? If you use information from all folds to select features for fold 1, then you are leaking the OOF targets of fold 1 by using feature selection information from folds 2,3,4,5", "votes": null }, { "id": "2150134", "postDate": "02/19/2023 01:16:36", "content": "Hi Chris, I don't create features about targets, only about something like the elapsed times and index numbers between two events.
", "rawMarkdown": "Hi Chris, I don't create features about targets, only about something like the elapsed times and index numbers between two events.", "votes": null }, { "id": "2150140", "postDate": "02/19/2023 01:20:13", "content": "Yes, but how do you pick the top 100 features?
", "rawMarkdown": "Yes, but how do you pick the top 100 features?", "votes": null }, { "id": "2150141", "postDate": "02/19/2023 01:20:35", "content": "Do you use feature importance from folds 2,3,4,5 to help select features for fold 1? If so, that is a leak.
", "rawMarkdown": "Do you use feature importance from folds 2,3,4,5 to help select features for fold 1? If so, that is a leak.", "votes": null }, { "id": "2150148", "postDate": "02/19/2023 01:27:02", "content": "Oh it is. I used the average feature importance values to pick 'top' 100 features.
\nin each fold:
fold_importance_df = pd.DataFrame()\nfold_importance_df[\"feature\"] = FEATURES\nfold_importance_df[\"importance\"] = clf.feature_importances_\nfold_importance_df[\"fold\"] = i + 1\nfeature_importance_df = pd.concat([feature_importance_df, fold_importance_df], axis=0)\nthen:
\nfeature_importance_df = feature_importance_df.groupby(['feature'])['importance'].agg(['mean']).sort_values(by='mean', ascending=False)\ndisplay(feature_importance_df.head(10))\nFeature selection has lots of opportunities for introducing leaks. For example if we do permutation importance, that is using the targets to pick features. So it results in an optimistic CV score. With all methods we must be careful that we aren't using too much information from targets.
", "rawMarkdown": "Feature selection has lots of opportunities for introducing leaks. For example if we do permutation importance, that is using the targets to pick features. So it results in an optimistic CV score. With all methods we must be careful that we aren't using too much information from targets.", "votes": null }, { "id": "2180935", "postDate": "03/14/2023 07:35:14", "content": "I faced the same situation. Thanks so much, Chris and Joseph!! I learn so much from your discussion.
", "rawMarkdown": "I faced the same situation. Thanks so much, Chris and Joseph!! I learn so much from your discussion.", "votes": null }, { "id": "2193197", "postDate": "03/23/2023 06:14:36", "content": "hi Chris, if the above method leads to cv leakage as you said, then how to select features in cv that will not lead to cv leakage, or is there any other better method? Thank you very much for your reply
", "rawMarkdown": "hi Chris, if the above method leads to cv leakage as you said, then how to select features in cv that will not lead to cv leakage, or is there any other better method? Thank you very much for your reply", "votes": null } ], "comments": [ { "id": 2150127, "author_name": "cdeotte", "author_url": "", "post_date": "02/19/2023 01:10:34", "content": "It could be a CV leak. How do you choose your features? If you use information from all folds to select features for fold 1, then you are leaking the OOF targets of fold 1 by using feature selection information from folds 2,3,4,5
", "votes": null, "replies": [ { "id": 2150134, "author_name": "takanashihumbert", "author_url": "", "post_date": "02/19/2023 01:16:36", "content": "Hi Chris, I don't create features about targets, only about something like the elapsed times and index numbers between two events.
", "votes": null, "replies": [ { "id": 2150140, "author_name": "cdeotte", "author_url": "", "post_date": "02/19/2023 01:20:13", "content": "Yes, but how do you pick the top 100 features?
", "votes": null, "replies": [] }, { "id": 2150141, "author_name": "cdeotte", "author_url": "", "post_date": "02/19/2023 01:20:35", "content": "Do you use feature importance from folds 2,3,4,5 to help select features for fold 1? If so, that is a leak.
", "votes": null, "replies": [ { "id": 2150148, "author_name": "takanashihumbert", "author_url": "", "post_date": "02/19/2023 01:27:02", "content": "Oh it is. I used the average feature importance values to pick 'top' 100 features.
\nin each fold:
fold_importance_df = pd.DataFrame()\nfold_importance_df[\"feature\"] = FEATURES\nfold_importance_df[\"importance\"] = clf.feature_importances_\nfold_importance_df[\"fold\"] = i + 1\nfeature_importance_df = pd.concat([feature_importance_df, fold_importance_df], axis=0)\nthen:
\nfeature_importance_df = feature_importance_df.groupby(['feature'])['importance'].agg(['mean']).sort_values(by='mean', ascending=False)\ndisplay(feature_importance_df.head(10))\nFeature selection has lots of opportunities for introducing leaks. For example if we do permutation importance, that is using the targets to pick features. So it results in an optimistic CV score. With all methods we must be careful that we aren't using too much information from targets.
", "votes": null, "replies": [] } ] }, { "id": 2193197, "author_name": "shiyusha", "author_url": "", "post_date": "03/23/2023 06:14:36", "content": "hi Chris, if the above method leads to cv leakage as you said, then how to select features in cv that will not lead to cv leakage, or is there any other better method? Thank you very much for your reply
", "votes": null, "replies": [] } ] }, { "id": 2180935, "author_name": "mengvision", "author_url": "", "post_date": "03/14/2023 07:35:14", "content": "I faced the same situation. Thanks so much, Chris and Joseph!! I learn so much from your discussion.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2150121": "For each group_level part, I use corresponding data to make feature engineering and train models(without historical and future informations or any encoding). What I am confused is that lesser features I use, higher CV it is, but lower LB score as result. Some examples are below:\n| origin features(each level) | CV | most important features(each level) | CV | LB |\n| --- | --- | --- | --- | --- |\n| 300,500,600 | 0.6933 | not reduced | | 0.693 |\n| 300,500,600 | 0.6935 | 200,300,400 | 0.6943 | 0.692 |\n| 500,800,1000 | 0.6935 | 100,110,120 | 0.6958 | 0.691 |\n\nHow could this be?", "2150127": "It could be a CV leak. How do you choose your features? If you use information from all folds to select features for fold 1, then you are leaking the OOF targets of fold 1 by using feature selection information from folds 2,3,4,5", "2150134": "Hi Chris, I don't create features about targets, only about something like the elapsed times and index numbers between two events.", "2150140": "Yes, but how do you pick the top 100 features?", "2150141": "Do you use feature importance from folds 2,3,4,5 to help select features for fold 1? If so, that is a leak.", "2150148": "Oh it is. I used the average feature importance values to pick 'top' 100 features.\nin each fold:\n```\nfold_importance_df = pd.DataFrame()\nfold_importance_df[\"feature\"] = FEATURES\nfold_importance_df[\"importance\"] = clf.feature_importances_\nfold_importance_df[\"fold\"] = i + 1\nfeature_importance_df = pd.concat([feature_importance_df, fold_importance_df], axis=0)\n```\nthen:\n```\nfeature_importance_df = feature_importance_df.groupby(['feature'])['importance'].agg(['mean']).sort_values(by='mean', ascending=False)\ndisplay(feature_importance_df.head(10))\n```", "2150149": "Feature selection has lots of opportunities for introducing leaks. For example if we do permutation importance, that is using the targets to pick features. So it results in an optimistic CV score. With all methods we must be careful that we aren't using too much information from targets.", "2180935": "I faced the same situation. Thanks so much, Chris and Joseph!! I learn so much from your discussion.", "2193197": "hi Chris, if the above method leads to cv leakage as you said, then how to select features in cv that will not lead to cv leakage, or is there any other better method? Thank you very much for your reply" }, "source": "meta" }