Amex Metric Score is coming to be 0.99145420 (test data) and 0.999958889 (training data) but the competition public leaderboard score shows 0.57.
\nCan someone help to understand the possible reason behind this?
", "messages": [ { "id": 1907886, "postDate": "2022-08-21T07:04:36.097Z", "content": "Just look into your model's feature importance, you can use Shapley summary (beeswarm) plot for example https://shap.readthedocs.io/en/latest/example_notebooks/api_examples/plots/beeswarm.html to identify most important features, my guess is the most important one, or first top N, contain the leak.
\nOther option is to try to build single feature models and see if any model is much better than it should be.
\nAlso when @imeintanis suggests to have a look at feature names:
\nprint([col for col in X_train.columns if 'target' in col])\nit prints every column that has the substring \"target\" as part of it, for example \"encoded_target\" would match, while your code
\n'target' in S.columns\nwould not work, as it only prints the column name if it is the exact text \"target\".
\nI believe you either leak the target in the features, or if you train an ensemble, you might be training it on train + test and then evaluating on test.
\nGood luck with finding the issue.
", "rawMarkdown": "Just look into your model's feature importance, you can use Shapley summary (beeswarm) plot for example https://shap.readthedocs.io/en/latest/example_notebooks/api_examples/plots/beeswarm.html to identify most important features, my guess is the most important one, or first top N, contain the leak.\n\nOther option is to try to build single feature models and see if any model is much better than it should be.\n\nAlso when @imeintanis suggests to have a look at feature names:\n```\nprint([col for col in X_train.columns if 'target' in col])\n```\nit prints every column that has the substring \"target\" as part of it, for example \"encoded_target\" would match, while your code\n```\n'target' in S.columns\n```\nwould not work, as it only prints the column name if it is the exact text \"target\".\n\nI believe you either leak the target in the features, or if you train an ensemble, you might be training it on train + test and then evaluating on test.\n\nGood luck with finding the issue.", "votes": 5, "replies": [ { "id": 1908042, "postDate": "2022-08-21T10:00:41.717Z", "content": "Exactly!! Thank you for making it clear
\nI bet it contains somethink like \"target_diff\", \"target_mean\", etc or \"target_x\" after multiple merges
", "rawMarkdown": "Exactly!! Thank you for making it clear\n\nI bet it contains somethink like \"target_diff\", \"target_mean\", etc or \"target_x\" after multiple merges", "votes": 1 }, { "id": 1908084, "postDate": "2022-08-21T10:56:45.957Z", "content": "Thank you for your responses.
\nTarget has not been chased at all to contain any column like these.
\nTrain and test data are also separate.
Maybe the issue is multicollinear columns.
", "rawMarkdown": "Thank you for your responses. \nTarget has not been chased at all to contain any column like these.\nTrain and test data are also separate. \n\nMaybe the issue is multicollinear columns." }, { "id": 1908086, "postDate": "2022-08-21T10:59:22.967Z", "content": "print([col for col in X_train.columns if 'target' in col])
\nOutput for this is an empty list.
Do you have target-encoding features in your training data?
", "rawMarkdown": "Do you have target-encoding features in your training data?", "votes": 1, "replies": [ { "id": 1907594, "postDate": "2022-08-20T23:39:33.850Z", "content": "Hey,
\nno, haven't tried target-encoding.
\nBut have done one hot encoding.
U might be having target info in your training data.
", "rawMarkdown": "U might be having target info in your training data. ", "votes": 1, "replies": [ { "id": 1907397, "postDate": "2022-08-20T18:22:48.970Z", "content": "No, target data has been removed before splitting the dataset.
", "rawMarkdown": "No, target data has been removed before splitting the dataset." }, { "id": 1907558, "postDate": "2022-08-20T22:28:26.490Z", "content": "Certainly you have target col (or derivative) in your features, search better.. I'd suggest to print out and check your features again
", "rawMarkdown": "Certainly you have target col (or derivative) in your features, search better.. I'd suggest to print out and check your features again", "votes": 1 }, { "id": 1907595, "postDate": "2022-08-20T23:40:51.737Z", "content": "target col has been removed from X_train and X_test.
", "rawMarkdown": "target col has been removed from X_train and X_test. " }, { "id": 1907661, "postDate": "2022-08-21T00:46:04.887Z", "content": "I bet some target info is there! check each feature you have created one by one!
\nor use this print statement just before you pass it to fit
\nprint([col for col in X_train.columns if 'target' in col])
PS: there is no \"real\" model that can score 99% :)
", "rawMarkdown": "I bet some target info is there! check each feature you have created one by one! \nor use this print statement just before you pass it to fit\n`print([col for col in X_train.columns if 'target' in col])`\n\nPS: there is no \"real\" model that can score 99% :) ", "votes": 1 }, { "id": 1907669, "postDate": "2022-08-21T00:53:29.473Z", "content": "I ran this statement.
\n'target' in S.columns
Output: False
\nS is my training data.
Maybe you have changed the name or something else? Search for the values in your dataframe. Maybe you have used the target column and aggregated it within a different feature etc. etc.
\nThere is no way to score this high and that low on public lb
\nGood luck!
", "rawMarkdown": "Maybe you have changed the name or something else? Search for the values in your dataframe. Maybe you have used the target column and aggregated it within a different feature etc. etc.\n\nThere is no way to score this high and that low on public lb\n\nGood luck!" } ] }, { "id": 1907189, "postDate": "2022-08-20T15:08:11.087Z", "content": "Looks like you have a leak in your training. Is the target in your training data?
", "rawMarkdown": "Looks like you have a leak in your training. Is the target in your training data?", "votes": 1, "replies": [ { "id": 1907398, "postDate": "2022-08-20T18:23:34.570Z", "content": "No, target is not part of X_train and X_test
", "rawMarkdown": "No, target is not part of X_train and X_test" }, { "id": 1907633, "postDate": "2022-08-21T00:13:44.700Z", "content": "You definitively have a leak somewhere in your pipeline. 0.99 is impossible to reach on the test set.
", "rawMarkdown": "You definitively have a leak somewhere in your pipeline. 0.99 is impossible to reach on the test set.", "votes": 2 }, { "id": 1907639, "postDate": "2022-08-21T00:24:06.733Z", "content": "Trying to figure it out.
", "rawMarkdown": "Trying to figure it out." } ] }, { "id": 1907671, "postDate": "2022-08-21T00:54:48.403Z", "content": "I ran this statement.
\n'target' in S.columns
Output: False
\nS is my training data.
Dump out the column names in your dataset to a CSV dataset and go through them one by one.
", "rawMarkdown": "Dump out the column names in your dataset to a CSV dataset and go through them one by one." } ] }, { "id": 1907177, "postDate": "2022-08-20T14:58:18.180Z", "content": "Amex Metric Score is coming to be 0.99145420 (test data) and 0.999958889 (training data) but the competition public leaderboard score shows 0.57.
\nCan someone help to understand the possible reason behind this?
", "rawMarkdown": "Amex Metric Score is coming to be 0.99145420 (test data) and 0.999958889 (training data) but the competition public leaderboard score shows 0.57.\n\nCan someone help to understand the possible reason behind this?", "votes": -5 }, { "id": 1907906, "postDate": "2022-08-21T07:16:42.650Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1907886, "author_name": "MartinBarus", "author_url": "", "post_date": "2022-08-21T07:04:36.097000", "content": "Just look into your model's feature importance, you can use Shapley summary (beeswarm) plot for example https://shap.readthedocs.io/en/latest/example_notebooks/api_examples/plots/beeswarm.html to identify most important features, my guess is the most important one, or first top N, contain the leak.
\nOther option is to try to build single feature models and see if any model is much better than it should be.
\nAlso when @imeintanis suggests to have a look at feature names:
\nprint([col for col in X_train.columns if 'target' in col])\nit prints every column that has the substring \"target\" as part of it, for example \"encoded_target\" would match, while your code
\n'target' in S.columns\nwould not work, as it only prints the column name if it is the exact text \"target\".
\nI believe you either leak the target in the features, or if you train an ensemble, you might be training it on train + test and then evaluating on test.
\nGood luck with finding the issue.
", "votes": 5, "replies": [ { "id": 1908042, "author_name": "Ioannis M", "author_url": "", "post_date": "2022-08-21T10:00:41.717000", "content": "Exactly!! Thank you for making it clear
\nI bet it contains somethink like \"target_diff\", \"target_mean\", etc or \"target_x\" after multiple merges
", "votes": 1, "replies": [] }, { "id": 1908084, "author_name": "Neha", "author_url": "", "post_date": "2022-08-21T10:56:45.957000", "content": "Thank you for your responses.
\nTarget has not been chased at all to contain any column like these.
\nTrain and test data are also separate.
Maybe the issue is multicollinear columns.
", "votes": 0, "replies": [] }, { "id": 1908086, "author_name": "Neha", "author_url": "", "post_date": "2022-08-21T10:59:22.967000", "content": "print([col for col in X_train.columns if 'target' in col])
\nOutput for this is an empty list.
Do you have target-encoding features in your training data?
", "votes": 1, "replies": [ { "id": 1907594, "author_name": "Neha", "author_url": "", "post_date": "2022-08-20T23:39:33.850000", "content": "Hey,
\nno, haven't tried target-encoding.
\nBut have done one hot encoding.
U might be having target info in your training data.
", "votes": 1, "replies": [ { "id": 1907397, "author_name": "Neha", "author_url": "", "post_date": "2022-08-20T18:22:48.970000", "content": "No, target data has been removed before splitting the dataset.
", "votes": 0, "replies": [] }, { "id": 1907558, "author_name": "Ioannis M", "author_url": "", "post_date": "2022-08-20T22:28:26.490000", "content": "Certainly you have target col (or derivative) in your features, search better.. I'd suggest to print out and check your features again
", "votes": 1, "replies": [] }, { "id": 1907595, "author_name": "Neha", "author_url": "", "post_date": "2022-08-20T23:40:51.737000", "content": "target col has been removed from X_train and X_test.
", "votes": 0, "replies": [] }, { "id": 1907661, "author_name": "Ioannis M", "author_url": "", "post_date": "2022-08-21T00:46:04.887000", "content": "I bet some target info is there! check each feature you have created one by one!
\nor use this print statement just before you pass it to fit
\nprint([col for col in X_train.columns if 'target' in col])
PS: there is no \"real\" model that can score 99% :)
", "votes": 1, "replies": [] }, { "id": 1907669, "author_name": "Neha", "author_url": "", "post_date": "2022-08-21T00:53:29.473000", "content": "I ran this statement.
\n'target' in S.columns
Output: False
\nS is my training data.
Maybe you have changed the name or something else? Search for the values in your dataframe. Maybe you have used the target column and aggregated it within a different feature etc. etc.
\nThere is no way to score this high and that low on public lb
\nGood luck!
", "votes": 0, "replies": [] } ] }, { "id": 1907189, "author_name": "Jake", "author_url": "", "post_date": "2022-08-20T15:08:11.087000", "content": "Looks like you have a leak in your training. Is the target in your training data?
", "votes": 1, "replies": [ { "id": 1907398, "author_name": "Neha", "author_url": "", "post_date": "2022-08-20T18:23:34.570000", "content": "No, target is not part of X_train and X_test
", "votes": 0, "replies": [] }, { "id": 1907633, "author_name": "Fritz Cremer", "author_url": "", "post_date": "2022-08-21T00:13:44.700000", "content": "You definitively have a leak somewhere in your pipeline. 0.99 is impossible to reach on the test set.
", "votes": 2, "replies": [] }, { "id": 1907639, "author_name": "Neha", "author_url": "", "post_date": "2022-08-21T00:24:06.733000", "content": "Trying to figure it out.
", "votes": 0, "replies": [] } ] }, { "id": 1907671, "author_name": "Neha", "author_url": "", "post_date": "2022-08-21T00:54:48.403000", "content": "I ran this statement.
\n'target' in S.columns
Output: False
\nS is my training data.
Dump out the column names in your dataset to a CSV dataset and go through them one by one.
", "votes": 0, "replies": [] } ] }, { "id": 1907906, "author_name": "", "author_url": "", "post_date": "2022-08-21T07:16:42.650000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1907886": "Just look into your model's feature importance, you can use Shapley summary (beeswarm) plot for example https://shap.readthedocs.io/en/latest/example_notebooks/api_examples/plots/beeswarm.html to identify most important features, my guess is the most important one, or first top N, contain the leak.\n\nOther option is to try to build single feature models and see if any model is much better than it should be.\n\nAlso when @imeintanis suggests to have a look at feature names:\n```\nprint([col for col in X_train.columns if 'target' in col])\n```\nit prints every column that has the substring \"target\" as part of it, for example \"encoded_target\" would match, while your code\n```\n'target' in S.columns\n```\nwould not work, as it only prints the column name if it is the exact text \"target\".\n\nI believe you either leak the target in the features, or if you train an ensemble, you might be training it on train + test and then evaluating on test.\n\nGood luck with finding the issue.", "1907537": "Do you have target-encoding features in your training data?", "1907324": "U might be having target info in your training data. ", "1907189": "Looks like you have a leak in your training. Is the target in your training data?", "1907671": "I ran this statement.\n'target' in S.columns\n\nOutput: False\nS is my training data.", "1907177": "Amex Metric Score is coming to be 0.99145420 (test data) and 0.999958889 (training data) but the competition public leaderboard score shows 0.57.\n\nCan someone help to understand the possible reason behind this?", "1907906": "" } }