OK. Maybe not all you need. But they do improve LightGBM.
\nI tried a simple experiment that worked pretty well:
\nIn this competition, we get information about clients of AMEX over time.
\nMost high-scoring notebooks in this competition focused on aggregating the information per client and creating a single row of extracted features: One for each client.
One of such agg functions is last.
A quick examination revealed that the last feature is extremely powerful for predicting if a client defaults or not (well.. makes sense..).
So I took this concept two steps further:
\nlast feature: I added a first feature. first, last pair:This improved my LightGBM model to the point that it overtook my whole LightGBM + Catboost + XGB ensemble.
I uploaded a dataset containing the extracted lag features and updated the final model predictions (only LightGBM this time) for everyone to play with.
Next Experiment (currently running):
\nMore \"lag features\" variations - Taking into consideration other parts of the time series.
\nwill keep you updated.
\n", "messages": [ { "id": 1845782, "postDate": "2022-07-06T15:16:01.410Z", "content": "Credits: The model is based on this amazing notebook.
\n
OK. Maybe not all you need. But they do improve LightGBM.
\nI tried a simple experiment that worked pretty well:
\nIn this competition, we get information about clients of AMEX over time.
\nMost high-scoring notebooks in this competition focused on aggregating the information per client and creating a single row of extracted features: One for each client.
One of such agg functions is last.
A quick examination revealed that the last feature is extremely powerful for predicting if a client defaults or not (well.. makes sense..).
So I took this concept two steps further:
\nlast feature: I added a first feature. first, last pair:This improved my LightGBM model to the point that it overtook my whole LightGBM + Catboost + XGB ensemble.
I uploaded a dataset containing the extracted lag features and updated the final model predictions (only LightGBM this time) for everyone to play with.
Next Experiment (currently running):
\nMore \"lag features\" variations - Taking into consideration other parts of the time series.
\nwill keep you updated.
\n", "rawMarkdown": "## Lag Features ~~Are All~~ You Need\n\n\nOK. Maybe not **all** you need. But they do **improve LightGBM**.\n\nI tried a simple experiment that worked pretty well:\n\n### Lag Features\n\nIn this competition, we get information about clients of AMEX over time. \nMost high-scoring notebooks in this competition focused on aggregating the information per client and creating a single row of extracted features: One for each client.\n\n**One of such agg functions is `last`**.\n\nA quick examination revealed that the `last` feature is extremely powerful for predicting if a client defaults or not (well.. makes sense..). \n\nSo I took this concept two steps further: \n\n- **\"First\" feature:** Just like the `last` feature: I added a `first` feature. \n- **\"Lag\" features** to capture the change over time about each client I calculated two features for every `first`, `last` pair:\n - **\"Last - First\":** The change from we first see the client to the last time we see the client.\n - **\"Last / First\":** The fractional difference from we first see the client to the last time we see the client.\n\nThis improved my `LightGBM` model to the point that it overtook my whole `LightGBM` + `Catboost` + `XGB` ensemble.\n\n- **The notebook can be found [here](https://www.kaggle.com/code/thedevastator/lag-features-are-all-you-need/)**\n\nI uploaded a [dataset](https://www.kaggle.com/datasets/thedevastator/amex-fe/settings) containing the extracted lag features and updated the final [model predictions](https://www.kaggle.com/datasets/thedevastator/amex-predictions) (only `LightGBM` this time) for everyone to play with. \n\nCredits: The model is based on this amazing notebook.
\n
Missed this gorilla lying in plain sight :D. Thanks for opening my eyes.
", "rawMarkdown": "Missed this gorilla lying in plain sight :D. Thanks for opening my eyes.", "votes": 1 }, { "id": 1849357, "postDate": "2022-07-09T12:51:00.120Z", "content": "Hi, how are you able to account for the customers with less than 13 rows (specifically customers with 1 row) for features like (last/first)?
", "rawMarkdown": "Hi, how are you able to account for the customers with less than 13 rows (specifically customers with 1 row) for features like (last/first)?" }, { "id": 1847813, "postDate": "2022-07-08T06:59:43.510Z", "content": "Thanks man. The subtraction is similar to the first difference in a ARIMA (x,1,x) model.
", "rawMarkdown": "Thanks man. The subtraction is similar to the first difference in a ARIMA (x,1,x) model. " }, { "id": 1846317, "postDate": "2022-07-07T02:46:40Z", "rawMarkdown": "", "votes": 1, "isDeleted": true, "replies": [ { "id": 1846346, "postDate": "2022-07-07T03:14:20.320Z", "content": "GBT at least would get there pretty indirectly -- a tree has no direct way of understanding feature1 - feature2, only lots of iterations of something like if feature1 > large_value and feature2 < small_value then extract something about that spread. In general, an intrinsic limitation (but also benefit!) of trees is that they do not reason about different features as if they were on the same scale -- features scales are completely independent. To be fair, a neural network would be able to directly extract max-min as a hidden layer feature but of course suffers other drawbacks on a dataset like this.
\nA good way to think about this is that even if your model can approximate a feature extremely well, the model should be simpler and better if that feature were just fed in directly. Another good way to think about this is that the most impactful feature engineering is often guided by the information structure that a specific model is most likely to miss / struggle to recreate from scratch. Those small adjustments are the sorts of things that matter at the 3rd or 4th decimal place, even when they may be less important in real-world modeling.
", "rawMarkdown": "GBT at least would get there pretty indirectly -- a tree has no direct way of understanding feature1 - feature2, only lots of iterations of something like if feature1 > large_value and feature2 < small_value then extract something about that spread. In general, an intrinsic limitation (but also benefit!) of trees is that they do not reason about different features as if they were on the same scale -- features scales are completely independent. To be fair, a neural network would be able to directly extract max-min as a hidden layer feature but of course suffers other drawbacks on a dataset like this. \n\nA good way to think about this is that even if your model can approximate a feature extremely well, the model should be simpler and better if that feature were just fed in directly. Another good way to think about this is that the most impactful feature engineering is often guided by the information structure that a specific model is most likely to miss / struggle to recreate from scratch. Those small adjustments are the sorts of things that matter at the 3rd or 4th decimal place, even when they may be less important in real-world modeling.", "votes": 17 }, { "id": 1846388, "postDate": "2022-07-07T03:55:53.947Z", "rawMarkdown": "", "isDeleted": true } ] } ], "comments": [ { "id": 1845893, "author_name": "Karan Dora", "author_url": "", "post_date": "2022-07-06T16:54:53.460000", "content": "Missed this gorilla lying in plain sight :D. Thanks for opening my eyes.
", "votes": 1, "replies": [] }, { "id": 1849357, "author_name": "Tarrasque9", "author_url": "", "post_date": "2022-07-09T12:51:00.120000", "content": "Hi, how are you able to account for the customers with less than 13 rows (specifically customers with 1 row) for features like (last/first)?
", "votes": 0, "replies": [] }, { "id": 1847813, "author_name": "Ben Fung", "author_url": "", "post_date": "2022-07-08T06:59:43.510000", "content": "Thanks man. The subtraction is similar to the first difference in a ARIMA (x,1,x) model.
", "votes": 0, "replies": [] }, { "id": 1846317, "author_name": "", "author_url": "", "post_date": "2022-07-07T02:46:40", "content": "", "votes": 1, "replies": [ { "id": 1846346, "author_name": "Joe Eddy", "author_url": "", "post_date": "2022-07-07T03:14:20.320000", "content": "GBT at least would get there pretty indirectly -- a tree has no direct way of understanding feature1 - feature2, only lots of iterations of something like if feature1 > large_value and feature2 < small_value then extract something about that spread. In general, an intrinsic limitation (but also benefit!) of trees is that they do not reason about different features as if they were on the same scale -- features scales are completely independent. To be fair, a neural network would be able to directly extract max-min as a hidden layer feature but of course suffers other drawbacks on a dataset like this.
\nA good way to think about this is that even if your model can approximate a feature extremely well, the model should be simpler and better if that feature were just fed in directly. Another good way to think about this is that the most impactful feature engineering is often guided by the information structure that a specific model is most likely to miss / struggle to recreate from scratch. Those small adjustments are the sorts of things that matter at the 3rd or 4th decimal place, even when they may be less important in real-world modeling.
", "votes": 17, "replies": [] }, { "id": 1846388, "author_name": "", "author_url": "", "post_date": "2022-07-07T03:55:53.947000", "content": "", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1845782": "## Lag Features ~~Are All~~ You Need\n\n\nOK. Maybe not **all** you need. But they do **improve LightGBM**.\n\nI tried a simple experiment that worked pretty well:\n\n### Lag Features\n\nIn this competition, we get information about clients of AMEX over time. \nMost high-scoring notebooks in this competition focused on aggregating the information per client and creating a single row of extracted features: One for each client.\n\n**One of such agg functions is `last`**.\n\nA quick examination revealed that the `last` feature is extremely powerful for predicting if a client defaults or not (well.. makes sense..). \n\nSo I took this concept two steps further: \n\n- **\"First\" feature:** Just like the `last` feature: I added a `first` feature. \n- **\"Lag\" features** to capture the change over time about each client I calculated two features for every `first`, `last` pair:\n - **\"Last - First\":** The change from we first see the client to the last time we see the client.\n - **\"Last / First\":** The fractional difference from we first see the client to the last time we see the client.\n\nThis improved my `LightGBM` model to the point that it overtook my whole `LightGBM` + `Catboost` + `XGB` ensemble.\n\n- **The notebook can be found [here](https://www.kaggle.com/code/thedevastator/lag-features-are-all-you-need/)**\n\nI uploaded a [dataset](https://www.kaggle.com/datasets/thedevastator/amex-fe/settings) containing the extracted lag features and updated the final [model predictions](https://www.kaggle.com/datasets/thedevastator/amex-predictions) (only `LightGBM` this time) for everyone to play with. \n\n