![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F073db90f34626879e50d014bc5edfbd1%2FSelection_999(4983).png?generation=1707454745891137&alt=media\")
my first though on \"feature_definitions.csv\" is to use language model to let my predicter knows the meaning of the features. i wonder any work on this?
\nrough plan (pre-alpha version):
\nmy first though on \"feature_definitions.csv\" is to use language model to let my predicter knows the meaning of the features. i wonder any work on this?
\nrough plan (pre-alpha version):
\na related work.
\nactually this allows you to use external data (which may have different colums)
\n
![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F186a1dd13f2835d85a6f1c75fc8ed54b%2FSelection_999(4966).png?generation=1707417216215323&alt=media\")
https://arize.com/blog-course/applying-large-language-models-to-tabular-data/
", "rawMarkdown": "a related work.\n\nactually this allows you to use external data (which may have different colums)\n\n\n.png?generation=1707454745891137&alt=media)\n\n.png?generation=1707417216215323&alt=media)\n\nhttps://arize.com/blog-course/applying-large-language-models-to-tabular-data/", "votes": null }, { "id": "2643777", "postDate": "02/09/2024 05:15:15", "content": "google search term:
\ncredit scoring, lead time, scorecard, gini , rating, stability, Volatility (?), credit cycle, drift, postproduction monitoring
\n…. yet to find paper that talks about scorecard stability over time ….
\nmaybe this:
\nScorecards Backtesting
\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F2233ec91be5388f8b6a318888ff9a341%2FSelection_999(4986).png?generation=1707455966999095&alt=media\")
\nhttps://altair.com/newsroom/articles/credit-scoring-series-part-nine-scorecard-implementation-deployment-production-and-monitoring
\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F7667005f3bb739790c304691bd910d85%2FSelection_999(4987).png?generation=1707455977399899&alt=media\")
i was talking to my banker friend. he was suggesting this prompt:
\n\"now i describe the economic background: in year 2020, there …,2022, … covid, … trade war, ….
\nwe are a home credit bank, we have a customer record with age 67, xxx as occupation, …. table to text \"
not sure if this works, but it is something very new
\nMambaTab: A Simple Yet Effective Approach for Handling Tabular Data
\nhttps://arxiv.org/pdf/2401.08867.pdf
![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F3d49f6818ff4e06105d7c269802badc4%2FSelection_999(5004).png?generation=1707497249957935&alt=media\")
\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F69d73d65794b64ccf1448663ab11b624%2FSelection_999(5005).png?generation=1707497263320644&alt=media\")
In those tables above XGBoost is losing to other methods in almost all datasets. On Kaggle however, we see that it wins almost all the time for tabular problems. That makes me a bit skeptical about the reported results.
\nMy intuition is that for classification NNs have even less advantage over XGBoost compared to a regression, where the basic problem of GBDTs is that they cannot predict outside of seen target ranges. And yet, GBDTs are still winning even in non-stationary problems as was shown in a recent ENEFIT competition (we don't have final results yet). For classification, there are only 0's and 1's. Boosting seems to be ideal for Classification final layer, as long as we are able to find relevant features.
\nThat said, NNs might be useful as a diversification in a blend.
", "rawMarkdown": "In those tables above XGBoost is losing to other methods in almost all datasets. On Kaggle however, we see that it wins almost all the time for tabular problems. That makes me a bit skeptical about the reported results.\n\nMy intuition is that for classification NNs have even less advantage over XGBoost compared to a regression, where the basic problem of GBDTs is that they cannot predict outside of seen target ranges. And yet, GBDTs are still winning even in non-stationary problems as was shown in a recent ENEFIT competition (we don't have final results yet). For classification, there are only 0's and 1's. Boosting seems to be ideal for Classification final layer, as long as we are able to find relevant features.\n\nThat said, NNs might be useful as a diversification in a blend.", "votes": null }, { "id": "2646686", "postDate": "02/11/2024 06:22:24", "content": "\n\nuse language model to let my predicter knows the meaning of the features
\n
Did you have any success in this @hengck23 ? For me so far the most effective method of understanding features is studying correlations between features themselves and features and the target
", "rawMarkdown": "> use language model to let my predicter knows the meaning of the features\n\nDid you have any success in this @hengck23 ? For me so far the most effective method of understanding features is studying correlations between features themselves and features and the target", "votes": null }, { "id": "2647466", "postDate": "02/11/2024 15:00:57", "content": "I think they didn't done any proper feature engineer before using XGBoost, and it stated that they didn't tune the XGBoost parameter extensively. I think GBDT's can derive significant advantages from well-crafted features and optimal parameter settings.
", "rawMarkdown": "I think they didn't done any proper feature engineer before using XGBoost, and it stated that they didn't tune the XGBoost parameter extensively. I think GBDT's can derive significant advantages from well-crafted features and optimal parameter settings.", "votes": null } ], "comments": [ { "id": 2643288, "author_name": "hengck23", "author_url": "", "post_date": "02/08/2024 18:33:46", "content": "a related work.
\nactually this allows you to use external data (which may have different colums)
\n
https://arize.com/blog-course/applying-large-language-models-to-tabular-data/
", "votes": null, "replies": [ { "id": 2644344, "author_name": "hengck23", "author_url": "", "post_date": "02/09/2024 12:02:48", "content": "i was talking to my banker friend. he was suggesting this prompt:
\n\"now i describe the economic background: in year 2020, there …,2022, … covid, … trade war, ….
\nwe are a home credit bank, we have a customer record with age 67, xxx as occupation, …. table to text \"
google search term:
\ncredit scoring, lead time, scorecard, gini , rating, stability, Volatility (?), credit cycle, drift, postproduction monitoring
\n…. yet to find paper that talks about scorecard stability over time ….
\nmaybe this:
\nScorecards Backtesting
\n
\nhttps://altair.com/newsroom/articles/credit-scoring-series-part-nine-scorecard-implementation-deployment-production-and-monitoring
\n
not sure if this works, but it is something very new
\nMambaTab: A Simple Yet Effective Approach for Handling Tabular Data
\nhttps://arxiv.org/pdf/2401.08867.pdf
\n
In those tables above XGBoost is losing to other methods in almost all datasets. On Kaggle however, we see that it wins almost all the time for tabular problems. That makes me a bit skeptical about the reported results.
\nMy intuition is that for classification NNs have even less advantage over XGBoost compared to a regression, where the basic problem of GBDTs is that they cannot predict outside of seen target ranges. And yet, GBDTs are still winning even in non-stationary problems as was shown in a recent ENEFIT competition (we don't have final results yet). For classification, there are only 0's and 1's. Boosting seems to be ideal for Classification final layer, as long as we are able to find relevant features.
\nThat said, NNs might be useful as a diversification in a blend.
", "votes": null, "replies": [ { "id": 2647466, "author_name": "yuchen2066", "author_url": "", "post_date": "02/11/2024 15:00:57", "content": "I think they didn't done any proper feature engineer before using XGBoost, and it stated that they didn't tune the XGBoost parameter extensively. I think GBDT's can derive significant advantages from well-crafted features and optimal parameter settings.
", "votes": null, "replies": [] } ] } ] }, { "id": 2646686, "author_name": "narsil", "author_url": "", "post_date": "02/11/2024 06:22:24", "content": "\n\nuse language model to let my predicter knows the meaning of the features
\n
Did you have any success in this @hengck23 ? For me so far the most effective method of understanding features is studying correlations between features themselves and features and the target
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2643284": "my first though on \"feature_definitions.csv\" is to use language model to let my predicter knows the meaning of the features. i wonder any work on this?\n\nrough plan (pre-alpha version):\n- train basic model with usual loss\n- modified loss for competition stability metric\n- overfit train datat to get upper bound of performance\n- model explainability with and without regards to stability (i.e. explainability over time)", "2643288": "a related work.\n\nactually this allows you to use external data (which may have different colums)\n\n\n.png?generation=1707454745891137&alt=media)\n\n.png?generation=1707417216215323&alt=media)\n\nhttps://arize.com/blog-course/applying-large-language-models-to-tabular-data/", "2643777": "google search term:\n\ncredit scoring, lead time, scorecard, gini , rating, stability, Volatility (?), credit cycle, drift, postproduction monitoring\n\n.... yet to find paper that talks about scorecard stability over time ....\n\nmaybe this:\nScorecards Backtesting\n.png?generation=1707455966999095&alt=media)\n\nhttps://altair.com/newsroom/articles/credit-scoring-series-part-nine-scorecard-implementation-deployment-production-and-monitoring\n.png?generation=1707455977399899&alt=media)", "2644344": "i was talking to my banker friend. he was suggesting this prompt:\n\n\"now i describe the economic background: in year 2020, there ...,2022, ... covid, ... trade war, ....\nwe are a home credit bank, we have a customer record with age 67, xxx as occupation, .... table to text \"", "2644752": "not sure if this works, but it is something very new\n\nMambaTab: A Simple Yet Effective Approach for Handling Tabular Data\nhttps://arxiv.org/pdf/2401.08867.pdf\n\n.png?generation=1707497249957935&alt=media)\n.png?generation=1707497263320644&alt=media)", "2645168": "In those tables above XGBoost is losing to other methods in almost all datasets. On Kaggle however, we see that it wins almost all the time for tabular problems. That makes me a bit skeptical about the reported results.\n\nMy intuition is that for classification NNs have even less advantage over XGBoost compared to a regression, where the basic problem of GBDTs is that they cannot predict outside of seen target ranges. And yet, GBDTs are still winning even in non-stationary problems as was shown in a recent ENEFIT competition (we don't have final results yet). For classification, there are only 0's and 1's. Boosting seems to be ideal for Classification final layer, as long as we are able to find relevant features.\n\nThat said, NNs might be useful as a diversification in a blend.", "2646686": "> use language model to let my predicter knows the meaning of the features\n\nDid you have any success in this @hengck23 ? For me so far the most effective method of understanding features is studying correlations between features themselves and features and the target", "2647466": "I think they didn't done any proper feature engineer before using XGBoost, and it stated that they didn't tune the XGBoost parameter extensively. I think GBDT's can derive significant advantages from well-crafted features and optimal parameter settings." }, "source": "meta" }