2022-07-26 update: see the code in this public notebook https://www.kaggle.com/code/mrandri19/0-796-lb-sharing-my-ablations-dart-less-xgb
\nI am trying to achieve a single DART-less model with >0.795 CV/LB score.
\nI am happy with the model and hyperparameters so I am working on feature engineering and feature selection.
All the results have been obtained via sklearn's 5-fold StratifiedKFold cross-validation with seed=42.
\nKeep in mind that cdeotte says that the standard deviation of a model trained on different seeds is 0.0012.
\nWith a standard deviation of this magnitude, I don't feel confident in differences smaller than 0.0024, so take my results with a pinch of salt.
Below is the description of the model, hyperparameters and the ablation studies I have ran so far:
\nXGBoost trained with early stopping using the amex metric.
\nIn particular I use 9999 boosting rounds and 500 early stopping rounds.
\nEach model takes 10-20 minutes to train on a Kaggle GPU instance.
Taken from the excellent notebook https://www.kaggle.com/code/jiweiliu/rapids-cudf-feature-engineering-xgb.
\n'max_depth': 7,\n'eta': 0.03,\n\n'subsample': 0.88,\n'colsample_bytree': 0.5,\n\n'objective': 'binary:logistic',\n\n'tree_method': 'gpu_hist',\n\n'random_state': 42,\n\n'gamma': 1.5,\n'min_child_weight': 8,\n'lambda': 70,\nThe features, selecting the last month of data for each customer, minus customer_ID and S_2, fed directly to the model.
\nAs raddar suggests, there is uniform noise injected to the data, we can ignore the noise by throwing signals smaller
\nthan 0.01 by rounding to 2 decimal digits.
After filling the missing values with the mean, compute the pairwise pearson correlation between all 188 features.
\nThen drop features that are >95% correlated with others.
So fare we have only used the last value from each feature, let's use the first, last, mean, std, min, max as well.
\nSince we have a lot more features, we can again try to drop correlated ones.
\nWe can count the number of missing values in each column, for every customer.
\nFor every column, this is an integer in [0, 13].
Replicate the feature enginering performed by Jiwei Liu in this notebook: https://www.kaggle.com/code/jiweiliu/rapids-cudf-feature-engineering-xgb.
\nIn practice, compute the subtraction of a bunch of B, D, S features minus two P (Payment) features: P_2, P_3
Replicate the feature engineering by thedevastator described in described in
\nhttps://www.kaggle.com/code/thedevastator/lag-features-are-all-you-need
This is very weird, as these features come from one of the best publicly available notebooks.
\nMy hypothesis is that too many (>1500) features combined with an aggressive 0.5 colsample_bytree leads to
\nsome the model missing some interactions, but I am curious of the reader's opinion.
https://www.kaggle.com/code/illidan7/amex-basic-feature-engineering-1500-features
\n2022-07-26 update: see the code in this public notebook https://www.kaggle.com/code/mrandri19/0-796-lb-sharing-my-ablations-dart-less-xgb
\nI am trying to achieve a single DART-less model with >0.795 CV/LB score.
\nI am happy with the model and hyperparameters so I am working on feature engineering and feature selection.
All the results have been obtained via sklearn's 5-fold StratifiedKFold cross-validation with seed=42.
\nKeep in mind that cdeotte says that the standard deviation of a model trained on different seeds is 0.0012.
\nWith a standard deviation of this magnitude, I don't feel confident in differences smaller than 0.0024, so take my results with a pinch of salt.
Below is the description of the model, hyperparameters and the ablation studies I have ran so far:
\nXGBoost trained with early stopping using the amex metric.
\nIn particular I use 9999 boosting rounds and 500 early stopping rounds.
\nEach model takes 10-20 minutes to train on a Kaggle GPU instance.
Taken from the excellent notebook https://www.kaggle.com/code/jiweiliu/rapids-cudf-feature-engineering-xgb.
\n'max_depth': 7,\n'eta': 0.03,\n\n'subsample': 0.88,\n'colsample_bytree': 0.5,\n\n'objective': 'binary:logistic',\n\n'tree_method': 'gpu_hist',\n\n'random_state': 42,\n\n'gamma': 1.5,\n'min_child_weight': 8,\n'lambda': 70,\nThe features, selecting the last month of data for each customer, minus customer_ID and S_2, fed directly to the model.
\nAs raddar suggests, there is uniform noise injected to the data, we can ignore the noise by throwing signals smaller
\nthan 0.01 by rounding to 2 decimal digits.
After filling the missing values with the mean, compute the pairwise pearson correlation between all 188 features.
\nThen drop features that are >95% correlated with others.
So fare we have only used the last value from each feature, let's use the first, last, mean, std, min, max as well.
\nSince we have a lot more features, we can again try to drop correlated ones.
\nWe can count the number of missing values in each column, for every customer.
\nFor every column, this is an integer in [0, 13].
Replicate the feature enginering performed by Jiwei Liu in this notebook: https://www.kaggle.com/code/jiweiliu/rapids-cudf-feature-engineering-xgb.
\nIn practice, compute the subtraction of a bunch of B, D, S features minus two P (Payment) features: P_2, P_3
Replicate the feature engineering by thedevastator described in described in
\nhttps://www.kaggle.com/code/thedevastator/lag-features-are-all-you-need
This is very weird, as these features come from one of the best publicly available notebooks.
\nMy hypothesis is that too many (>1500) features combined with an aggressive 0.5 colsample_bytree leads to
\nsome the model missing some interactions, but I am curious of the reader's opinion.
https://www.kaggle.com/code/illidan7/amex-basic-feature-engineering-1500-features
\nHi Andrea, update on my end: I went live with my XGB notebook today: https://www.kaggle.com/code/roberthatch/xgboost-pyramid-test-predictions
\nCV 0.7968 (~0.7965-7970); LB 0.797 and edged out other 0.797 public notebooks.
\nThe hyper-parameters aren't all that optimized, the feature engineering is unoptimized and pretty untested. But I'm very happy with the results of my \"Pyramid\" experiment.
\nEssentially, I tried using other methods besides DART that I suspected might reduce \"over-specialization\", without the performance penalty that DART + XGB + GPU has. It was very successful, using two or three tricks, depending how you count it:
\nGreat work, @mrandri19 , my best XGB gbt has CV 0.7958 and LB 0.796 (seed 42). Point (h) caused me the same question - I don't know why, but lags \"last-first\" and \"last-avg\" work worse together than if you leave one of them. Perhaps dart is more loyal to the number of features than gbt, but it's still strange. I also tried adding kurtosis and skew of most important features, but it didn't improve the result.
", "rawMarkdown": "Great work, @mrandri19 , my best XGB gbt has CV 0.7958 and LB 0.796 (seed 42). Point (h) caused me the same question - I don't know why, but lags \"last-first\" and \"last-avg\" work worse together than if you leave one of them. Perhaps dart is more loyal to the number of features than gbt, but it's still strange. I also tried adding kurtosis and skew of most important features, but it didn't improve the result.", "votes": 1, "replies": [ { "id": 1853290, "postDate": "2022-07-12T19:24:19.427Z", "content": "Humm, that's very interesting, thanks for sharing.
\nSo you tried ragnar123's features (last-first, last-avg) but also did not see any improvement.
\nI guess I should bite the bullet and try using DART (perhaps with LGBM as I hear it is faster for than XGB's DART).
It's true. XGB's DART is veeeery slow. I spent more than 2 hours learning 1 of 5 fold, but didn't wait for the end…
", "rawMarkdown": "It's true. XGB's DART is veeeery slow. I spent more than 2 hours learning 1 of 5 fold, but didn't wait for the end...", "votes": 1 }, { "id": 1854491, "postDate": "2022-07-13T18:28:52.467Z", "content": "Dart vs non-Dart:
\nWith my own dataset, running everything in Kaggle, I got the following:
\nI'd be careful coming to any conclusions on a single test, but, if not caring about CPU vs GPU, I wouldn't be too shocked if crazy low learning rate and row and col_sample could get non-Dart CV equal to Dart in similar time of 5 hours (60 min per fold) vs 5 CPUs doing Dart at once at 5 hours per fold. Shrug.
", "rawMarkdown": "Dart vs non-Dart:\n\nWith my own dataset, running everything in Kaggle, I got the following:\n* XGB GPU non-Dart (varying seed and slight tweaks to hyper-params on different runs): CV 0.7951-0.7959, one hour training (~13 min per fold)\n* LGBM CPU Dart: CV 0.7965, 5 hours training each fold in parallel, plus about an hour test inference using the 5 model outputs.\n\nI'd be careful coming to any conclusions on a single test, but, if not caring about CPU vs GPU, I wouldn't be too shocked if crazy low learning rate and row and col_sample could get non-Dart CV equal to Dart in similar time of 5 hours (60 min per fold) vs 5 CPUs doing Dart at once at 5 hours per fold. Shrug." }, { "id": 1855640, "postDate": "2022-07-14T18:41:43.350Z", "content": "I had a thought inspired by this sub thread:
\nhttps://www.kaggle.com/competitions/amex-default-prediction/discussion/337160
\nBasically, the issue with any two last-[thing], is that you are probably adding some highly correlated columns. This will hurt tree diversity. To deal with that, you need to drop sampling way down. But 'last' features are crucial… so let's double them(!).
\nOh wait, top notebook does both those things with 20% feature sampling and rounded last in addition to last. Hmm!
", "rawMarkdown": "I had a thought inspired by this sub thread:\n\nhttps://www.kaggle.com/competitions/amex-default-prediction/discussion/337160\n\nBasically, the issue with any two last-[thing], is that you are probably adding some highly correlated columns. This will hurt tree diversity. To deal with that, you need to drop sampling way down. But 'last' features are crucial... so let's double them(!). \n\nOh wait, top notebook does both those things with 20% feature sampling and rounded last in addition to last. Hmm!", "votes": 1 }, { "id": 1858043, "postDate": "2022-07-16T15:50:20.847Z", "content": "@mrandri19 I want to brag my new best results - XGB gbt CV(5 folds) 0.79662 (seed 42) LB 0.797. It was so difficult😅
", "rawMarkdown": "@mrandri19 I want to brag my new best results - XGB gbt CV(5 folds) 0.79662 (seed 42) LB 0.797. It was so difficult😅", "votes": 1 }, { "id": 1858162, "postDate": "2022-07-16T17:56:12.680Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 1858755, "postDate": "2022-07-17T07:22:10.297Z", "content": "The information provided is so clear and easy to understand, Awesome post :)
\nLooking forward to see more !!!
\nlet's grow together.
Great work, @mrandri19 , my best XGB gbt has CV 0.7958 and LB 0.796 (seed 42). Point (h) caused me the same question - I don't know why, but lags \"last-first\" and \"last-avg\" work worse together than if you leave one of them. Perhaps dart is more loyal to the number of features than gbt, but it's still strange. I also tried adding kurtosis and skew of most important features, but it didn't improve the result.
", "rawMarkdown": "Great work, @mrandri19 , my best XGB gbt has CV 0.7958 and LB 0.796 (seed 42). Point (h) caused me the same question - I don't know why, but lags \"last-first\" and \"last-avg\" work worse together than if you leave one of them. Perhaps dart is more loyal to the number of features than gbt, but it's still strange. I also tried adding kurtosis and skew of most important features, but it didn't improve the result.", "votes": -4 }, { "id": 1862506, "postDate": "2022-07-19T18:52:52.693Z", "content": "@mrandri19 How are your research progressing? A week ago I was convinced that using DART is very early and it was true) At the moment I have 0.7973 CV(5) and 0.797 LB! And you know what? This is not the limit! I think the possible best CV result is around 0.7975 for XGB gbt.
", "rawMarkdown": "@mrandri19 How are your research progressing? A week ago I was convinced that using DART is very early and it was true) At the moment I have 0.7973 CV(5) and 0.797 LB! And you know what? This is not the limit! I think the possible best CV result is around 0.7975 for XGB gbt.", "replies": [ { "id": 1872318, "postDate": "2022-07-26T20:17:51.517Z", "content": "@dmitryuarov I am slowly progressing on feature engineering. Right now I have:
\nThe biggest boosts have come from last, count, nunique for categoricals and last - mean.
", "rawMarkdown": "@dmitryuarov I am slowly progressing on feature engineering. Right now I have:\n\n- DART-less XGB 0.7951 CV 0.796 LB\n- DART-less LGBM 0.7953 CV 0.796 LB\n- (Work-In-Progress) NN 0.78XX CV (haven't submitted yet to LB and did not try with all folds as I still am iterating quickly)\n\nThe biggest boosts have come from last, count, nunique for categoricals and last - mean." } ] }, { "id": 1861261, "postDate": "2022-07-18T22:42:10.480Z", "content": "Thank you for sharing your ablations studies !
", "rawMarkdown": "Thank you for sharing your ablations studies ! \n" }, { "id": 1856280, "postDate": "2022-07-15T08:31:03.007Z", "content": "thanks for sharing. on the std of the models, i wonder what std you get from 5 folds? mine is about 0.004.
", "rawMarkdown": "thanks for sharing. on the std of the models, i wonder what std you get from 5 folds? mine is about 0.004. ", "replies": [ { "id": 1860812, "postDate": "2022-07-18T15:25:50.227Z", "content": "around 0.002 or 0.003
", "rawMarkdown": "around 0.002 or 0.003" } ] }, { "id": 1853608, "postDate": "2022-07-13T02:10:13.627Z", "content": "In my experience (well, experiments), adding even more good features only really helps once you decrease colsample even lower AND lower Learning Rate. (0.01 eta and 35% col for me with ~1300 features)
", "rawMarkdown": "In my experience (well, experiments), adding even more good features only really helps once you decrease colsample even lower AND lower Learning Rate. (0.01 eta and 35% col for me with ~1300 features)", "replies": [ { "id": 1853695, "postDate": "2022-07-13T04:41:02.183Z", "content": "Re: my experiments and re your quote: \"I am trying to achieve a single DART-less model with >0.795 CV/LB score.\"
\nI've got: 0.79699 LB and 0.79557 CV. 74min train time for 5 folds.
\nAll XGB with feature engg and a only a little hyper param tinkering. Within the LB 0.796s, I'm listed first, so my score is about 0.79699.
\nTo get that score, I used the exact features in the current version of my public feature engg notebook: https://www.kaggle.com/code/roberthatch/amex-feature-engg-gpu-or-cpu-process-in-chunks
\nAnd there's some interesting features you might want to try, but, as mentioned in above comment, I wouldn't be surprised if what you really need is some hyper-parameter tweaks and those alone get you your goal of > 0.795 CV/LB
\nHyper-params:
\nExcept as below, should be identical to: https://www.kaggle.com/code/jiweiliu/rapids-cudf-feature-engineering-xgb.
Features engg besides the stuff in other notebooks and mentioned above.
\n(Again, from here: https://www.kaggle.com/code/roberthatch/amex-feature-engg-gpu-or-cpu-process-in-chunks )
But all my experiments were about as unscientific as yours were scientific. Constantly changing 3 variables at once :)
\nLet me know if you try any of the above, whether params or features!
\nP.s.: planning to share my full notebook publicly, not just the feature engg, but working on an even higher score, with one additional innovation… and having some major pathfinding issues to sort out.
", "rawMarkdown": "Re: my experiments and re your quote: \"I am trying to achieve a single DART-less model with >0.795 CV/LB score.\"\n\nI've got: 0.79699 LB and 0.79557 CV. 74min train time for 5 folds.\n\nAll XGB with feature engg and a only a little hyper param tinkering. Within the LB 0.796s, I'm listed first, so my score is about 0.79699.\n\nTo get that score, I used the exact features in the current version of my public feature engg notebook: https://www.kaggle.com/code/roberthatch/amex-feature-engg-gpu-or-cpu-process-in-chunks\n\nAnd there's some interesting features you might want to try, but, as mentioned in above comment, I wouldn't be surprised if what you really need is some hyper-parameter tweaks and those alone get you your goal of > 0.795 CV/LB\n\nHyper-params:\nExcept as below, should be identical to: https://www.kaggle.com/code/jiweiliu/rapids-cudf-feature-engineering-xgb.\n1. After getting good results with 0.01 learning rate, I tried this and got nearly the same train time and result, but marginally better in both, iirc. 'num_parallel_tree':3, 'learning_rate':0.03,\n*EDIT*: if using num_parallel_tree, oddly enough you have to modify model.best_ntree_limit by adding integer divide by num_parallel_tree. Example:\n model.predict(dvalid, iteration_range=(0,model.best_ntree_limit//TREE_MULTIPLE))\n2. 'objective':'rank:pairwise', 'eval_metric':'auc',\n3. Sampling: 'subsample':0.75, 'colsample_bytree': 0.20,\nNote that I think 35% cols gave better CV results, but I didn't create submission csv on that 0.7959 CV run, and haven't had GPU time to go back and fine-tune.\n\nFeatures engg besides the stuff in other notebooks and mentioned above.\n(Again, from here: https://www.kaggle.com/code/roberthatch/amex-feature-engg-gpu-or-cpu-process-in-chunks ) \n1. Simplified Hull Moving Average\n2. Magnitude = max-min. (And dropped min and max)\n3. CurrentLevel = (last-min) / (max-min)\n\nBut all my experiments were about as unscientific as yours were scientific. Constantly changing 3 variables at once :)\n\nLet me know if you try any of the above, whether params or features!\n\nP.s.: planning to share my full notebook publicly, not just the feature engg, but working on an even higher score, with one additional innovation... and having some major pathfinding issues to sort out.", "votes": 2 }, { "id": 1854245, "postDate": "2022-07-13T14:35:09.800Z", "content": "Wow, this is great thanks! I will try some of your suggestions and update you.
", "rawMarkdown": "Wow, this is great thanks! I will try some of your suggestions and update you.", "votes": 1 } ] }, { "id": 1853600, "postDate": "2022-07-13T01:54:28.247Z", "content": "Thanks for sharing your ablation studies!
\nAre you using all of the features in the dataset? If not, what feature selection method do you use?
Yes, I am using all of the features.
\nI would like to try feature selection using feature importances but I did not have time yet/am not sure what to use among SHAP/Permutation Importance/Null Importance.
great work, thanks for sharing your ideas!
", "rawMarkdown": "great work, thanks for sharing your ideas!" }, { "id": 1853294, "postDate": "2022-07-12T19:31:55.050Z", "content": "Nice .. did lag features decrease ur CV/LB . I think one lag feature which ragner described did certainly help
", "rawMarkdown": "Nice .. did lag features decrease ur CV/LB . I think one lag feature which ragner described did certainly help ", "replies": [ { "id": 1853302, "postDate": "2022-07-12T19:41:11.707Z", "content": "Thanks :)
\nDo you mean the lag features (last - first, last - avg) described by ragnar or the ones (last / first, last - first) described by thedevastator?
I mean both .. I think Ragner last-Avg made sense to me the others I havent check the feature importance yet to assess .. but Ragner ones were up there in FE.
", "rawMarkdown": "I mean both .. I think Ragner last-Avg made sense to me the others I havent check the feature importance yet to assess .. but Ragner ones were up there in FE. ", "votes": 1 } ] }, { "id": 1852203, "postDate": "2022-07-11T21:38:26.727Z", "content": "Always interesting to look at good feature engineering experiments, well done!
", "rawMarkdown": "Always interesting to look at good feature engineering experiments, well done!" }, { "id": 1852120, "postDate": "2022-07-11T19:32:41.097Z", "content": "Very well curated and summarized!
", "rawMarkdown": "Very well curated and summarized!" }, { "id": 1857388, "postDate": "2022-07-16T05:14:05.723Z", "content": "thanks for this help for us
", "rawMarkdown": "thanks for this help for us ", "votes": -3 } ], "comments": [ { "id": 1864319, "author_name": "Robert Hatch", "author_url": "", "post_date": "2022-07-21T00:20:10.483000", "content": "Hi Andrea, update on my end: I went live with my XGB notebook today: https://www.kaggle.com/code/roberthatch/xgboost-pyramid-test-predictions
\nCV 0.7968 (~0.7965-7970); LB 0.797 and edged out other 0.797 public notebooks.
\nThe hyper-parameters aren't all that optimized, the feature engineering is unoptimized and pretty untested. But I'm very happy with the results of my \"Pyramid\" experiment.
\nEssentially, I tried using other methods besides DART that I suspected might reduce \"over-specialization\", without the performance penalty that DART + XGB + GPU has. It was very successful, using two or three tricks, depending how you count it:
\nGreat work, @mrandri19 , my best XGB gbt has CV 0.7958 and LB 0.796 (seed 42). Point (h) caused me the same question - I don't know why, but lags \"last-first\" and \"last-avg\" work worse together than if you leave one of them. Perhaps dart is more loyal to the number of features than gbt, but it's still strange. I also tried adding kurtosis and skew of most important features, but it didn't improve the result.
", "votes": 1, "replies": [ { "id": 1853290, "author_name": "Andrea Cognolato", "author_url": "", "post_date": "2022-07-12T19:24:19.427000", "content": "Humm, that's very interesting, thanks for sharing.
\nSo you tried ragnar123's features (last-first, last-avg) but also did not see any improvement.
\nI guess I should bite the bullet and try using DART (perhaps with LGBM as I hear it is faster for than XGB's DART).
It's true. XGB's DART is veeeery slow. I spent more than 2 hours learning 1 of 5 fold, but didn't wait for the end…
", "votes": 1, "replies": [] }, { "id": 1854491, "author_name": "Robert Hatch", "author_url": "", "post_date": "2022-07-13T18:28:52.467000", "content": "Dart vs non-Dart:
\nWith my own dataset, running everything in Kaggle, I got the following:
\nI'd be careful coming to any conclusions on a single test, but, if not caring about CPU vs GPU, I wouldn't be too shocked if crazy low learning rate and row and col_sample could get non-Dart CV equal to Dart in similar time of 5 hours (60 min per fold) vs 5 CPUs doing Dart at once at 5 hours per fold. Shrug.
", "votes": 0, "replies": [] }, { "id": 1855640, "author_name": "Robert Hatch", "author_url": "", "post_date": "2022-07-14T18:41:43.350000", "content": "I had a thought inspired by this sub thread:
\nhttps://www.kaggle.com/competitions/amex-default-prediction/discussion/337160
\nBasically, the issue with any two last-[thing], is that you are probably adding some highly correlated columns. This will hurt tree diversity. To deal with that, you need to drop sampling way down. But 'last' features are crucial… so let's double them(!).
\nOh wait, top notebook does both those things with 20% feature sampling and rounded last in addition to last. Hmm!
", "votes": 1, "replies": [] }, { "id": 1858043, "author_name": "Dmitry Uarov", "author_url": "", "post_date": "2022-07-16T15:50:20.847000", "content": "@mrandri19 I want to brag my new best results - XGB gbt CV(5 folds) 0.79662 (seed 42) LB 0.797. It was so difficult😅
", "votes": 1, "replies": [] }, { "id": 1858162, "author_name": "", "author_url": "", "post_date": "2022-07-16T17:56:12.680000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1858755, "author_name": "Bilal Suppal", "author_url": "", "post_date": "2022-07-17T07:22:10.297000", "content": "The information provided is so clear and easy to understand, Awesome post :)
\nLooking forward to see more !!!
\nlet's grow together.
Great work, @mrandri19 , my best XGB gbt has CV 0.7958 and LB 0.796 (seed 42). Point (h) caused me the same question - I don't know why, but lags \"last-first\" and \"last-avg\" work worse together than if you leave one of them. Perhaps dart is more loyal to the number of features than gbt, but it's still strange. I also tried adding kurtosis and skew of most important features, but it didn't improve the result.
", "votes": -4, "replies": [] }, { "id": 1862506, "author_name": "Dmitry Uarov", "author_url": "", "post_date": "2022-07-19T18:52:52.693000", "content": "@mrandri19 How are your research progressing? A week ago I was convinced that using DART is very early and it was true) At the moment I have 0.7973 CV(5) and 0.797 LB! And you know what? This is not the limit! I think the possible best CV result is around 0.7975 for XGB gbt.
", "votes": 0, "replies": [ { "id": 1872318, "author_name": "Andrea Cognolato", "author_url": "", "post_date": "2022-07-26T20:17:51.517000", "content": "@dmitryuarov I am slowly progressing on feature engineering. Right now I have:
\nThe biggest boosts have come from last, count, nunique for categoricals and last - mean.
", "votes": 0, "replies": [] } ] }, { "id": 1861261, "author_name": "Hou zhangyi ", "author_url": "", "post_date": "2022-07-18T22:42:10.480000", "content": "Thank you for sharing your ablations studies !
", "votes": 0, "replies": [] }, { "id": 1856280, "author_name": "yimacs", "author_url": "", "post_date": "2022-07-15T08:31:03.007000", "content": "thanks for sharing. on the std of the models, i wonder what std you get from 5 folds? mine is about 0.004.
", "votes": 0, "replies": [ { "id": 1860812, "author_name": "Andrea Cognolato", "author_url": "", "post_date": "2022-07-18T15:25:50.227000", "content": "around 0.002 or 0.003
", "votes": 0, "replies": [] } ] }, { "id": 1853608, "author_name": "Robert Hatch", "author_url": "", "post_date": "2022-07-13T02:10:13.627000", "content": "In my experience (well, experiments), adding even more good features only really helps once you decrease colsample even lower AND lower Learning Rate. (0.01 eta and 35% col for me with ~1300 features)
", "votes": 0, "replies": [ { "id": 1853695, "author_name": "Robert Hatch", "author_url": "", "post_date": "2022-07-13T04:41:02.183000", "content": "Re: my experiments and re your quote: \"I am trying to achieve a single DART-less model with >0.795 CV/LB score.\"
\nI've got: 0.79699 LB and 0.79557 CV. 74min train time for 5 folds.
\nAll XGB with feature engg and a only a little hyper param tinkering. Within the LB 0.796s, I'm listed first, so my score is about 0.79699.
\nTo get that score, I used the exact features in the current version of my public feature engg notebook: https://www.kaggle.com/code/roberthatch/amex-feature-engg-gpu-or-cpu-process-in-chunks
\nAnd there's some interesting features you might want to try, but, as mentioned in above comment, I wouldn't be surprised if what you really need is some hyper-parameter tweaks and those alone get you your goal of > 0.795 CV/LB
\nHyper-params:
\nExcept as below, should be identical to: https://www.kaggle.com/code/jiweiliu/rapids-cudf-feature-engineering-xgb.
Features engg besides the stuff in other notebooks and mentioned above.
\n(Again, from here: https://www.kaggle.com/code/roberthatch/amex-feature-engg-gpu-or-cpu-process-in-chunks )
But all my experiments were about as unscientific as yours were scientific. Constantly changing 3 variables at once :)
\nLet me know if you try any of the above, whether params or features!
\nP.s.: planning to share my full notebook publicly, not just the feature engg, but working on an even higher score, with one additional innovation… and having some major pathfinding issues to sort out.
", "votes": 2, "replies": [] }, { "id": 1854245, "author_name": "Andrea Cognolato", "author_url": "", "post_date": "2022-07-13T14:35:09.800000", "content": "Wow, this is great thanks! I will try some of your suggestions and update you.
", "votes": 1, "replies": [] } ] }, { "id": 1853600, "author_name": "The Devastator", "author_url": "", "post_date": "2022-07-13T01:54:28.247000", "content": "Thanks for sharing your ablation studies!
\nAre you using all of the features in the dataset? If not, what feature selection method do you use?
Yes, I am using all of the features.
\nI would like to try feature selection using feature importances but I did not have time yet/am not sure what to use among SHAP/Permutation Importance/Null Importance.
great work, thanks for sharing your ideas!
", "votes": 0, "replies": [] }, { "id": 1853294, "author_name": "Gaurav Rawat", "author_url": "", "post_date": "2022-07-12T19:31:55.050000", "content": "Nice .. did lag features decrease ur CV/LB . I think one lag feature which ragner described did certainly help
", "votes": 0, "replies": [ { "id": 1853302, "author_name": "Andrea Cognolato", "author_url": "", "post_date": "2022-07-12T19:41:11.707000", "content": "Thanks :)
\nDo you mean the lag features (last - first, last - avg) described by ragnar or the ones (last / first, last - first) described by thedevastator?
I mean both .. I think Ragner last-Avg made sense to me the others I havent check the feature importance yet to assess .. but Ragner ones were up there in FE.
", "votes": 1, "replies": [] } ] }, { "id": 1852203, "author_name": "Oleg Sidorshin", "author_url": "", "post_date": "2022-07-11T21:38:26.727000", "content": "Always interesting to look at good feature engineering experiments, well done!
", "votes": 0, "replies": [] }, { "id": 1852120, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2022-07-11T19:32:41.097000", "content": "Very well curated and summarized!
", "votes": 0, "replies": [] }, { "id": 1857388, "author_name": "Bilal Suppal", "author_url": "", "post_date": "2022-07-16T05:14:05.723000", "content": "thanks for this help for us
", "votes": -3, "replies": [] } ], "raw_markdown_by_id": { "1852093": "2022-07-26 update: see the code in this public notebook https://www.kaggle.com/code/mrandri19/0-796-lb-sharing-my-ablations-dart-less-xgb\n\nI am trying to achieve a single DART-less model with >0.795 CV/LB score.\nI am happy with the model and hyperparameters so I am working on feature engineering and feature selection.\n\nAll the results have been obtained via sklearn's 5-fold StratifiedKFold cross-validation with seed=42.\nKeep in mind that cdeotte says that the standard deviation of a model trained on different seeds is 0.0012.\nWith a standard deviation of this magnitude, I don't feel confident in differences smaller than 0.0024, so take my results with a pinch of salt.\n\nBelow is the description of the model, hyperparameters and the ablation studies I have ran so far:\n\n## Model\n\nXGBoost trained with early stopping using the amex metric.\nIn particular I use 9999 boosting rounds and 500 early stopping rounds.\nEach model takes 10-20 minutes to train on a Kaggle GPU instance.\n\n### Hyperparameters\n\nTaken from the excellent notebook https://www.kaggle.com/code/jiweiliu/rapids-cudf-feature-engineering-xgb.\n\n```\n'max_depth': 7,\n'eta': 0.03,\n\n'subsample': 0.88,\n'colsample_bytree': 0.5,\n \n'objective': 'binary:logistic',\n \n'tree_method': 'gpu_hist',\n \n'random_state': 42,\n \n'gamma': 1.5,\n'min_child_weight': 8,\n'lambda': 70,\n```\n\n## Ablation studies\n\n### (a) Baseline\n\nThe features, selecting the last month of data for each customer, minus customer_ID and S_2, fed directly to the model.\n- **CV: 0.7885**\n- 9.5min training time\n- 188 features\n\n### (b) Round to 2 decimal digits\n\nAs raddar suggests, there is uniform noise injected to the data, we can ignore the noise by throwing signals smaller\nthan 0.01 by rounding to 2 decimal digits.\n- **CV: 0.7908, +0.0023** compared with (a)\n- 9.5min training time\n- 188 features\n\n### (c) Drop correlated features\n\nAfter filling the missing values with the mean, compute the pairwise pearson correlation between all 188 features.\nThen drop features that are >95% correlated with others.\n\n- **CV: 0.7905, -0.0003** compared with (b)\n- 10.5min training time\n- 178 features\n\n### (d) Add customer aggregated features\n\nSo fare we have only used the last value from each feature, let's use the first, last, mean, std, min, max as well.\n\n- **CV: 0.7949, +0.0041** compared with (b)\n- 21.5min training time\n- 1073 features\n\n### (e) Add customer aggregated features, then drop correlated features\n\nSince we have a lot more features, we can again try to drop correlated ones.\n\n- **CV: 0.7950, +0.0001** compared with (d)\n- 22.5min training time\n- 875 features\n\n### (f) Add missing count features\n\nWe can count the number of missing values in each column, for every customer.\nFor every column, this is an integer in [0, 13].\n\n- **CV: 0.7946, -0.0003** compared with (d)\n- 24.5min training time\n- 1261 features\n\n### (g) Add difference features\n\nReplicate the feature enginering performed by Jiwei Liu in this notebook: https://www.kaggle.com/code/jiweiliu/rapids-cudf-feature-engineering-xgb.\nIn practice, compute the subtraction of a bunch of B, D, S features minus two P (Payment) features: P_2, P_3\n\n- **CV: 0.7953, +0.0004** compared with (d)\n- 20.9min training time\n- 1157 features\n\n### (h) Add lag features\n\nReplicate the feature engineering by thedevastator described in described in\nhttps://www.kaggle.com/code/thedevastator/lag-features-are-all-you-need\n\n- **CV: 0.7937, -0.0016** compared with (g)\n- 24.2min training time\n- 1511 features\n\nThis is very weird, as these features come from one of the best publicly available notebooks.\nMy hypothesis is that too many (>1500) features combined with an aggressive 0.5 colsample_bytree leads to\nsome the model missing some interactions, but I am curious of the reader's opinion.\n\n### (i) Illidan7's Feature Set\n\nhttps://www.kaggle.com/code/illidan7/amex-basic-feature-engineering-1500-features\n\n- **CV: 0.7950**\n- 41.2min training time\n- 1486 features\n\n### (j) Add ragnar123' first - mean lag feature\n\n- **CV: 0.7956**\n- 27.9min training time\n- 974 features\n\n\n## Conclusions and next steps\n\n- Rounding seems to be beneficial, should we round the values before performing the aggreagations, then?\n- Adding first, mean, std, min, max aggregations has the biggest impact on performance, but they are almost 900 additional features. How to identify the ones that actually bring benefits?\n- It is not clear if dropping correlated features has an effect. Why doesn't it reduce the training time? Is it because we need more trees?\n- The difference features give a small (4 basis points) boost. I believe they are worth using as they are only 40.\n- Next, I want to check 's https://www.kaggle.com/code/ragnar123/amex-lgbm-dart-cv-0-7977", "1864319": "Hi Andrea, update on my end: I went live with my XGB notebook today: https://www.kaggle.com/code/roberthatch/xgboost-pyramid-test-predictions \n\nCV 0.7968 (~0.7965-7970); LB 0.797 and edged out other 0.797 public notebooks.\n\nThe hyper-parameters aren't all that optimized, the feature engineering is unoptimized and pretty untested. But I'm *very* happy with the results of my \"Pyramid\" experiment.\n\nEssentially, I tried using other methods besides DART that I suspected might reduce \"over-specialization\", without the performance penalty that DART + XGB + GPU has. It was very successful, using two or three tricks, depending how you count it:\n* Boosted forest early on, very directly reduces impact of initial tree, and a little more evenly spreads even at later points (with forest size of 100, then tree 101 has same impact as tree 200).\n* Rescale the prediction results retroactively to give more 'room' for impact from later trees.\n* These two innovations supported by the basic method of stopping a model early, and using it as the basis of the next model. There might be a more elegant way to do that, but the one I found that definitely works is to input the predictions from model A before starting training model B using \"set_base_margin\".", "1853206": "Great work, @mrandri19 , my best XGB gbt has CV 0.7958 and LB 0.796 (seed 42). Point (h) caused me the same question - I don't know why, but lags \"last-first\" and \"last-avg\" work worse together than if you leave one of them. Perhaps dart is more loyal to the number of features than gbt, but it's still strange. I also tried adding kurtosis and skew of most important features, but it didn't improve the result.", "1858755": "The information provided is so clear and easy to understand, Awesome post :)\nLooking forward to see more !!!\nlet's grow together.", "1857384": "Great work, @mrandri19 , my best XGB gbt has CV 0.7958 and LB 0.796 (seed 42). Point (h) caused me the same question - I don't know why, but lags \"last-first\" and \"last-avg\" work worse together than if you leave one of them. Perhaps dart is more loyal to the number of features than gbt, but it's still strange. I also tried adding kurtosis and skew of most important features, but it didn't improve the result.", "1862506": "@mrandri19 How are your research progressing? A week ago I was convinced that using DART is very early and it was true) At the moment I have 0.7973 CV(5) and 0.797 LB! And you know what? This is not the limit! I think the possible best CV result is around 0.7975 for XGB gbt.", "1861261": "Thank you for sharing your ablations studies ! \n", "1856280": "thanks for sharing. on the std of the models, i wonder what std you get from 5 folds? mine is about 0.004. ", "1853608": "In my experience (well, experiments), adding even more good features only really helps once you decrease colsample even lower AND lower Learning Rate. (0.01 eta and 35% col for me with ~1300 features)", "1853600": "Thanks for sharing your ablation studies! \nAre you using all of the features in the dataset? If not, what feature selection method do you use?\n\n", "1853486": "great work, thanks for sharing your ideas!", "1853294": "Nice .. did lag features decrease ur CV/LB . I think one lag feature which ragner described did certainly help ", "1852203": "Always interesting to look at good feature engineering experiments, well done!", "1852120": "Very well curated and summarized!", "1857388": "thanks for this help for us " } }