First and foremost we would like to thank Raddar for his cleaned and compressed dataset [1] (which we used) and Martin for his two magnificent DART notebooks [2][3]. We finally used a stacking of XGBoost, LightGBM and TabNet. When blending our solution with that of Martin we obtained a Private score of 0.80761, and playing the hypothetical what if game this would have landed around 138th on the LB. However, although acutely aware that our solution was not competitive, we still went ahead with selecting only our own work, and followed the sage advice of Raddar in submitting our best CV, and (our best CV + our best LB)/2 works. Doing this left our CV score on the wrong side of the CV of the work published by Martin, and consequently way down the LB, but with no regrets whatsoever.
\nThings perhaps of mention:\nFeature engineering: We spent an inordinate amount of time with the features in conjunction with the 13GB GPU memory limit on kaggle, and with each new feature sadly eventually having to sacrifice in the end around 30 other features (team-mate @danielhanchen helped with that process) to stay within the notebook memory. In retrospect we should really have heavily sub-sampled the training data (if it wasn't for the noisy D component in the metric needing lots of data that would definitely be the way to go) in order to keep as many of these features as possible as almost all of them added just a little something (we did try the random under-sampling RUSBoostClassifier from imbalanced-ensemble, but it did not work well).\nAs well as a lot of futher cleaning to remove outliers etc, things of particular note were:
\nS_8 and S_13 seemed reminiscent of some sort of FICO credit rating scores. With that in mind we created an additional two new binary features with a split (np.where) chosen by fitting just these features alone with a RandomForest 'stump' with the idea of 'good' or 'bad' creditor in mind.S_3/S_7 helped.D_115, D_118 and D_119 also helped a littleround(2) trick suggested by Jiwei Liu [4]Estimators:
\nnum_parallel_tree=7 parameter, which slowed down the training by around 7 times, but made for a smoother training curve and a CV with a lower standard deviation.max_depth=4) as going deeper didn't really seem to helpThe OOF predictions of these estimators were then calibrated and then fed into a stacking ensemble.
\nWe were also aware that the distribution of some of the features in the test data were significantly different from those in the training data (some wonderful notebooks on this were made on this by Pavel Vodolazov [6][7]). In view of the fact that usually tree based estimators cannot extrapolate we tried out the linear-tree estimator, and also the gblinear booster in XGBoost, but neither were used in the end; they may well have eventually performed better on the test data for the features that had covariate shifts, but there was little way of telling beforehand.
All in all a very enjoyable competition and many thanks to AmEx for providing us with tabular data to play with, and kaggle for the computational resources!
\nAll the best,\ncarl
", "messages": [ { "id": "1913368", "postDate": "08/25/2022 09:50:23", "content": "First and foremost we would like to thank Raddar for his cleaned and compressed dataset [1] (which we used) and Martin for his two magnificent DART notebooks [2][3]. We finally used a stacking of XGBoost, LightGBM and TabNet. When blending our solution with that of Martin we obtained a Private score of 0.80761, and playing the hypothetical what if game this would have landed around 138th on the LB. However, although acutely aware that our solution was not competitive, we still went ahead with selecting only our own work, and followed the sage advice of Raddar in submitting our best CV, and (our best CV + our best LB)/2 works. Doing this left our CV score on the wrong side of the CV of the work published by Martin, and consequently way down the LB, but with no regrets whatsoever.
\nThings perhaps of mention:\nFeature engineering: We spent an inordinate amount of time with the features in conjunction with the 13GB GPU memory limit on kaggle, and with each new feature sadly eventually having to sacrifice in the end around 30 other features (team-mate @danielhanchen helped with that process) to stay within the notebook memory. In retrospect we should really have heavily sub-sampled the training data (if it wasn't for the noisy D component in the metric needing lots of data that would definitely be the way to go) in order to keep as many of these features as possible as almost all of them added just a little something (we did try the random under-sampling RUSBoostClassifier from imbalanced-ensemble, but it did not work well).\nAs well as a lot of futher cleaning to remove outliers etc, things of particular note were:
\nS_8 and S_13 seemed reminiscent of some sort of FICO credit rating scores. With that in mind we created an additional two new binary features with a split (np.where) chosen by fitting just these features alone with a RandomForest 'stump' with the idea of 'good' or 'bad' creditor in mind.S_3/S_7 helped.D_115, D_118 and D_119 also helped a littleround(2) trick suggested by Jiwei Liu [4]Estimators:
\nnum_parallel_tree=7 parameter, which slowed down the training by around 7 times, but made for a smoother training curve and a CV with a lower standard deviation.max_depth=4) as going deeper didn't really seem to helpThe OOF predictions of these estimators were then calibrated and then fed into a stacking ensemble.
\nWe were also aware that the distribution of some of the features in the test data were significantly different from those in the training data (some wonderful notebooks on this were made on this by Pavel Vodolazov [6][7]). In view of the fact that usually tree based estimators cannot extrapolate we tried out the linear-tree estimator, and also the gblinear booster in XGBoost, but neither were used in the end; they may well have eventually performed better on the test data for the features that had covariate shifts, but there was little way of telling beforehand.
All in all a very enjoyable competition and many thanks to AmEx for providing us with tabular data to play with, and kaggle for the computational resources!
\nAll the best,\ncarl
", "rawMarkdown": "First and foremost we would like to thank Raddar for his cleaned and compressed dataset [[1]](https://www.kaggle.com/datasets/raddar/amex-data-integer-dtypes-parquet-format) (which we used) and Martin for his two magnificent DART notebooks [[2]](https://www.kaggle.com/code/ragnar123/amex-lgbm-dart-cv-0-7963)[[3]](https://www.kaggle.com/code/ragnar123/amex-lgbm-dart-cv-0-7977). We finally used a stacking of XGBoost, LightGBM and TabNet. When blending our solution with that of Martin we obtained a Private score of 0.80761, and playing the hypothetical *what if* game this would have landed around 138th on the LB. However, although acutely aware that our solution was not competitive, we still went ahead with selecting only our own work, and followed the sage advice of Raddar in submitting our best CV, and (our best CV + our best LB)/2 works. Doing this left our CV score on the wrong side of the CV of the work published by Martin, and consequently way down the LB, but with no regrets whatsoever. \n\nThings perhaps of mention:\nFeature engineering: We spent an inordinate amount of time with the features in conjunction with the 13GB GPU memory limit on kaggle, and with each new feature sadly eventually having to sacrifice in the end around 30 other features (team-mate @danielhanchen helped with that process) to stay within the notebook memory. In retrospect we should really have heavily sub-sampled the training data (if it wasn't for the noisy *D* component in the metric needing lots of data that would definitely be the way to go) in order to keep as many of these features as possible as almost all of them added just a little something (we did try the random under-sampling [RUSBoostClassifier](https://imbalanced-ensemble.readthedocs.io/en/latest/api/ensemble/_autosummary/imbalanced_ensemble.ensemble.RUSBoostClassifier.html) from [imbalanced-ensemble](https://imbalanced-ensemble.readthedocs.io/en/latest/index.html), but it did not work well).\nAs well as a lot of futher cleaning to remove outliers *etc*, things of particular note were:\n* The features `S_8` and `S_13` seemed reminiscent of some sort of FICO credit rating scores. With that in mind we created an additional two new binary features with a split (`np.where`) chosen by fitting just these features alone with a RandomForest 'stump' with the idea of 'good' or 'bad' creditor in mind.\n* As well as generating the usual aggregated features for all of the (up to) 13 statements, we found that also using such aggregations just over the last 3 statements, and the last 6 statements, proved to produce some very informative features.\n* We found that the feature `S_3/S_7` helped.\n* We found that the average of the three features `D_115`, `D_118` and `D_119` also helped a little\n* On the float features we used the `round(2)` trick suggested by Jiwei Liu [[4]](https://www.kaggle.com/code/jiweiliu/amex-catboost-rounding-trick)\n* Our final number of features after aggregations was 1145\n\nEstimators:\n* We started with also using CatBoost, but in the end we found the CV scores were not on par with those of XGBoost and LGBM\n* We used DART with LGBM thanks to the work of Martin (but not with XGBoost; incredibly slow!)\n* Our only novelty with XGBoost was using the `num_parallel_tree=7` parameter, which slowed down the training by around 7 times, but made for a smoother training curve and a CV with a lower standard deviation.\n* As per the magnificent notbook by Chris Deotte [[5]](https://www.kaggle.com/code/cdeotte/xgboost-starter-0-79391) we used surprisingly shallow estimators (`max_depth=4`) as going deeper didn't really seem to help\n* Combined multiple runs of each estimator, each using different CV splitting seeds\n* Although TabNet alone did not have a spectacular CV score despite the best efforts of my team-mate @optimo , it helped every ensemble it touched.\n\nThe OOF predictions of these estimators were then calibrated and then fed into a stacking ensemble.\n\nWe were also aware that the distribution of some of the features in the test data were significantly different from those in the training data (some wonderful notebooks on this were made on this by Pavel Vodolazov [[6]](https://www.kaggle.com/code/pavelvod/amex-eda-revealing-time-patterns-of-features)[[7]](https://www.kaggle.com/code/pavelvod/amex-eda-even-more-insane-time-patterns-revealed)). In view of the fact that usually tree based estimators cannot extrapolate we tried out the [`linear-tree`](https://github.com/cerlymarco/linear-tree) estimator, and also the `gblinear` booster in XGBoost, but neither were used in the end; they may well have eventually performed better on the test data for the features that had covariate shifts, but there was little way of telling beforehand. \n\n\nAll in all a very enjoyable competition and many thanks to AmEx for providing us with tabular data to play with, and kaggle for the computational resources!\n\nAll the best,\ncarl", "votes": null }, { "id": "1913413", "postDate": "08/25/2022 10:12:31", "content": "great Carl!, All respect for you for selecting only your own work, I didn't have the same courage but I really think that is what kaggle should be about.
", "rawMarkdown": "great Carl!, All respect for you for selecting only your own work, I didn't have the same courage but I really think that is what kaggle should be about.", "votes": null }, { "id": "1913428", "postDate": "08/25/2022 10:18:09", "content": "Dear @pabuoro
\nThanks! Despite the very modest result I think that, just as with science, it is useful sometimes to read about what didn't make the cut as well as the wonderful work that did!
\nAll the best,
\ncarl
Hi Carl. thanks for the feedback ! I also spent quite some time on time aggregation (first 6 v.s. last 6) and the usage of linear model for boosting. I also didn't get much out of it.
", "rawMarkdown": "Hi Carl. thanks for the feedback ! I also spent quite some time on time aggregation (first 6 v.s. last 6) and the usage of linear model for boosting. I also didn't get much out of it.", "votes": null }, { "id": "1914987", "postDate": "08/26/2022 15:33:14", "content": "Wonderful collaborating with you @carlmcbrideellis and @optimo! Apologies I wasn't that helpful in the end.
", "rawMarkdown": "Wonderful collaborating with you @carlmcbrideellis and @optimo! Apologies I wasn't that helpful in the end.", "votes": null }, { "id": "1915089", "postDate": "08/26/2022 16:49:35", "content": "Dear @danielhanchen
\nIt was wonderful having you onboard! Working together as a team is so much greater than the sum of all its parts; meeting new collaborators and learning together is one of the greatest rewards there is on kaggle, or anywhere else for that matter!
\nUn gran abrazo,
\ncarl
During the competition I learned some really interesting tricks for memory management.
\nXGBoost allows batched loading which really helps (store features in CPU, load into GPU). And float16 is viable, saved in pickle format. (though I probably should've kept the most important features float32). I managed 10k features in kaggle notebook! But then didn't spend the time on generating more features so stayed at \"only\" 3k.
\nMemory management was tough, for sure!
", "rawMarkdown": "During the competition I learned some really interesting tricks for memory management. \n\nXGBoost allows batched loading which really helps (store features in CPU, load into GPU). And float16 is viable, saved in pickle format. (though I probably should've kept the most important features float32). I managed 10k features in kaggle notebook! But then didn't spend the time on generating more features so stayed at \"only\" 3k. \n\nMemory management was tough, for sure!", "votes": null }, { "id": "1915123", "postDate": "08/26/2022 17:20:36", "content": "Dear @roberthatch
\nThat is really interesting to know!
\nI did vaguely think about paying for some cloud time; apart from big ones like P_2 most of the features contributed very little but something, and all together they do make a difference, so I didn't enjoy dropping any of them (even though the permutation importance work of AmbrosM indicated that some of the features were decidedly unhelpful. Chris Deotte found that \"There are only 35 features [of his 1366 features] that are statistically significant hurt the model.\") and would have liked to keep them all and let each estimator do its work. I cant remember exactly where but I remember one winner back in the golden age of kaggle tabular competitions saying that in general he never dropped any features.
So resource availability and/or management was a big factor in this competition. There was an interesting topic posted a little back by James Trotman \"Incoming Feature: Paid User Plans?\" which led to some thought provoking questions about the long term future regarding the size of kaggle competitions w.r.t. kaggle resources.
\nAll the best,
\ncarl
The short version of tricks I used:
\nAnd lots of smaller batch jobs and multiple notebooks. Even for 3k features, I had three separate feature engg notebooks and combined them later.
", "rawMarkdown": "The short version of tricks I used:\n* Convert subset of features to float16 in CPU memory, save in pickle format\n* Update iterLoad___ class in xgb starter notebook to handle list of dataframes instead of single. That way if I had 9gb features, I didn't have to concat them into single dataframes and run out of memory. (2 bytes times 10,000 times 458,000 is ~9GB pure data). I never found a way to concat three 3GB dataframes into single frame in 16gb mem. Probably something like dask could've worked better(?) but I had limited time to learn yet another tool. \n\nAnd lots of smaller batch jobs and multiple notebooks. Even for 3k features, I had three separate feature engg notebooks and combined them later.", "votes": null }, { "id": "1915146", "postDate": "08/26/2022 17:51:06", "content": "Dear @roberthatch
\nOne thing I would really like to see in the future would be kaggle tabular competitions (!) that perhaps move away from monolithic csv files and perhaps provide a connection to a database via PySpark. For example, as of Spark 3.2 one can use pandas via import pyspark.pandas as ps without the need for UDFs, and would also provide an environment where people could practice their SQL skills (something that all employers want) and, thanks to lazy evaluation, may not be so taxing on notebook CPU/GPU…
All the best,
\ncarl
great Carl!, All respect for you for selecting only your own work, I didn't have the same courage but I really think that is what kaggle should be about.
", "votes": null, "replies": [ { "id": 1913428, "author_name": "carlmcbrideellis", "author_url": "", "post_date": "08/25/2022 10:18:09", "content": "Dear @pabuoro
\nThanks! Despite the very modest result I think that, just as with science, it is useful sometimes to read about what didn't make the cut as well as the wonderful work that did!
\nAll the best,
\ncarl
Hi Carl. thanks for the feedback ! I also spent quite some time on time aggregation (first 6 v.s. last 6) and the usage of linear model for boosting. I also didn't get much out of it.
", "votes": null, "replies": [] }, { "id": 1914987, "author_name": "danielhanchen", "author_url": "", "post_date": "08/26/2022 15:33:14", "content": "Wonderful collaborating with you @carlmcbrideellis and @optimo! Apologies I wasn't that helpful in the end.
", "votes": null, "replies": [ { "id": 1915089, "author_name": "carlmcbrideellis", "author_url": "", "post_date": "08/26/2022 16:49:35", "content": "Dear @danielhanchen
\nIt was wonderful having you onboard! Working together as a team is so much greater than the sum of all its parts; meeting new collaborators and learning together is one of the greatest rewards there is on kaggle, or anywhere else for that matter!
\nUn gran abrazo,
\ncarl
During the competition I learned some really interesting tricks for memory management.
\nXGBoost allows batched loading which really helps (store features in CPU, load into GPU). And float16 is viable, saved in pickle format. (though I probably should've kept the most important features float32). I managed 10k features in kaggle notebook! But then didn't spend the time on generating more features so stayed at \"only\" 3k.
\nMemory management was tough, for sure!
", "votes": null, "replies": [ { "id": 1915123, "author_name": "carlmcbrideellis", "author_url": "", "post_date": "08/26/2022 17:20:36", "content": "Dear @roberthatch
\nThat is really interesting to know!
\nI did vaguely think about paying for some cloud time; apart from big ones like P_2 most of the features contributed very little but something, and all together they do make a difference, so I didn't enjoy dropping any of them (even though the permutation importance work of AmbrosM indicated that some of the features were decidedly unhelpful. Chris Deotte found that \"There are only 35 features [of his 1366 features] that are statistically significant hurt the model.\") and would have liked to keep them all and let each estimator do its work. I cant remember exactly where but I remember one winner back in the golden age of kaggle tabular competitions saying that in general he never dropped any features.
So resource availability and/or management was a big factor in this competition. There was an interesting topic posted a little back by James Trotman \"Incoming Feature: Paid User Plans?\" which led to some thought provoking questions about the long term future regarding the size of kaggle competitions w.r.t. kaggle resources.
\nAll the best,
\ncarl
The short version of tricks I used:
\nAnd lots of smaller batch jobs and multiple notebooks. Even for 3k features, I had three separate feature engg notebooks and combined them later.
", "votes": null, "replies": [] }, { "id": 1915146, "author_name": "carlmcbrideellis", "author_url": "", "post_date": "08/26/2022 17:51:06", "content": "Dear @roberthatch
\nOne thing I would really like to see in the future would be kaggle tabular competitions (!) that perhaps move away from monolithic csv files and perhaps provide a connection to a database via PySpark. For example, as of Spark 3.2 one can use pandas via import pyspark.pandas as ps without the need for UDFs, and would also provide an environment where people could practice their SQL skills (something that all employers want) and, thanks to lazy evaluation, may not be so taxing on notebook CPU/GPU…
All the best,
\ncarl