A huge huge thank you to @andrew60909 and @ryotak12 for an excellent team experience. We learned a lot and had fun, and I appreciate their patience with my extended, largely unsuccessful efforts to make a transformer work really well 😂. Thanks also to Amex and Kaggle for hosting this interesting competition and to @raddar and everyone else who shared resources that made this competition much more accessible.
\nI've made my personal code public here: https://github.com/JEddy92/amex_default_kaggle. I tried to create a reasonably general structure for division of FE and modeling, with reusable functionality for training and logging results across a variety of models. There is still some clean up to do and it's not as polished as would be ideal!
\nFeel free to ask us anything if you have questions about our approach!
\nWe used logistic regression to ensemble ~60 sets of predictions with diversity obtained from different models, feature sets, and data views. Models included the usual mix of gradient boosters (LGBM, LGBM dart, XGBoost, CatBoost), MLPs, TabNet, and Transformers with a few special tricks. Data views included aggregated data, fully flattened data (13 raw features per customer), sequential data (Transformer), and augmented data (shifted statements forward by 1 and appended to original). Very detailed feature information to follow -- they are split across our 3 different team members' different models, but this aims to be a complete compilation of what we all used.
\n\\(\\mathcal{D}_{fulltrain}\\) : Original full size train-set (5531451 rows)
\n\\(\\mathcal{D}_{train}\\) : Train-set (458913 rows)
\n\\(\\mathcal{D}_{fulltest}\\) : Original full size test-set
\n\\(\\mathcal{D}_{test}\\) : Test-set
\nThe train-sets with the best CV score for each of us are generated by \\(\\mathcal{D}_{fulltrain}\\) with these features:
\nNumeric
\nTrend : Oldest month as 1 and the newest month as 13, take the average over the following periods
\nCategory
\nP_2 mean (across train and test) of each categoryPCA
\nKNN-based target encoding methods
\nNested model
\nWe added the labels to \\(\\mathcal{D}_{fulltrain}\\), then trained a LGBM model. The purpose of doing this is to capture what kinds of records, behavior and attributes will cause default. Instead of aggregating features into 458913 rows, this method can let the model learn some extra information from a more \"base\" level. After that, for each customer_ID we will have a \"predicted target\" that has the same length as the number of records in each customer_ID.
Finally, to merge it into \\(\\mathcal{D}_{train}\\) : Train-set (458913 rows), we aggregate similarly to other features: mean, std, max, min, first, last, man-min, last-mean, last/std.
An interesting point here is that the last value of our \"predicted value\" with \\(\\mathcal{D}_{fulltrain}\\) scores 0.785 on the Amex metric, while the score of the max value is around 0.62 -- indicating the importance of the last value.
Target/Count encoding: On \\(\\mathcal{D}_{train}\\)
Target encoding with \\(\\mathcal{D}_{fulltrain}\\)
Other functions to aggregate \\(\\mathcal{D}_{fulltrain}\\)
\nClustering methods: Kmeans, DBSCAN on important feature(e.g. P_2_last or B_1_last(including other measurements)). Using raw clusters largely degraded our CV and we're not sure about the reason, so count/target encoding was used here.
pred0 feature: using full data to predict specific important features via 2 datasets: \\(\\mathcal{D}_{train}\\) and the fullsize one. For example, if we want to predict P_2 in the former case or predict P_2_last in the latter case, we will excluded all \"P-related\" feature and train a lgbm model on it.
Probit, Logit models with regularization on different parts of features (e.g. S-related, B-related sets), like how we make pred0 features, we also exclude related features when we construct the predictions
Time-series based methods to predict important features in the next statement(e.g. P_2, B_1….)
Curve fitting, smoothing, denoising:
\ncustomer_ID that have >10 statements.Leaf-embedding: An old trick. We mixed the approaches from this paper and this kaggle code. We use lgbm with 1round + huge amount of leaves + super high regularization to embed the \"whole\" data, which has only 1 dimension. And of course using only categorical data or expanding the embedding matrix to the dimension of nrounds is also possible.
Most of the above methods are not really able to improve the best CV in a single model, so we turn to utilize it in different models to generate more diversity. Most of our lgbm models with the above selected methods can have a score ranging from CV.796~.801/ PbLB.797~.80 / PriLB.804~.806, and some of them can generate pretty good diversity. But we are not going to do a detailed ablation study to check the improvement in the PrivateLB of each method :P
GBDT
\nTabNet
\nground truth - tabnet’s prediction with LightGBM and then finalize predictions as tabnet's prediction + resid predictionMLP
\nTransformer
\nP_2 value from entire train+test. Data Augmented ~2x by statement shift strategyExample Dart hyper-parameters from Angus, no special sauce here:) :
\nlgb_param <- list(boosting_type = 'dart',\n objective = \"binary\",\n metric = amex,\n learning_rate = 0.02,\n num_leaves = 48,\n feature_fraction = 0.1,\n #bagging_freq = 1,\n #bagging_fraction = 1,\n min_child_weight = 1,\n lambda_l1 = 1,\n lambda_l2 = 64,\n skip_drop = 0.8\n)\nPrivate set specialization: We tried using the KNN result mentioned above to estimate which samples are closer to Private LB data. In particular, we calculated the nearest 500 points for each sample, then the ratio of each set in those 500 points. After that, we took the top 30% of samples which are closer to private LB and ran a logistic regression to decide the weight of our final ensemble. But unfortunately, the CV score dropped a lot. The CV of the top 30% data is ~0.88, but overall(450k data) Amex-score is ~0.7812. We also tried the top 30% \"public ratio\" one. The CV is ~0.7259 and the overall CV is 0.7896711. It's possible that using adversarial validation predictions trained on top importance features would be a more reliable option.
Transfer learning on P_2: this idea unfortunately came up very close to the end before it could pay off and probably was worth pursuing further. This was an attempt at exploiting the test data - pretrain a transformer on train+test to predict P_2 (because it's strongly related to the target), then fine-tune on the actual target with train. This is like a worse version of the knowledge distillation techniques that were used to exploit the test data more for neural net training.
Psuedo labeling: We only got 0.00005 improvement on CV (LB doesn't seem tio change), and we have tuned the size and what proportion of test-set can be used for it but didn't have real luck. However, log loss and AUC may be high enough to do this more optimally and get a considerable improvement.
Ensembling methods, we tried lgbm with 50 seeds bagging, LR and optimization methods like md1*par1 + md2*par2.... with Nelder-Mead or L-BFGS-B solver. LR performed the best (CV.8031), and lgbm seemed to be severely overfitting.
We fell hard from 3rd Public -> 21st Private, with 6 submissions that would have scored gold. In hindsight, CV here did not seem to be a great measurement for the private LB, even though CV improvement aligned very well with public LB. We probably had some bad luck, but 2 things we may have benefited from doing differently are:
\nA huge huge thank you to @andrew60909 and @ryotak12 for an excellent team experience. We learned a lot and had fun, and I appreciate their patience with my extended, largely unsuccessful efforts to make a transformer work really well 😂. Thanks also to Amex and Kaggle for hosting this interesting competition and to @raddar and everyone else who shared resources that made this competition much more accessible.
\nI've made my personal code public here: https://github.com/JEddy92/amex_default_kaggle. I tried to create a reasonably general structure for division of FE and modeling, with reusable functionality for training and logging results across a variety of models. There is still some clean up to do and it's not as polished as would be ideal!
\nFeel free to ask us anything if you have questions about our approach!
\nWe used logistic regression to ensemble ~60 sets of predictions with diversity obtained from different models, feature sets, and data views. Models included the usual mix of gradient boosters (LGBM, LGBM dart, XGBoost, CatBoost), MLPs, TabNet, and Transformers with a few special tricks. Data views included aggregated data, fully flattened data (13 raw features per customer), sequential data (Transformer), and augmented data (shifted statements forward by 1 and appended to original). Very detailed feature information to follow -- they are split across our 3 different team members' different models, but this aims to be a complete compilation of what we all used.
\n\\(\\mathcal{D}_{fulltrain}\\) : Original full size train-set (5531451 rows)
\n\\(\\mathcal{D}_{train}\\) : Train-set (458913 rows)
\n\\(\\mathcal{D}_{fulltest}\\) : Original full size test-set
\n\\(\\mathcal{D}_{test}\\) : Test-set
\nThe train-sets with the best CV score for each of us are generated by \\(\\mathcal{D}_{fulltrain}\\) with these features:
\nNumeric
\nTrend : Oldest month as 1 and the newest month as 13, take the average over the following periods
\nCategory
\nP_2 mean (across train and test) of each categoryPCA
\nKNN-based target encoding methods
\nNested model
\nWe added the labels to \\(\\mathcal{D}_{fulltrain}\\), then trained a LGBM model. The purpose of doing this is to capture what kinds of records, behavior and attributes will cause default. Instead of aggregating features into 458913 rows, this method can let the model learn some extra information from a more \"base\" level. After that, for each customer_ID we will have a \"predicted target\" that has the same length as the number of records in each customer_ID.
Finally, to merge it into \\(\\mathcal{D}_{train}\\) : Train-set (458913 rows), we aggregate similarly to other features: mean, std, max, min, first, last, man-min, last-mean, last/std.
An interesting point here is that the last value of our \"predicted value\" with \\(\\mathcal{D}_{fulltrain}\\) scores 0.785 on the Amex metric, while the score of the max value is around 0.62 -- indicating the importance of the last value.
Target/Count encoding: On \\(\\mathcal{D}_{train}\\)
Target encoding with \\(\\mathcal{D}_{fulltrain}\\)
Other functions to aggregate \\(\\mathcal{D}_{fulltrain}\\)
\nClustering methods: Kmeans, DBSCAN on important feature(e.g. P_2_last or B_1_last(including other measurements)). Using raw clusters largely degraded our CV and we're not sure about the reason, so count/target encoding was used here.
pred0 feature: using full data to predict specific important features via 2 datasets: \\(\\mathcal{D}_{train}\\) and the fullsize one. For example, if we want to predict P_2 in the former case or predict P_2_last in the latter case, we will excluded all \"P-related\" feature and train a lgbm model on it.
Probit, Logit models with regularization on different parts of features (e.g. S-related, B-related sets), like how we make pred0 features, we also exclude related features when we construct the predictions
Time-series based methods to predict important features in the next statement(e.g. P_2, B_1….)
Curve fitting, smoothing, denoising:
\ncustomer_ID that have >10 statements.Leaf-embedding: An old trick. We mixed the approaches from this paper and this kaggle code. We use lgbm with 1round + huge amount of leaves + super high regularization to embed the \"whole\" data, which has only 1 dimension. And of course using only categorical data or expanding the embedding matrix to the dimension of nrounds is also possible.
Most of the above methods are not really able to improve the best CV in a single model, so we turn to utilize it in different models to generate more diversity. Most of our lgbm models with the above selected methods can have a score ranging from CV.796~.801/ PbLB.797~.80 / PriLB.804~.806, and some of them can generate pretty good diversity. But we are not going to do a detailed ablation study to check the improvement in the PrivateLB of each method :P
GBDT
\nTabNet
\nground truth - tabnet’s prediction with LightGBM and then finalize predictions as tabnet's prediction + resid predictionMLP
\nTransformer
\nP_2 value from entire train+test. Data Augmented ~2x by statement shift strategyExample Dart hyper-parameters from Angus, no special sauce here:) :
\nlgb_param <- list(boosting_type = 'dart',\n objective = \"binary\",\n metric = amex,\n learning_rate = 0.02,\n num_leaves = 48,\n feature_fraction = 0.1,\n #bagging_freq = 1,\n #bagging_fraction = 1,\n min_child_weight = 1,\n lambda_l1 = 1,\n lambda_l2 = 64,\n skip_drop = 0.8\n)\nPrivate set specialization: We tried using the KNN result mentioned above to estimate which samples are closer to Private LB data. In particular, we calculated the nearest 500 points for each sample, then the ratio of each set in those 500 points. After that, we took the top 30% of samples which are closer to private LB and ran a logistic regression to decide the weight of our final ensemble. But unfortunately, the CV score dropped a lot. The CV of the top 30% data is ~0.88, but overall(450k data) Amex-score is ~0.7812. We also tried the top 30% \"public ratio\" one. The CV is ~0.7259 and the overall CV is 0.7896711. It's possible that using adversarial validation predictions trained on top importance features would be a more reliable option.
Transfer learning on P_2: this idea unfortunately came up very close to the end before it could pay off and probably was worth pursuing further. This was an attempt at exploiting the test data - pretrain a transformer on train+test to predict P_2 (because it's strongly related to the target), then fine-tune on the actual target with train. This is like a worse version of the knowledge distillation techniques that were used to exploit the test data more for neural net training.
Psuedo labeling: We only got 0.00005 improvement on CV (LB doesn't seem tio change), and we have tuned the size and what proportion of test-set can be used for it but didn't have real luck. However, log loss and AUC may be high enough to do this more optimally and get a considerable improvement.
Ensembling methods, we tried lgbm with 50 seeds bagging, LR and optimization methods like md1*par1 + md2*par2.... with Nelder-Mead or L-BFGS-B solver. LR performed the best (CV.8031), and lgbm seemed to be severely overfitting.
We fell hard from 3rd Public -> 21st Private, with 6 submissions that would have scored gold. In hindsight, CV here did not seem to be a great measurement for the private LB, even though CV improvement aligned very well with public LB. We probably had some bad luck, but 2 things we may have benefited from doing differently are:
\nthanks! its very helpful for beginner how to approach problem
", "rawMarkdown": "thanks! its very helpful for beginner how to approach problem", "votes": 1 }, { "id": 1920201, "postDate": "2022-08-31T01:52:54.433Z", "content": "Thanks for sharing your solution @aquatic! Very insightful and it shows the amount of work that you put in!
", "rawMarkdown": "Thanks for sharing your solution @aquatic! Very insightful and it shows the amount of work that you put in!", "votes": 1 }, { "id": 1918478, "postDate": "2022-08-29T15:50:58.550Z", "content": "Wow, you did a lot of great work!
\nI know you varied the data for your models, but was it the same 1.2k base after feature selection for all models using the \"normal\" set of data? I wonder if \"less aggressive feature selection\" is another possible thing that, in retrospect, could've helped the private lb score?
", "rawMarkdown": "Wow, you did a lot of great work!\n\nI know you varied the data for your models, but was it the same 1.2k base after feature selection for all models using the \"normal\" set of data? I wonder if \"less aggressive feature selection\" is another possible thing that, in retrospect, could've helped the private lb score?", "votes": 1, "replies": [ { "id": 1918500, "postDate": "2022-08-29T16:10:03.987Z", "content": "Note I'm asking because I see so few differences that seem critical (that I noticed) between your solution and top 2 solutions.
\nHmm, maybe top two had more emphasis on a normalization factor? You had monthly based ranking, but top solution had ranking of a single customer's statements across months AND ranking across customers for a given month, AND aggregated those things. (Also could feature selection have removed some of the monthly ranking or any other normalization that you did do?)
", "rawMarkdown": "Note I'm asking because I see so few differences that seem critical (that I noticed) between your solution and top 2 solutions. \n\nHmm, maybe top two had more emphasis on a normalization factor? You had monthly based ranking, but top solution had ranking of a single customer's statements across months AND ranking across customers for a given month, AND aggregated those things. (Also could feature selection have removed some of the monthly ranking or any other normalization that you did do?)", "votes": 1 }, { "id": 1918718, "postDate": "2022-08-29T19:17:40.697Z", "content": "Thanks!
\nThat method was specifically what @ryotak12 used for selection (especially for some of the feature sets that would otherwise just be enormous). For my models, I actually didn't use selection, more like feed-forward additions (and actually included different/older versions of models that had a subset of the total feature set I used, these made the cut in the ensemble). Feature selection is hard, especially on this problem, and I thought it'd be more time efficient to prioritize diversity for my work.
\nIf I'm remembering correctly, top 2 solutions both didn't use stacking so that may be one differentiating point. Perhaps our stacking overfits to CV / Public LB relative to the highest scoring Private solutions.
", "rawMarkdown": "Thanks!\n\nThat method was specifically what @ryotak12 used for selection (especially for some of the feature sets that would otherwise just be enormous). For my models, I actually didn't use selection, more like feed-forward additions (and actually included different/older versions of models that had a subset of the total feature set I used, these made the cut in the ensemble). Feature selection is hard, especially on this problem, and I thought it'd be more time efficient to prioritize diversity for my work. \n\nIf I'm remembering correctly, top 2 solutions both didn't use stacking so that may be one differentiating point. Perhaps our stacking overfits to CV / Public LB relative to the highest scoring Private solutions.", "votes": 2 }, { "id": 1919011, "postDate": "2022-08-30T04:09:45.923Z", "content": "As Joe said, only my feature set was applied to this selection method.
\nI generated a large number of features at beginning and could not train them all 20k over features.
\nSo, I had to select features due to training time.
Congratulations Joe!
\nCould you please share more about feature selection part? After you get the permutation importance of 20k features, how do you select 1200 features? Did you just pick the top 1200 features? If then, how do you decide how many features? Thanks
I will answer your question.
\nI could not train 20k features at once, because it takes too long time and memory.
\nSo, at first, I divided it into 15 subsets and experimented with 1500 features each.
\nAt this stage, I selected features that decrease by more than 1e-5 in G-metric.
\nThe reason that this threshold was chosen is that experiments had shown that features that do not contribute to improving metric can also results -1e-5.
\nThese are repeated and finally the experiment is conducted with a single set of features. And 1200 features were obtained.
Thanks for your reply.
\nChosing negative 1e-5 by experiments is really a fantastic idea! Usually people drop features with negative G-metric.
\nMay I ask when training using 1500 features, is it necessary to train 5 models if I use 5 folds? In that case, it would be very time-consuming.
\nAlso do you permutate feature on the oof data? It seems to me permutation importance on the oof would be better, cause it represents the ability a feature can generalize on unseen data or not.
I used 3 folds, and learning_rate=0.1.
\nThis still took a huge amount of time for feature selection.
\nIf I had used RAPIDs FIL, it would have been faster. But I didn't know.
\n(Chris introduces it in his solution. https://www.kaggle.com/competitions/amex-default-prediction/discussion/347641)
Yes, oof data.
", "rawMarkdown": "I used 3 folds, and learning_rate=0.1.\nThis still took a huge amount of time for feature selection.\nIf I had used RAPIDs FIL, it would have been faster. But I didn't know.\n(Chris introduces it in his solution. https://www.kaggle.com/competitions/amex-default-prediction/discussion/347641)\n\nYes, oof data.", "votes": 2 } ] }, { "id": 1917636, "postDate": "2022-08-28T23:57:33.703Z", "content": "Hello @aquatic, impressive model architecture; thanks for sharing
", "rawMarkdown": "Hello @aquatic, impressive model architecture; thanks for sharing", "votes": 1 }, { "id": 1917630, "postDate": "2022-08-28T23:37:00.770Z", "content": "Congratulations Joe and team. I'm sorry to see you drop on private. I was excited to see your huge climb on public LB in the last days.
\nCan you explain \"residual learning\" more? Is this training a model on the error of another model?
", "rawMarkdown": "Congratulations Joe and team. I'm sorry to see you drop on private. I was excited to see your huge climb on public LB in the last days.\n\nCan you explain \"residual learning\" more? Is this training a model on the error of another model?", "votes": 1, "replies": [ { "id": 1917645, "postDate": "2022-08-29T00:18:24.850Z", "content": "Thanks for the kind words Chris!
\n@andrew60909 brought this method to the team and referenced the winning Mercari solution. That's exactly it, we took OOF predictions, trained a regression LGBM model on some of our feature sets to predict resid = y_train - oof_pred_train, and then generated the test predictions as pred_test + pred_resid_test (and new OOF as oof_pred_val + pred_resid_val). This improved our TabNet and Transformer models by .001-.002 while still yielding a diverse final model. We did find it to be a little finicky - it overfits quickly so needs to be well regularized, and I think only really good features help.
Would love to hear if Angus has more thoughts, but to me it's almost like a mini post-processor, or a hybrid gradient boosting model where you're boosting off of a strong prediction initial prediction instead of the usual best constant prediction starting point.
", "rawMarkdown": "Thanks for the kind words Chris!\n\n@andrew60909 brought this method to the team and referenced the [winning Mercari solution](https://www.kaggle.com/c/mercari-price-suggestion-challenge/discussion/50256). That's exactly it, we took OOF predictions, trained a regression LGBM model on some of our feature sets to predict `resid = y_train - oof_pred_train`, and then generated the test predictions as `pred_test + pred_resid_test` (and new OOF as `oof_pred_val + pred_resid_val`). This improved our TabNet and Transformer models by .001-.002 while still yielding a diverse final model. We did find it to be a little finicky - it overfits quickly so needs to be well regularized, and I think only really good features help. \n\nWould love to hear if Angus has more thoughts, but to me it's almost like a mini post-processor, or a hybrid gradient boosting model where you're boosting off of a strong prediction initial prediction instead of the usual best constant prediction starting point. ", "votes": 6 }, { "id": 1917725, "postDate": "2022-08-29T02:47:59.393Z", "content": "Just as Joe said - it's actually more like post-processing, but it can generate pretty good diversity and boost the score of the relatively weak learner. We can also consider it as a kind of non-linear stacking. As for final ensembling, having these 2 models simultaneously will be a good idea too.
", "rawMarkdown": "Just as Joe said - it's actually more like post-processing, but it can generate pretty good diversity and boost the score of the relatively weak learner. We can also consider it as a kind of non-linear stacking. As for final ensembling, having these 2 models simultaneously will be a good idea too.", "votes": 4 }, { "id": 1918493, "postDate": "2022-08-29T16:02:23.080Z", "content": "If I understand correctly, you can also look at it as what LGBM (or XGB) is doing every round anyways. Each tree is predicting to eliminate the residual up to that point.
\nJoe, Angus, do you know if this has been tried much at all on the output of the entire final ensemble? Can it be effective as the final step?
", "rawMarkdown": "If I understand correctly, you can also look at it as what LGBM (or XGB) is doing every round anyways. Each tree is predicting to eliminate the residual up to that point. \n\nJoe, Angus, do you know if this has been tried much at all on the output of the entire final ensemble? Can it be effective as the final step?", "votes": 2 }, { "id": 1918723, "postDate": "2022-08-29T19:26:23.230Z", "content": "Haven't tried this, but I wouldn't expect it to help an ensemble. It should work if there is a systematic way to use features and a different model/view to explain errors made by a specific model/view, but I would think everything systematic is already in an ensemble that includes the GBT methods as base models. Also the error margins that the method can work with just become slimmer and slimmer as the model its applied to improves. I think part of why we saw it work for Transformer and TabNet but not MLP is that the data view for the former two is further removed from the flat aggregated data + GBT view.
", "rawMarkdown": "Haven't tried this, but I wouldn't expect it to help an ensemble. It should work if there is a systematic way to use features and a different model/view to explain errors made by a specific model/view, but I would think everything systematic is already in an ensemble that includes the GBT methods as base models. Also the error margins that the method can work with just become slimmer and slimmer as the model its applied to improves. I think part of why we saw it work for Transformer and TabNet but not MLP is that the data view for the former two is further removed from the flat aggregated data + GBT view.", "votes": 2 } ] }, { "id": 1920878, "postDate": "2022-08-31T13:04:42.617Z", "content": "Hi @aquatic, May I invite you to participate in this survey regarding your experience on Kaggle (10 min)? This is not a scam. We are a group of researchers at the City University of Hong Kong. The survey link is: https://cityuhk.questionpro.com/survey-of-kaggle-contestants
", "rawMarkdown": "Hi @aquatic, May I invite you to participate in this survey regarding your experience on Kaggle (10 min)? This is not a scam. We are a group of researchers at the City University of Hong Kong. The survey link is: https://cityuhk.questionpro.com/survey-of-kaggle-contestants", "votes": -3 }, { "id": 1917646, "postDate": "2022-08-29T00:21:06.407Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ], "comments": [ { "id": 1921355, "author_name": "asjad2024", "author_url": "", "post_date": "2022-08-31T18:59:21.167000", "content": "thanks! its very helpful for beginner how to approach problem
", "votes": 1, "replies": [] }, { "id": 1920201, "author_name": "Oscar Aguilar", "author_url": "", "post_date": "2022-08-31T01:52:54.433000", "content": "Thanks for sharing your solution @aquatic! Very insightful and it shows the amount of work that you put in!
", "votes": 1, "replies": [] }, { "id": 1918478, "author_name": "Robert Hatch", "author_url": "", "post_date": "2022-08-29T15:50:58.550000", "content": "Wow, you did a lot of great work!
\nI know you varied the data for your models, but was it the same 1.2k base after feature selection for all models using the \"normal\" set of data? I wonder if \"less aggressive feature selection\" is another possible thing that, in retrospect, could've helped the private lb score?
", "votes": 1, "replies": [ { "id": 1918500, "author_name": "Robert Hatch", "author_url": "", "post_date": "2022-08-29T16:10:03.987000", "content": "Note I'm asking because I see so few differences that seem critical (that I noticed) between your solution and top 2 solutions.
\nHmm, maybe top two had more emphasis on a normalization factor? You had monthly based ranking, but top solution had ranking of a single customer's statements across months AND ranking across customers for a given month, AND aggregated those things. (Also could feature selection have removed some of the monthly ranking or any other normalization that you did do?)
", "votes": 1, "replies": [] }, { "id": 1918718, "author_name": "Joe Eddy", "author_url": "", "post_date": "2022-08-29T19:17:40.697000", "content": "Thanks!
\nThat method was specifically what @ryotak12 used for selection (especially for some of the feature sets that would otherwise just be enormous). For my models, I actually didn't use selection, more like feed-forward additions (and actually included different/older versions of models that had a subset of the total feature set I used, these made the cut in the ensemble). Feature selection is hard, especially on this problem, and I thought it'd be more time efficient to prioritize diversity for my work.
\nIf I'm remembering correctly, top 2 solutions both didn't use stacking so that may be one differentiating point. Perhaps our stacking overfits to CV / Public LB relative to the highest scoring Private solutions.
", "votes": 2, "replies": [] }, { "id": 1919011, "author_name": "Ryota", "author_url": "", "post_date": "2022-08-30T04:09:45.923000", "content": "As Joe said, only my feature set was applied to this selection method.
\nI generated a large number of features at beginning and could not train them all 20k over features.
\nSo, I had to select features due to training time.
Congratulations Joe!
\nCould you please share more about feature selection part? After you get the permutation importance of 20k features, how do you select 1200 features? Did you just pick the top 1200 features? If then, how do you decide how many features? Thanks
I will answer your question.
\nI could not train 20k features at once, because it takes too long time and memory.
\nSo, at first, I divided it into 15 subsets and experimented with 1500 features each.
\nAt this stage, I selected features that decrease by more than 1e-5 in G-metric.
\nThe reason that this threshold was chosen is that experiments had shown that features that do not contribute to improving metric can also results -1e-5.
\nThese are repeated and finally the experiment is conducted with a single set of features. And 1200 features were obtained.
Thanks for your reply.
\nChosing negative 1e-5 by experiments is really a fantastic idea! Usually people drop features with negative G-metric.
\nMay I ask when training using 1500 features, is it necessary to train 5 models if I use 5 folds? In that case, it would be very time-consuming.
\nAlso do you permutate feature on the oof data? It seems to me permutation importance on the oof would be better, cause it represents the ability a feature can generalize on unseen data or not.
I used 3 folds, and learning_rate=0.1.
\nThis still took a huge amount of time for feature selection.
\nIf I had used RAPIDs FIL, it would have been faster. But I didn't know.
\n(Chris introduces it in his solution. https://www.kaggle.com/competitions/amex-default-prediction/discussion/347641)
Yes, oof data.
", "votes": 2, "replies": [] } ] }, { "id": 1917636, "author_name": "C4rl05/V", "author_url": "", "post_date": "2022-08-28T23:57:33.703000", "content": "Hello @aquatic, impressive model architecture; thanks for sharing
", "votes": 1, "replies": [] }, { "id": 1917630, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-08-28T23:37:00.770000", "content": "Congratulations Joe and team. I'm sorry to see you drop on private. I was excited to see your huge climb on public LB in the last days.
\nCan you explain \"residual learning\" more? Is this training a model on the error of another model?
", "votes": 1, "replies": [ { "id": 1917645, "author_name": "Joe Eddy", "author_url": "", "post_date": "2022-08-29T00:18:24.850000", "content": "Thanks for the kind words Chris!
\n@andrew60909 brought this method to the team and referenced the winning Mercari solution. That's exactly it, we took OOF predictions, trained a regression LGBM model on some of our feature sets to predict resid = y_train - oof_pred_train, and then generated the test predictions as pred_test + pred_resid_test (and new OOF as oof_pred_val + pred_resid_val). This improved our TabNet and Transformer models by .001-.002 while still yielding a diverse final model. We did find it to be a little finicky - it overfits quickly so needs to be well regularized, and I think only really good features help.
Would love to hear if Angus has more thoughts, but to me it's almost like a mini post-processor, or a hybrid gradient boosting model where you're boosting off of a strong prediction initial prediction instead of the usual best constant prediction starting point.
", "votes": 6, "replies": [] }, { "id": 1917725, "author_name": "Angus Chang", "author_url": "", "post_date": "2022-08-29T02:47:59.393000", "content": "Just as Joe said - it's actually more like post-processing, but it can generate pretty good diversity and boost the score of the relatively weak learner. We can also consider it as a kind of non-linear stacking. As for final ensembling, having these 2 models simultaneously will be a good idea too.
", "votes": 4, "replies": [] }, { "id": 1918493, "author_name": "Robert Hatch", "author_url": "", "post_date": "2022-08-29T16:02:23.080000", "content": "If I understand correctly, you can also look at it as what LGBM (or XGB) is doing every round anyways. Each tree is predicting to eliminate the residual up to that point.
\nJoe, Angus, do you know if this has been tried much at all on the output of the entire final ensemble? Can it be effective as the final step?
", "votes": 2, "replies": [] }, { "id": 1918723, "author_name": "Joe Eddy", "author_url": "", "post_date": "2022-08-29T19:26:23.230000", "content": "Haven't tried this, but I wouldn't expect it to help an ensemble. It should work if there is a systematic way to use features and a different model/view to explain errors made by a specific model/view, but I would think everything systematic is already in an ensemble that includes the GBT methods as base models. Also the error margins that the method can work with just become slimmer and slimmer as the model its applied to improves. I think part of why we saw it work for Transformer and TabNet but not MLP is that the data view for the former two is further removed from the flat aggregated data + GBT view.
", "votes": 2, "replies": [] } ] }, { "id": 1920878, "author_name": "Yang Liu", "author_url": "", "post_date": "2022-08-31T13:04:42.617000", "content": "Hi @aquatic, May I invite you to participate in this survey regarding your experience on Kaggle (10 min)? This is not a scam. We are a group of researchers at the City University of Hong Kong. The survey link is: https://cityuhk.questionpro.com/survey-of-kaggle-contestants
", "votes": -3, "replies": [] }, { "id": 1917646, "author_name": "", "author_url": "", "post_date": "2022-08-29T00:21:06.407000", "content": "", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "1917580": "A huge huge thank you to @andrew60909 and @ryotak12 for an excellent team experience. We learned a lot and had fun, and I appreciate their patience with my extended, largely unsuccessful efforts to make a transformer work really well 😂. Thanks also to Amex and Kaggle for hosting this interesting competition and to @raddar and everyone else who shared resources that made this competition much more accessible. \n\nI've made my personal code public here: https://github.com/JEddy92/amex_default_kaggle. I tried to create a reasonably general structure for division of FE and modeling, with reusable functionality for training and logging results across a variety of models. There is still some clean up to do and it's not as polished as would be ideal!\n\nFeel free to ask us anything if you have questions about our approach!\n\n## Solution Summary\n\nWe used logistic regression to ensemble ~60 sets of predictions with diversity obtained from different models, feature sets, and data views. Models included the usual mix of gradient boosters (LGBM, LGBM dart, XGBoost, CatBoost), MLPs, TabNet, and Transformers with a few special tricks. Data views included aggregated data, fully flattened data (13 raw features per customer), sequential data (Transformer), and augmented data (shifted statements forward by 1 and appended to original). Very detailed feature information to follow -- they are split across our 3 different team members' different models, but this aims to be a complete compilation of what we all used. \n\n\n