Not so secretly a brag post, sorry. My best single model would've scored 49th overall. Oh right, it was my only model, and it did! 0.80798 private, 0.79889 pub. I'll post a little bit of details in the comments.
\nMost medal winners did great ensembles, what were your best single model scores?
", "messages": [ { "id": "1914188", "postDate": "08/25/2022 20:07:40", "content": "Not so secretly a brag post, sorry. My best single model would've scored 49th overall. Oh right, it was my only model, and it did! 0.80798 private, 0.79889 pub. I'll post a little bit of details in the comments.
\nMost medal winners did great ensembles, what were your best single model scores?
", "rawMarkdown": "Not so secretly a brag post, sorry. My best single model would've scored 49th overall. Oh right, it was my only model, and it did! 0.80798 private, 0.79889 pub. I'll post a little bit of details in the comments. \n\nMost medal winners did great ensembles, what were your best single model scores?", "votes": null }, { "id": "1914194", "postDate": "08/25/2022 20:19:22", "content": "That's a great single model Robert. Below are my best two single models. Each is 10 folds and the average of a few seeds:
\nNN Transformer:
\nCV 0.7980, Public LB 0.7988, Private LB 0.8076
LGBM Dart:
\nCV 0.7990, Public LB 0.7992, Private LB 0.8070
Ensemble of above two models:
\nCV 0.8000, Public LB 0.8010, Private LB 0.80840
Model: XGB pyramid with 3098 features. 49th place.
\nMy 'single' model had a couple steps:
\nMy original plan was to finish the big days overdue prediction 1-2 weeks ago, get a .800, then approach a few top scorers about teaming up with evidence I could contribute. Instead I finished it at the 11th hour AND it didn't do that great on public. Shrug.
\nI'll take good luck on private, bad luck on public any day, though! Either a great model or just a great seed, lol.
", "rawMarkdown": "My original plan was to finish the big days overdue prediction 1-2 weeks ago, get a .800, then approach a few top scorers about teaming up with evidence I could contribute. Instead I finished it at the 11th hour AND it didn't do that great on public. Shrug. \n\nI'll take good luck on private, bad luck on public any day, though! Either a great model or just a great seed, lol.", "votes": null }, { "id": "1914205", "postDate": "08/25/2022 20:32:51", "content": "My best single model:
\nLGBM DART with ~2000 features,
\nPrivate Score: 0.80759, Public Score: 0.79966
In step 2 do you use the predictions from step 1? That's a cool idea.
", "rawMarkdown": "In step 2 do you use the predictions from step 1? That's a cool idea.", "votes": null }, { "id": "1914218", "postDate": "08/25/2022 21:00:17", "content": "Yes, I think it happened to be extra effective on the different population issues of the private leaderboard. The step one was given S_2_month (0-31) so had the ability if the model wanted to distinguish between the different datasets if it was helpful in predicting missed payments.
\nI also predicted 0 balance next month (well, 0 days overdue D_39) independently as another step 1 model. But I am doubtful whether that mattered compared with the key prediction of going a month with no payment. And the model wasn't very good at predicting that one! Maybe if I had time to tune it would've gotten a little better, shrug.
\nThose two prediction (do_1, do_2) were then fed into the aggregate features. 2*16 they became my last 32 features.
\n(Last, min, max, mean, std, first, lag 1, last-mean, hull moving avg, 5 different exponential avgs, max-mean, last-min/max-min)
\nI would've liked to try combos of megafeatures: p2+do_1, p2-do_1, maybe a few others. But ran out of time and GPU both :)
", "rawMarkdown": "Yes, I think it happened to be extra effective on the different population issues of the private leaderboard. The step one was given S_2_month (0-31) so had the ability if the model wanted to distinguish between the different datasets if it was helpful in predicting missed payments. \n\nI also predicted 0 balance next month (well, 0 days overdue D_39) independently as another step 1 model. But I am doubtful whether that mattered compared with the key prediction of going a month with no payment. And the model wasn't very good at predicting that one! Maybe if I had time to tune it would've gotten a little better, shrug. \n\nThose two prediction (do_1, do_2) were then fed into the aggregate features. 2*16 they became my last 32 features. \n\n(Last, min, max, mean, std, first, lag 1, last-mean, hull moving avg, 5 different exponential avgs, max-mean, last-min/max-min)\n\nI would've liked to try combos of megafeatures: p2+do_1, p2-do_1, maybe a few others. But ran out of time and GPU both :)", "votes": null }, { "id": "1914222", "postDate": "08/25/2022 21:04:09", "content": "Nice DART score! Silver medal with that alone :)
", "rawMarkdown": "Nice DART score! Silver medal with that alone :)", "votes": null }, { "id": "1914227", "postDate": "08/25/2022 21:14:05", "content": "I like this concept of meta features. I need to try this in my next comp. I guess the basic idea is to use the data to predict missing data or a variation of the given targets. Teams in 11th place here and 12th place here also predicted additional features for their models.
", "rawMarkdown": "I like this concept of meta features. I need to try this in my next comp. I guess the basic idea is to use the data to predict missing data or a variation of the given targets. Teams in 11th place [here][1] and 12th place [here][2] also predicted additional features for their models.\n\n[1]: https://www.kaggle.com/competitions/amex-default-prediction/discussion/347668\n[2]: https://www.kaggle.com/competitions/amex-default-prediction/discussion/347786", "votes": null }, { "id": "1914236", "postDate": "08/25/2022 21:27:55", "content": "My best single model:
\nLGBM DART
\nPrivate Score: 0.80711, Public Score: 0.79945
My best ensemble (not selected as final unfortunately):
\n39 base models (mostly LGBMs, some CNNs, Transformers and even a Naive Bayes )
\n3 second level models (MLPs) averaged
\nPrivate Score: 0.80849, Public Score: 0.80060
There were a bunch of opportunities on this competition, for sure! I think more focus on that next competition will be a priority for me. At least if it shares some of the traits of this competition with lots of data, and really a ton of unlabeled data.
\nAnything that helped create a feature based on the \"shape\" of the 13 statements seems really big. I wanted a KNN per column, for instance.
\nThe high level concept is the idea of ensemble diversity, but applied to single base features. \"last\" is one way to predict the relevant value of the 13 data points. \"mean\" is another. Ema (exponential average) is another. But meta features via model predictions like one team's LSTM(?) is yet another diverse way of trying to extract signal.
\nI think doing it multiple diverse ways could get a top 10 score. Just a hunch.
", "rawMarkdown": "There were a bunch of opportunities on this competition, for sure! I think more focus on that next competition will be a priority for me. At least if it shares some of the traits of this competition with lots of data, and really a ton of unlabeled data. \n\nAnything that helped create a feature based on the \"shape\" of the 13 statements seems really big. I wanted a KNN per column, for instance. \n\nThe high level concept is the idea of ensemble diversity, but applied to single base features. \"last\" is one way to predict the relevant value of the 13 data points. \"mean\" is another. Ema (exponential average) is another. But meta features via model predictions like one team's LSTM(?) is yet another diverse way of trying to extract signal. \n\nI think doing it multiple diverse ways could get a top 10 score. Just a hunch.", "votes": null }, { "id": "1914274", "postDate": "08/25/2022 23:34:55", "content": "single Lgb private 8089
", "rawMarkdown": "single Lgb private 8089", "votes": null }, { "id": "1914277", "postDate": "08/25/2022 23:43:44", "content": "Excellent conversations are going on; I need to learn from the best solutions.
", "rawMarkdown": "Excellent conversations are going on; I need to learn from the best solutions.", "votes": null }, { "id": "1914286", "postDate": "08/26/2022 00:24:03", "content": "LGBM DART with ~3800 features
\nPrivate : 0.80796
\nPublic : 0.80010
\nI augmentated training data by deleting the first n of each customer's sequence and adding them as new training data.
This single model had a higher private score than the ensemble model I chose last.
", "rawMarkdown": "LGBM DART with ~3800 features\nPrivate : 0.80796\nPublic : 0.80010\nI augmentated training data by deleting the first n of each customer's sequence and adding them as new training data.\n\nThis single model had a higher private score than the ensemble model I chose last.", "votes": null }, { "id": "1914311", "postDate": "08/26/2022 01:36:05", "content": "That's a great idea on training augmentation!
", "rawMarkdown": "That's a great idea on training augmentation!", "votes": null }, { "id": "1914440", "postDate": "08/26/2022 05:25:22", "content": "LGBM DART, private 0.80720
", "rawMarkdown": "LGBM DART, private 0.80720", "votes": null }, { "id": "1914467", "postDate": "08/26/2022 05:41:27", "content": "Wow @mahluo that's amazing. I'm looking forward to reading your writeup and learning about your model.
", "rawMarkdown": "Wow @mahluo that's amazing. I'm looking forward to reading your writeup and learning about your model.", "votes": null }, { "id": "1914522", "postDate": "08/26/2022 06:48:03", "content": "Echo @cdeotte , great work. I was rooting for you before the big reveal, 802 was really impressive.
", "rawMarkdown": "Echo @cdeotte , great work. I was rooting for you before the big reveal, 802 was really impressive.", "votes": null }, { "id": "1914708", "postDate": "08/26/2022 10:29:44", "content": "So your final model is worse than that single model o:?
", "rawMarkdown": "So your final model is worse than that single model o:?", "votes": null }, { "id": "1914868", "postDate": "08/26/2022 13:46:56", "content": "Yeap! :)
\nI made a lot of mistakes that I learned lessons from. Hopefully I can use them for the next tabular competitions.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4392121%2F1c2831bfe9d90a9930cad8157d717a81%2FBestSingleModel.png?generation=1661521464020931&alt=media\")
\nI am attaching my best single model predictions just in case someone wants to blend it with another model (NN, CatBoost, etc.)
\nhttps://www.kaggle.com/datasets/rasoulmojtahedzadeh/amex-default-prediction-best-single-model
I have a question here: since I started studying ML, I have always been told to avoid using too many features, so as not to increase the chances of overfitting. In other competitions, I have already noticed that after a certain amount of features, the performance of the model drops. Here Iβm seeing over 3k features.
\nHow do I know what the ideal limit is? The more features, the more complex the model needs to be?
", "rawMarkdown": "I have a question here: since I started studying ML, I have always been told to avoid using too many features, so as not to increase the chances of overfitting. In other competitions, I have already noticed that after a certain amount of features, the performance of the model drops. Here Iβm seeing over 3k features.\n\nHow do I know what the ideal limit is? The more features, the more complex the model needs to be?", "votes": null }, { "id": "1915232", "postDate": "08/26/2022 19:26:16", "content": "Not sure how to answer that. Tree boosting models, at least, should be robust even if given extra features.
\nFor this competition, my belief is that teaching the model the 13 statement \"shape\" of each base feature is the first key to the competition. By far the easiest way to teach the model is with a diverse set of features built from the (up to) 13 statements. So you have lots of features with minimal individual contribution, but help lead to smoother more accurate predictions, helping 'rank' the customers. 190 times X quickly becomes a large number!
", "rawMarkdown": "Not sure how to answer that. Tree boosting models, at least, should be robust even if given extra features.\n\nFor this competition, my belief is that teaching the model the 13 statement \"shape\" of each base feature is the first key to the competition. By far the easiest way to teach the model is with a diverse set of features built from the (up to) 13 statements. So you have lots of features with minimal individual contribution, but help lead to smoother more accurate predictions, helping 'rank' the customers. 190 times X quickly becomes a large number!", "votes": null }, { "id": "1915253", "postDate": "08/26/2022 19:50:03", "content": "Makes sense, @roberthatch . Thanks for your answer, and congratz!
", "rawMarkdown": "Makes sense, @roberthatch . Thanks for your answer, and congratz!", "votes": null }, { "id": "1915376", "postDate": "08/26/2022 23:32:59", "content": "@chiakiichimura Great idea. How did you choose N? Did you pick one N and then apply it to all customers? Or did you pick a random N for each customer?
@roberthatch what is your single model's CV and LBs?
", "rawMarkdown": "roberthatch what is your single model's CV and LBs?", "votes": null }, { "id": "1915414", "postDate": "08/27/2022 00:41:20", "content": "0.80798 private, 0.79889 pub. 46th overall. I expect a few winning teams might've had multiple higher scoring single models, but so far in this thread only one single model fared better (far better π)
\nNo CV! π I got the meta feature model finished the day before, did a CV and leaderboard test with higher training rateβ¦ And then intentionally trained on all training data. GPU hours just reset, maybe I'll get that CV score. From similar models, should be low 797xx. If much higher than that, it would be very encouraging, actually. It would imply that public LB score was the unexpected one, and it wasn't JUST that my model happened to have some kind of private LB secret sauce.
", "rawMarkdown": "0.80798 private, 0.79889 pub. 46th overall. I expect a few winning teams might've had multiple higher scoring single models, but so far in this thread only one single model fared better (far better π)\n\nNo CV! π I got the meta feature model finished the day before, did a CV and leaderboard test with higher training rate... And then intentionally trained on all training data. GPU hours just reset, maybe I'll get that CV score. From similar models, should be low 797xx. If much higher than that, it would be very encouraging, actually. It would imply that public LB score was the unexpected one, and it wasn't JUST that my model happened to have some kind of private LB secret sauce.", "votes": null }, { "id": "1915517", "postDate": "08/27/2022 03:23:17", "content": "LGBM Dart:
\nCV 0.7976, Public LB 0.79924, Private LB 0.80690
@cdeotte
\nThank you for your question.
\n\nHow did you choose N?
\n
I tried several patterns (N=4~11) and chose the one (N=8) with the highest local CV and public LB rank.
\n\nDid you pick one N and then apply it to all customers? Or did you pick a random N for each customer?
\n
I picked one N and then applied it to customers. If I applied to all customers it would cause out of memory, so I sampled customers randomly and applied to those.
", "rawMarkdown": "cdeotte \nThank you for your question.\n\n> How did you choose N?\n\nI tried several patterns (`N`=4~11) and chose the one (`N`=8) with the highest local CV and public LB rank.\n\n> Did you pick one N and then apply it to all customers? Or did you pick a random N for each customer?\n\nI picked one N and then applied it to customers. If I applied to all customers it would cause out of memory, so I sampled customers randomly and applied to those.", "votes": null } ], "comments": [ { "id": 1914194, "author_name": "cdeotte", "author_url": "", "post_date": "08/25/2022 20:19:22", "content": "That's a great single model Robert. Below are my best two single models. Each is 10 folds and the average of a few seeds:
\nNN Transformer:
\nCV 0.7980, Public LB 0.7988, Private LB 0.8076
LGBM Dart:
\nCV 0.7990, Public LB 0.7992, Private LB 0.8070
Ensemble of above two models:
\nCV 0.8000, Public LB 0.8010, Private LB 0.80840
My original plan was to finish the big days overdue prediction 1-2 weeks ago, get a .800, then approach a few top scorers about teaming up with evidence I could contribute. Instead I finished it at the 11th hour AND it didn't do that great on public. Shrug.
\nI'll take good luck on private, bad luck on public any day, though! Either a great model or just a great seed, lol.
", "votes": null, "replies": [] } ] }, { "id": 1914195, "author_name": "roberthatch", "author_url": "", "post_date": "08/25/2022 20:19:43", "content": "Model: XGB pyramid with 3098 features. 49th place.
\nMy 'single' model had a couple steps:
\nIn step 2 do you use the predictions from step 1? That's a cool idea.
", "votes": null, "replies": [] }, { "id": 1914218, "author_name": "roberthatch", "author_url": "", "post_date": "08/25/2022 21:00:17", "content": "Yes, I think it happened to be extra effective on the different population issues of the private leaderboard. The step one was given S_2_month (0-31) so had the ability if the model wanted to distinguish between the different datasets if it was helpful in predicting missed payments.
\nI also predicted 0 balance next month (well, 0 days overdue D_39) independently as another step 1 model. But I am doubtful whether that mattered compared with the key prediction of going a month with no payment. And the model wasn't very good at predicting that one! Maybe if I had time to tune it would've gotten a little better, shrug.
\nThose two prediction (do_1, do_2) were then fed into the aggregate features. 2*16 they became my last 32 features.
\n(Last, min, max, mean, std, first, lag 1, last-mean, hull moving avg, 5 different exponential avgs, max-mean, last-min/max-min)
\nI would've liked to try combos of megafeatures: p2+do_1, p2-do_1, maybe a few others. But ran out of time and GPU both :)
", "votes": null, "replies": [] }, { "id": 1914227, "author_name": "cdeotte", "author_url": "", "post_date": "08/25/2022 21:14:05", "content": "I like this concept of meta features. I need to try this in my next comp. I guess the basic idea is to use the data to predict missing data or a variation of the given targets. Teams in 11th place here and 12th place here also predicted additional features for their models.
", "votes": null, "replies": [] }, { "id": 1914255, "author_name": "roberthatch", "author_url": "", "post_date": "08/25/2022 22:27:40", "content": "There were a bunch of opportunities on this competition, for sure! I think more focus on that next competition will be a priority for me. At least if it shares some of the traits of this competition with lots of data, and really a ton of unlabeled data.
\nAnything that helped create a feature based on the \"shape\" of the 13 statements seems really big. I wanted a KNN per column, for instance.
\nThe high level concept is the idea of ensemble diversity, but applied to single base features. \"last\" is one way to predict the relevant value of the 13 data points. \"mean\" is another. Ema (exponential average) is another. But meta features via model predictions like one team's LSTM(?) is yet another diverse way of trying to extract signal.
\nI think doing it multiple diverse ways could get a top 10 score. Just a hunch.
", "votes": null, "replies": [] }, { "id": 1915377, "author_name": "cdeotte", "author_url": "", "post_date": "08/26/2022 23:34:43", "content": "@roberthatch what is your single model's CV and LBs?
", "votes": null, "replies": [] }, { "id": 1915414, "author_name": "roberthatch", "author_url": "", "post_date": "08/27/2022 00:41:20", "content": "0.80798 private, 0.79889 pub. 46th overall. I expect a few winning teams might've had multiple higher scoring single models, but so far in this thread only one single model fared better (far better π)
\nNo CV! π I got the meta feature model finished the day before, did a CV and leaderboard test with higher training rateβ¦ And then intentionally trained on all training data. GPU hours just reset, maybe I'll get that CV score. From similar models, should be low 797xx. If much higher than that, it would be very encouraging, actually. It would imply that public LB score was the unexpected one, and it wasn't JUST that my model happened to have some kind of private LB secret sauce.
", "votes": null, "replies": [] } ] }, { "id": 1914205, "author_name": "rasoulmojtahedzadeh", "author_url": "", "post_date": "08/25/2022 20:32:51", "content": "My best single model:
\nLGBM DART with ~2000 features,
\nPrivate Score: 0.80759, Public Score: 0.79966
Nice DART score! Silver medal with that alone :)
", "votes": null, "replies": [] }, { "id": 1914868, "author_name": "rasoulmojtahedzadeh", "author_url": "", "post_date": "08/26/2022 13:46:56", "content": "Yeap! :)
\nI made a lot of mistakes that I learned lessons from. Hopefully I can use them for the next tabular competitions.
\n
\nI am attaching my best single model predictions just in case someone wants to blend it with another model (NN, CatBoost, etc.)
\nhttps://www.kaggle.com/datasets/rasoulmojtahedzadeh/amex-default-prediction-best-single-model
My best single model:
\nLGBM DART
\nPrivate Score: 0.80711, Public Score: 0.79945
My best ensemble (not selected as final unfortunately):
\n39 base models (mostly LGBMs, some CNNs, Transformers and even a Naive Bayes )
\n3 second level models (MLPs) averaged
\nPrivate Score: 0.80849, Public Score: 0.80060
single Lgb private 8089
", "votes": null, "replies": [ { "id": 1914467, "author_name": "cdeotte", "author_url": "", "post_date": "08/26/2022 05:41:27", "content": "Wow @mahluo that's amazing. I'm looking forward to reading your writeup and learning about your model.
", "votes": null, "replies": [] }, { "id": 1914522, "author_name": "roberthatch", "author_url": "", "post_date": "08/26/2022 06:48:03", "content": "Echo @cdeotte , great work. I was rooting for you before the big reveal, 802 was really impressive.
", "votes": null, "replies": [] }, { "id": 1914847, "author_name": "fritzcremer", "author_url": "", "post_date": "08/26/2022 13:26:21", "content": "So your final model is worse than that single model o:?
", "votes": null, "replies": [] } ] }, { "id": 1914277, "author_name": "cv13j0", "author_url": "", "post_date": "08/25/2022 23:43:44", "content": "Excellent conversations are going on; I need to learn from the best solutions.
", "votes": null, "replies": [] }, { "id": 1914286, "author_name": "chiakiichimura", "author_url": "", "post_date": "08/26/2022 00:24:03", "content": "LGBM DART with ~3800 features
\nPrivate : 0.80796
\nPublic : 0.80010
\nI augmentated training data by deleting the first n of each customer's sequence and adding them as new training data.
This single model had a higher private score than the ensemble model I chose last.
", "votes": null, "replies": [ { "id": 1914311, "author_name": "roberthatch", "author_url": "", "post_date": "08/26/2022 01:36:05", "content": "That's a great idea on training augmentation!
", "votes": null, "replies": [] }, { "id": 1915223, "author_name": "leandrodestefani", "author_url": "", "post_date": "08/26/2022 19:11:13", "content": "I have a question here: since I started studying ML, I have always been told to avoid using too many features, so as not to increase the chances of overfitting. In other competitions, I have already noticed that after a certain amount of features, the performance of the model drops. Here Iβm seeing over 3k features.
\nHow do I know what the ideal limit is? The more features, the more complex the model needs to be?
", "votes": null, "replies": [] }, { "id": 1915232, "author_name": "roberthatch", "author_url": "", "post_date": "08/26/2022 19:26:16", "content": "Not sure how to answer that. Tree boosting models, at least, should be robust even if given extra features.
\nFor this competition, my belief is that teaching the model the 13 statement \"shape\" of each base feature is the first key to the competition. By far the easiest way to teach the model is with a diverse set of features built from the (up to) 13 statements. So you have lots of features with minimal individual contribution, but help lead to smoother more accurate predictions, helping 'rank' the customers. 190 times X quickly becomes a large number!
", "votes": null, "replies": [] }, { "id": 1915253, "author_name": "leandrodestefani", "author_url": "", "post_date": "08/26/2022 19:50:03", "content": "Makes sense, @roberthatch . Thanks for your answer, and congratz!
", "votes": null, "replies": [] }, { "id": 1915376, "author_name": "cdeotte", "author_url": "", "post_date": "08/26/2022 23:32:59", "content": "@chiakiichimura Great idea. How did you choose N? Did you pick one N and then apply it to all customers? Or did you pick a random N for each customer?
@cdeotte
\nThank you for your question.
\n\nHow did you choose N?
\n
I tried several patterns (N=4~11) and chose the one (N=8) with the highest local CV and public LB rank.
\n\nDid you pick one N and then apply it to all customers? Or did you pick a random N for each customer?
\n
I picked one N and then applied it to customers. If I applied to all customers it would cause out of memory, so I sampled customers randomly and applied to those.
", "votes": null, "replies": [] } ] }, { "id": 1914440, "author_name": "mohammadrahmati", "author_url": "", "post_date": "08/26/2022 05:25:22", "content": "LGBM DART, private 0.80720
", "votes": null, "replies": [] }, { "id": 1914708, "author_name": "hideyukizushi", "author_url": "", "post_date": "08/26/2022 10:29:44", "content": "LGBM Dart:
\nCV 0.7976, Public LB 0.79924, Private LB 0.80690