First of all, congratulations to all winners and medalists!
\nI have been reflecting on what I should have done better after my team fell 40+ places on the private leaderboard. Confusingly, looking at the private scores of my team’s submissions, I haven’t found any of my strategies/approaches would have deterministically outperformed the others in the private leaderboard. Here’s my list of thoughts.
\nSingle Model vs Model Ensemble: Most of the top scoring submissions are ensembles, but there are some single models on the top scoring list as well. I will still opt for an ensemble next time though.Heavy Ensemble vs Light/Simple Ensemble: No obvious pattern found. Many light ensemble models (1-layer stacking with logistic regression on just a few models) outperform heavy ensembles and vice versa. Specific Ensemble Strategy: Nothing seems to always work. For each strategy, we have many similar ensembles. They can range from top 30 to maybe 200th with similar CV and public LB.CV vs LB vs ?: We have quite a few private 808 subs and none of them are best CV or best LB. I can easily find a similar model with a better CV and LB and scores a lot worse on the private LB. NN/GBDT Ensemble vs Pure-GBDT: Again no conclusive observation here. Including NN in the ensemble doesn’t seem to deterministically make performance better or worse in my case. Sadly no immediate takeaways from my own solutions ☹️☹️☹️.
\nThings I learned post-competition that worked well for others:
\nHope you can share your experience here regarding what consistently worked great for you or your takeaways.
", "messages": [ { "id": "1913446", "postDate": "08/25/2022 10:26:35", "content": "First of all, congratulations to all winners and medalists!
\nI have been reflecting on what I should have done better after my team fell 40+ places on the private leaderboard. Confusingly, looking at the private scores of my team’s submissions, I haven’t found any of my strategies/approaches would have deterministically outperformed the others in the private leaderboard. Here’s my list of thoughts.
\nSingle Model vs Model Ensemble: Most of the top scoring submissions are ensembles, but there are some single models on the top scoring list as well. I will still opt for an ensemble next time though.Heavy Ensemble vs Light/Simple Ensemble: No obvious pattern found. Many light ensemble models (1-layer stacking with logistic regression on just a few models) outperform heavy ensembles and vice versa. Specific Ensemble Strategy: Nothing seems to always work. For each strategy, we have many similar ensembles. They can range from top 30 to maybe 200th with similar CV and public LB.CV vs LB vs ?: We have quite a few private 808 subs and none of them are best CV or best LB. I can easily find a similar model with a better CV and LB and scores a lot worse on the private LB. NN/GBDT Ensemble vs Pure-GBDT: Again no conclusive observation here. Including NN in the ensemble doesn’t seem to deterministically make performance better or worse in my case. Sadly no immediate takeaways from my own solutions ☹️☹️☹️.
\nThings I learned post-competition that worked well for others:
\nHope you can share your experience here regarding what consistently worked great for you or your takeaways.
", "rawMarkdown": "First of all, congratulations to all winners and medalists! \n\nI have been reflecting on what I should have done better after my team fell 40+ places on the private leaderboard. Confusingly, looking at the private scores of my team’s submissions, I haven’t found any of my strategies/approaches would have deterministically outperformed the others in the private leaderboard. Here’s my list of thoughts. \n1. `Single Model` vs `Model Ensemble`: Most of the top scoring submissions are ensembles, but there are some single models on the top scoring list as well. I will still opt for an ensemble next time though.\n2. `Heavy Ensemble` vs `Light/Simple Ensemble`: No obvious pattern found. Many light ensemble models (1-layer stacking with logistic regression on just a few models) outperform heavy ensembles and vice versa. \n3. `Specific Ensemble Strategy`: Nothing seems to always work. For each strategy, we have many similar ensembles. They can range from top 30 to maybe 200th with similar CV and public LB.\n4. `CV` vs `LB` vs `?`: We have quite a few private 808 subs and none of them are best CV or best LB. I can easily find a similar model with a better CV and LB and scores a lot worse on the private LB. \n5. `NN/GBDT Ensemble` vs `Pure-GBDT`: Again no conclusive observation here. Including NN in the ensemble doesn’t seem to deterministically make performance better or worse in my case. \n\nSadly no immediate takeaways from my own solutions ☹️☹️☹️. \n \n \n \n**Things I learned post-competition that worked well for others:**\n1. [Knowledge Distillation](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347641) from @cdeotte \n2. Capturing time-series dynamics within each column \n[SequentialEncoder](https://www.kaggle.com/code/pavelvod/27-place-sequentialencoder?scriptVersionId=104154431) from @pavelvod and [Mini-LSTMs](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347651) from @fritzcremer\n3. [Ensemble of multiple TabNets](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347688) from @raddar \n\n**Hope you can share your experience here regarding what consistently worked great for you or your takeaways.**", "votes": null }, { "id": "1913718", "postDate": "08/25/2022 13:14:14", "content": "Many thanks for the post and the links to other resources too! This is a great learning experience for beginners like me!
", "rawMarkdown": "Many thanks for the post and the links to other resources too! This is a great learning experience for beginners like me!", "votes": null }, { "id": "1922195", "postDate": "09/01/2022 10:33:10", "content": "Hi @raphael1123. May I invite you to participate in this survey regarding your experience on Kaggle (10 min)? This is not a scam. We are a group of researchers at the City University of Hong Kong. The survey link is: https://cityuhk.questionpro.com/survey-of-kaggle-contestants
", "rawMarkdown": "Hi @raphael1123. May I invite you to participate in this survey regarding your experience on Kaggle (10 min)? This is not a scam. We are a group of researchers at the City University of Hong Kong. The survey link is: https://cityuhk.questionpro.com/survey-of-kaggle-contestants", "votes": null } ], "comments": [ { "id": 1913718, "author_name": "ravi20076", "author_url": "", "post_date": "08/25/2022 13:14:14", "content": "Many thanks for the post and the links to other resources too! This is a great learning experience for beginners like me!
", "votes": null, "replies": [] }, { "id": 1922195, "author_name": "lystriving", "author_url": "", "post_date": "09/01/2022 10:33:10", "content": "Hi @raphael1123. May I invite you to participate in this survey regarding your experience on Kaggle (10 min)? This is not a scam. We are a group of researchers at the City University of Hong Kong. The survey link is: https://cityuhk.questionpro.com/survey-of-kaggle-contestants
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1913446": "First of all, congratulations to all winners and medalists! \n\nI have been reflecting on what I should have done better after my team fell 40+ places on the private leaderboard. Confusingly, looking at the private scores of my team’s submissions, I haven’t found any of my strategies/approaches would have deterministically outperformed the others in the private leaderboard. Here’s my list of thoughts. \n1. `Single Model` vs `Model Ensemble`: Most of the top scoring submissions are ensembles, but there are some single models on the top scoring list as well. I will still opt for an ensemble next time though.\n2. `Heavy Ensemble` vs `Light/Simple Ensemble`: No obvious pattern found. Many light ensemble models (1-layer stacking with logistic regression on just a few models) outperform heavy ensembles and vice versa. \n3. `Specific Ensemble Strategy`: Nothing seems to always work. For each strategy, we have many similar ensembles. They can range from top 30 to maybe 200th with similar CV and public LB.\n4. `CV` vs `LB` vs `?`: We have quite a few private 808 subs and none of them are best CV or best LB. I can easily find a similar model with a better CV and LB and scores a lot worse on the private LB. \n5. `NN/GBDT Ensemble` vs `Pure-GBDT`: Again no conclusive observation here. Including NN in the ensemble doesn’t seem to deterministically make performance better or worse in my case. \n\nSadly no immediate takeaways from my own solutions ☹️☹️☹️. \n \n \n \n**Things I learned post-competition that worked well for others:**\n1. [Knowledge Distillation](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347641) from @cdeotte \n2. Capturing time-series dynamics within each column \n[SequentialEncoder](https://www.kaggle.com/code/pavelvod/27-place-sequentialencoder?scriptVersionId=104154431) from @pavelvod and [Mini-LSTMs](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347651) from @fritzcremer\n3. [Ensemble of multiple TabNets](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347688) from @raddar \n\n**Hope you can share your experience here regarding what consistently worked great for you or your takeaways.**", "1913718": "Many thanks for the post and the links to other resources too! This is a great learning experience for beginners like me!", "1922195": "Hi @raphael1123. May I invite you to participate in this survey regarding your experience on Kaggle (10 min)? This is not a scam. We are a group of researchers at the City University of Hong Kong. The survey link is: https://cityuhk.questionpro.com/survey-of-kaggle-contestants" }, "source": "meta" }