\nWe noticed that ensemble weights are determined based on Public LB and overfit if based on CV.
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1985486%2F868d637a78b88773048c6f2c9b615402%2F2022-08-27%20063723.png?generation=1661549864910264&alt=media\")
I would like to thank the organizers for organizing this competition, the participants for sharing their many insights, and my teammates( @baosenguo @wimwim @scumufeng ). Now @scumufeng and I are promoted to kaggle master and @wimwim is reaching for GM.
\nThis competition will be unforgettable for me:)
Our solution is the result of ensembling several GBDT models , Transfomr, 2d-CNN, and GRU.
\nWe noticed that ensemble weights are determined based on Public LB and overfit if based on CV.
\n
We are using the dataset shared with us by raddar. The features are based on those shared by ragnar.(Thanks to both of you.)
\nI did not know this is called a meta feature.
\nThis feature was useful not only in GBDT, but also in Transformer.
\nIf added to chris's Transformer, the LB will increase from 0.790 to 793.
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1985486%2F9bc12bb44eee04b66decedd22ae185f1%2F2022-08-27%20055050.png?generation=1661547064786565&alt=media\")
Combine all features horizontally.
\nP_2_0 , P_2_1 , P_2_3 , P_2_4 , ... , P_2_12 , B_30_0 , ... \n XXX , XXX , XXX , XXX , ... , XXX , YYY , ...\nThis is his solution.
\nhttps://www.kaggle.com/competitions/amex-default-prediction/discussion/348118
All posts above 0.801 in Public LB were in the Gold zone in Private LB.(I prayed on the last day not to Shake down😣)
\nLet's enjoy kaggle! Thank you!!!!
", "messages": [ { "id": "1915338", "postDate": "08/26/2022 22:29:13", "content": "I would like to thank the organizers for organizing this competition, the participants for sharing their many insights, and my teammates( @baosenguo @wimwim @scumufeng ). Now @scumufeng and I are promoted to kaggle master and @wimwim is reaching for GM.
\nThis competition will be unforgettable for me:)
Our solution is the result of ensembling several GBDT models , Transfomr, 2d-CNN, and GRU.
\nWe noticed that ensemble weights are determined based on Public LB and overfit if based on CV.
\n
We are using the dataset shared with us by raddar. The features are based on those shared by ragnar.(Thanks to both of you.)
\nI did not know this is called a meta feature.
\nThis feature was useful not only in GBDT, but also in Transformer.
\nIf added to chris's Transformer, the LB will increase from 0.790 to 793.
Combine all features horizontally.
\nP_2_0 , P_2_1 , P_2_3 , P_2_4 , ... , P_2_12 , B_30_0 , ... \n XXX , XXX , XXX , XXX , ... , XXX , YYY , ...\nThis is his solution.
\nhttps://www.kaggle.com/competitions/amex-default-prediction/discussion/348118
All posts above 0.801 in Public LB were in the Gold zone in Private LB.(I prayed on the last day not to Shake down😣)
\nLet's enjoy kaggle! Thank you!!!!
", "rawMarkdown": "I would like to thank the organizers for organizing this competition, the participants for sharing their many insights, and my teammates( @baosenguo @wimwim @scumufeng ). Now @scumufeng and I are promoted to kaggle master and @wimwim is reaching for GM.\nThis competition will be unforgettable for me:)\n\n# Summary\nOur solution is the result of ensembling several GBDT models , Transfomr, 2d-CNN, and GRU.\nWe noticed that ensemble weights are determined based on Public LB and overfit if based on CV.\n\n\n\n\n# Features\nWe are using the dataset shared with us by raddar. The features are based on those shared by ragnar.(Thanks to both of you.)\n\n### meta feature\nI did not know this is called a meta feature.\nThis feature was useful not only in GBDT, but also in Transformer.\nIf added to [chris's Transformer](https://www.kaggle.com/code/cdeotte/tensorflow-transformer-0-790l), the LB will increase from 0.790 to 793.\n\n\n\n### Pivot\nCombine all features horizontally.\n\n```\nP_2_0 , P_2_1 , P_2_3 , P_2_4 , ... , P_2_12 , B_30_0 , ... \n XXX , XXX , XXX , XXX , ... , XXX , YYY , ...\n```\n\n# Model\n### GBDT\n- LightGBM\n - Almost no change from [this Notebook](https://www.kaggle.com/code/ragnar123/amex-lgbm-dart-cv-0-7977)\n - stratfiedKfold: 10\n- Catboost\n - Use GPU(I was surprised at how fast it was.)\n - parameter : default\n\n### Transformer\n#### zakopuro\n- Based on [chris's Notebook]((https://www.kaggle.com/code/cdeotte/tensorflow-transformer-0-790l)\n- Some additional features.(Mainly meta features)\n\n#### Patrick Yam\nThis is his solution.\nhttps://www.kaggle.com/competitions/amex-default-prediction/discussion/348118\n\n#### mufeng\n- Add meta feature to Patrick's transformer.\n - LB increased from 0.795 to 0.798.\n\n# Ensemble\n- Use 21 models.\n- Ensemble weight\n - Determined based on Public LB\n - In our case, the LB score will be lower if based on CV.(CV is 0.8016 or higher.)\n - We trusted Public LB more than CV because it is close to Private and has a sufficient amount of data.\n- Weights are not complicated. (For example, 0.1,0.2,... etc.)\n\n# Select Submit\n- Best LB\n - Public : 0.80199(2nd)\n - Priavte : 0.80881(6th)\n- Best LB*0.5 + Best CV *0.5\n - Public : 0.80154\n - Private : 0.80862(Gold zone)\n- Correlation check\n - Check the Public and Private correlation values for all predictions used in the ensemble to see that there are no significant differences.\n\nAll posts above 0.801 in Public LB were in the Gold zone in Private LB.(I prayed on the last day not to Shake down😣)\n\nLet's enjoy kaggle! Thank you!!!!", "votes": null }, { "id": "1915343", "postDate": "08/26/2022 22:38:32", "content": "Congratulations team. Great solution. I didn't know about these \"meta features\" but i see that many top teams used them. I will use them in my future competitions. Thanks for sharing your solution.
", "rawMarkdown": "Congratulations team. Great solution. I didn't know about these \"meta features\" but i see that many top teams used them. I will use them in my future competitions. Thanks for sharing your solution.", "votes": null }, { "id": "1915353", "postDate": "08/26/2022 22:54:53", "content": "Thank you very much.
\nI have learned a lot from you in this competition as well!
In your diagram, i see that you pretrain Transformer then finetune. What data did you pretrain with and what data did you finetune with?
", "rawMarkdown": "In your diagram, i see that you pretrain Transformer then finetune. What data did you pretrain with and what data did you finetune with?", "votes": null }, { "id": "1915362", "postDate": "08/26/2022 23:04:55", "content": "Patrick will write for more details.
\nMy understanding is that the pretrain for GBDT features and finetune using target.
This is his solution.
\nhttps://www.kaggle.com/competitions/amex-default-prediction/discussion/348118
Congrats on the gold medal. Nice solution and nice write-up. The \"meta-features\" approach is very interesting. If I understand correctly you did a number of rounds of adding these meta-features -- taking the average of the previous ones and keeping the last one intact. After how many rounds did CV/LB improvements stagnate?
", "rawMarkdown": "Congrats on the gold medal. Nice solution and nice write-up. The \"meta-features\" approach is very interesting. If I understand correctly you did a number of rounds of adding these meta-features -- taking the average of the previous ones and keeping the last one intact. After how many rounds did CV/LB improvements stagnate?", "votes": null }, { "id": "1916545", "postDate": "08/28/2022 01:28:31", "content": "Thank you.
\nFour \"meta-features\" were used. However, in the case of GBDT, the use of \"meta-features\" does not improve the score of the single model much. The model using this feature is more effective for ensembles.
\nIn Transformer's case, increasing the number by more than four had no effect.
Congratulations. I found very interesting the process of ensembling that wide variety of models and it is a good reference for future competitions. Thank you so much for this answer.
", "rawMarkdown": "Congratulations. I found very interesting the process of ensembling that wide variety of models and it is a good reference for future competitions. Thank you so much for this answer.", "votes": null }, { "id": "1924901", "postDate": "09/03/2022 14:01:36", "content": "Congrats for all of you that were promoted and thanks for sharing! I would like to ask a couple of questions:
\n1. The \"pivot\" part is only for the Transformers, correct?
\n2. From picture I interpret that each team member developed his own features, how different are from Martin's features?
\n3. I'm curious about the 2d-CNN model, do you have more info?
Thank you!
\nThe \"pivot\" part is only for the Transformers, correct?
\n-> This is a feature for the GBDT, not for the Transformer.
From picture I interpret that each team member developed his own features, how different are from Martin's features?
\n-> There are no major differences; diff features are added, rounding with float, etc.
I'm curious about the 2d-CNN model, do you have more info?
\n-> It is similar to this content.(https://www.kaggle.com/competitions/lish-moa/discussion/202256#1106810)
Thanks for the answers! Then the GBDT used pivoted features (i.e. the 13 pivoted statements) instead of aggregations of them? Or maybe It used both?
", "rawMarkdown": "Thanks for the answers! Then the GBDT used pivoted features (i.e. the 13 pivoted statements) instead of aggregations of them? Or maybe It used both?", "votes": null }, { "id": "1925755", "postDate": "09/04/2022 09:19:57", "content": "It used both.
", "rawMarkdown": "It used both.", "votes": null } ], "comments": [ { "id": 1915343, "author_name": "cdeotte", "author_url": "", "post_date": "08/26/2022 22:38:32", "content": "Congratulations team. Great solution. I didn't know about these \"meta features\" but i see that many top teams used them. I will use them in my future competitions. Thanks for sharing your solution.
", "votes": null, "replies": [ { "id": 1915353, "author_name": "zakopur0", "author_url": "", "post_date": "08/26/2022 22:54:53", "content": "Thank you very much.
\nI have learned a lot from you in this competition as well!
In your diagram, i see that you pretrain Transformer then finetune. What data did you pretrain with and what data did you finetune with?
", "votes": null, "replies": [] }, { "id": 1915362, "author_name": "zakopur0", "author_url": "", "post_date": "08/26/2022 23:04:55", "content": "Patrick will write for more details.
\nMy understanding is that the pretrain for GBDT features and finetune using target.
This is his solution.
\nhttps://www.kaggle.com/competitions/amex-default-prediction/discussion/348118
Congrats on the gold medal. Nice solution and nice write-up. The \"meta-features\" approach is very interesting. If I understand correctly you did a number of rounds of adding these meta-features -- taking the average of the previous ones and keeping the last one intact. After how many rounds did CV/LB improvements stagnate?
", "votes": null, "replies": [ { "id": 1916545, "author_name": "zakopur0", "author_url": "", "post_date": "08/28/2022 01:28:31", "content": "Thank you.
\nFour \"meta-features\" were used. However, in the case of GBDT, the use of \"meta-features\" does not improve the score of the single model much. The model using this feature is more effective for ensembles.
\nIn Transformer's case, increasing the number by more than four had no effect.
Congratulations. I found very interesting the process of ensembling that wide variety of models and it is a good reference for future competitions. Thank you so much for this answer.
", "votes": null, "replies": [] }, { "id": 1924901, "author_name": "delai50", "author_url": "", "post_date": "09/03/2022 14:01:36", "content": "Congrats for all of you that were promoted and thanks for sharing! I would like to ask a couple of questions:
\n1. The \"pivot\" part is only for the Transformers, correct?
\n2. From picture I interpret that each team member developed his own features, how different are from Martin's features?
\n3. I'm curious about the 2d-CNN model, do you have more info?
Thank you!
\nThe \"pivot\" part is only for the Transformers, correct?
\n-> This is a feature for the GBDT, not for the Transformer.
From picture I interpret that each team member developed his own features, how different are from Martin's features?
\n-> There are no major differences; diff features are added, rounding with float, etc.
I'm curious about the 2d-CNN model, do you have more info?
\n-> It is similar to this content.(https://www.kaggle.com/competitions/lish-moa/discussion/202256#1106810)
Thanks for the answers! Then the GBDT used pivoted features (i.e. the 13 pivoted statements) instead of aggregations of them? Or maybe It used both?
", "votes": null, "replies": [] } ] }, { "id": 1925755, "author_name": "zakopur0", "author_url": "", "post_date": "09/04/2022 09:19:57", "content": "It used both.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1915338": "I would like to thank the organizers for organizing this competition, the participants for sharing their many insights, and my teammates( @baosenguo @wimwim @scumufeng ). Now @scumufeng and I are promoted to kaggle master and @wimwim is reaching for GM.\nThis competition will be unforgettable for me:)\n\n# Summary\nOur solution is the result of ensembling several GBDT models , Transfomr, 2d-CNN, and GRU.\nWe noticed that ensemble weights are determined based on Public LB and overfit if based on CV.\n\n\n\n\n# Features\nWe are using the dataset shared with us by raddar. The features are based on those shared by ragnar.(Thanks to both of you.)\n\n### meta feature\nI did not know this is called a meta feature.\nThis feature was useful not only in GBDT, but also in Transformer.\nIf added to [chris's Transformer](https://www.kaggle.com/code/cdeotte/tensorflow-transformer-0-790l), the LB will increase from 0.790 to 793.\n\n\n\n### Pivot\nCombine all features horizontally.\n\n```\nP_2_0 , P_2_1 , P_2_3 , P_2_4 , ... , P_2_12 , B_30_0 , ... \n XXX , XXX , XXX , XXX , ... , XXX , YYY , ...\n```\n\n# Model\n### GBDT\n- LightGBM\n - Almost no change from [this Notebook](https://www.kaggle.com/code/ragnar123/amex-lgbm-dart-cv-0-7977)\n - stratfiedKfold: 10\n- Catboost\n - Use GPU(I was surprised at how fast it was.)\n - parameter : default\n\n### Transformer\n#### zakopuro\n- Based on [chris's Notebook]((https://www.kaggle.com/code/cdeotte/tensorflow-transformer-0-790l)\n- Some additional features.(Mainly meta features)\n\n#### Patrick Yam\nThis is his solution.\nhttps://www.kaggle.com/competitions/amex-default-prediction/discussion/348118\n\n#### mufeng\n- Add meta feature to Patrick's transformer.\n - LB increased from 0.795 to 0.798.\n\n# Ensemble\n- Use 21 models.\n- Ensemble weight\n - Determined based on Public LB\n - In our case, the LB score will be lower if based on CV.(CV is 0.8016 or higher.)\n - We trusted Public LB more than CV because it is close to Private and has a sufficient amount of data.\n- Weights are not complicated. (For example, 0.1,0.2,... etc.)\n\n# Select Submit\n- Best LB\n - Public : 0.80199(2nd)\n - Priavte : 0.80881(6th)\n- Best LB*0.5 + Best CV *0.5\n - Public : 0.80154\n - Private : 0.80862(Gold zone)\n- Correlation check\n - Check the Public and Private correlation values for all predictions used in the ensemble to see that there are no significant differences.\n\nAll posts above 0.801 in Public LB were in the Gold zone in Private LB.(I prayed on the last day not to Shake down😣)\n\nLet's enjoy kaggle! Thank you!!!!", "1915343": "Congratulations team. Great solution. I didn't know about these \"meta features\" but i see that many top teams used them. I will use them in my future competitions. Thanks for sharing your solution.", "1915353": "Thank you very much.\nI have learned a lot from you in this competition as well!", "1915359": "In your diagram, i see that you pretrain Transformer then finetune. What data did you pretrain with and what data did you finetune with?", "1915362": "Patrick will write for more details.\nMy understanding is that the pretrain for GBDT features and finetune using target.", "1915470": "This is his solution.\nhttps://www.kaggle.com/competitions/amex-default-prediction/discussion/348118", "1916004": "Congrats on the gold medal. Nice solution and nice write-up. The \"meta-features\" approach is very interesting. If I understand correctly you did a number of rounds of adding these meta-features -- taking the average of the previous ones and keeping the last one intact. After how many rounds did CV/LB improvements stagnate?", "1916545": "Thank you.\nFour \"meta-features\" were used. However, in the case of GBDT, the use of \"meta-features\" does not improve the score of the single model much. The model using this feature is more effective for ensembles.\nIn Transformer's case, increasing the number by more than four had no effect.", "1918329": "Congratulations. I found very interesting the process of ensembling that wide variety of models and it is a good reference for future competitions. Thank you so much for this answer.", "1924901": "Congrats for all of you that were promoted and thanks for sharing! I would like to ask a couple of questions:\n\n**1.** The \"pivot\" part is only for the Transformers, correct?\n**2.** From picture I interpret that each team member developed his own features, how different are from Martin's features?\n**3.** I'm curious about the 2d-CNN model, do you have more info?", "1925530": "Thank you!\n\n1. The \"pivot\" part is only for the Transformers, correct?\n-> This is a feature for the GBDT, not for the Transformer.\n\n2. From picture I interpret that each team member developed his own features, how different are from Martin's features?\n-> There are no major differences; diff features are added, rounding with float, etc.\n\n3. I'm curious about the 2d-CNN model, do you have more info?\n-> It is similar to this content.(https://www.kaggle.com/competitions/lish-moa/discussion/202256#1106810)", "1925746": "Thanks for the answers! Then the GBDT used pivoted features (i.e. the 13 pivoted statements) instead of aggregations of them? Or maybe It used both?", "1925755": "It used both." }, "source": "meta" }