![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4900136%2F7a02721f85aa7042adf4202cb9e7c624%2Fensemble.png?generation=1661418572999162&alt=media\")
Thank you Amex and Kaggle for hosting! Very fun competition with a lot of insights.
\nThis is the first of two posts of our ensemble @gandagorn . My leg of the ensemble consisted of taking different preprocessing methods and passing them through different architectures (Thank you @jiweiliu @lucasmorin @raddar ! for your excellent notebooks)
\nsecond part is https://www.kaggle.com/competitions/amex-default-prediction/discussion/348058
Sequentially removed features accordingly to split importance.
\nStandard lgbm trained with dart with one-sided focal loss added to the log-loss. The addition of the one-side focal loss was with the objective of having a better classification of the positive classes.
\nNothing special.
\nFully connected architecture using the same features that were used in the boosted trees. 20 folds CV.
\nProbably the most interesting model. A NN with double inputs: tabular features (from previous preprocessing) and full-time series of ranked raw features. The training process was as follows:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4900136%2F31f1f8ded2cba706ad095ed7595847fc%2Farchitecture.png?generation=1661419863567301&alt=media\")
For the Transformer, I used the standard TransformerEncoderBlock with 8 heads and 2 layers
", "messages": [ { "id": "1913348", "postDate": "08/25/2022 09:35:37", "content": "Thank you Amex and Kaggle for hosting! Very fun competition with a lot of insights.
\nThis is the first of two posts of our ensemble @gandagorn . My leg of the ensemble consisted of taking different preprocessing methods and passing them through different architectures (Thank you @jiweiliu @lucasmorin @raddar ! for your excellent notebooks)
\nsecond part is https://www.kaggle.com/competitions/amex-default-prediction/discussion/348058
Sequentially removed features accordingly to split importance.
\nStandard lgbm trained with dart with one-sided focal loss added to the log-loss. The addition of the one-side focal loss was with the objective of having a better classification of the positive classes.
\nNothing special.
\nFully connected architecture using the same features that were used in the boosted trees. 20 folds CV.
\nProbably the most interesting model. A NN with double inputs: tabular features (from previous preprocessing) and full-time series of ranked raw features. The training process was as follows:
\n
For the Transformer, I used the standard TransformerEncoderBlock with 8 heads and 2 layers
", "rawMarkdown": "Thank you Amex and Kaggle for hosting! Very fun competition with a lot of insights. \n\n## Solution\n\nThis is the first of two posts of our ensemble @gandagorn . My leg of the ensemble consisted of taking different preprocessing methods and passing them through different architectures (Thank you @jiweiliu @lucasmorin @raddar ! for your excellent notebooks) \nsecond part is [https://www.kaggle.com/competitions/amex-default-prediction/discussion/348058](url)\n\n\n\n\n## Preprocessing\n\n1. Used the @raddar integer features and raw features and then created separate pre-processing pipelines including standard aggregations (min, max means, last, diff) \n2. Added top 50 PCA Features. \n3. Globally ranked all the features and built products of features that shared the same trend. This boosted all the models. \n4. Used embedded layers for all categorical features. \n\n## PostProcessing\n\nSequentially removed features accordingly to split importance. \n\n## Models\n\n### LGBM with Focal Loss + logloss\n\nStandard lgbm trained with dart with one-sided focal loss added to the log-loss. The addition of the one-side focal loss was with the objective of having a better classification of the positive classes.\n\n### XGBoost\n\nNothing special.\n\n### FCN\n\nFully connected architecture using the same features that were used in the boosted trees. 20 folds CV.\n\n### FCN + Transformer Encoder\n\nProbably the most interesting model. A NN with double inputs: tabular features (from previous preprocessing) and full-time series of ranked raw features. The training process was as follows:\n\n1. Freeze Transformer weights and train the FCN architecture for ~30 epochs to produce 1d output [0,1]\n2. Freeze FCN weights and train the Transformer for ~30 epochs to produce 1d output [0,1]\n3. Freeze Transformer weights and train the FCN architecture for ~30 epochs to produce 1d output [0,1]\n\n\n\n\nFor the Transformer, I used the standard TransformerEncoderBlock with 8 heads and 2 layers", "votes": null }, { "id": "1913358", "postDate": "08/25/2022 09:38:31", "content": "Muchas felicidades José Antonio (y a Daniel)
", "rawMarkdown": "Muchas felicidades José Antonio (y a Daniel)", "votes": null }, { "id": "1913360", "postDate": "08/25/2022 09:40:07", "content": "Gracias Santiago
", "rawMarkdown": "Gracias Santiago", "votes": null }, { "id": "1913369", "postDate": "08/25/2022 09:51:10", "content": "Congratulations on your achievement @joseantonioalatorre, it would be nice to get these notebooks
", "rawMarkdown": "Congratulations on your achievement @joseantonioalatorre, it would be nice to get these notebooks", "votes": null }, { "id": "1913375", "postDate": "08/25/2022 09:54:52", "content": "This is really really impressive!
\nDo you plan to release the code?
I am afraid I cant because I use custom wrappers for the models that are not public so will take me looooong to re-write the code :/
", "rawMarkdown": "I am afraid I cant because I use custom wrappers for the models that are not public so will take me looooong to re-write the code :/", "votes": null }, { "id": "1913507", "postDate": "08/25/2022 10:46:12", "content": "Thanks a lot for the mention. Did you use more than base aggregate ?
", "rawMarkdown": "Thanks a lot for the mention. Did you use more than base aggregate ?", "votes": null }, { "id": "1913532", "postDate": "08/25/2022 10:55:49", "content": "No only min,max,mean last, and diff last-first
", "rawMarkdown": "No only min,max,mean last, and diff last-first", "votes": null }, { "id": "1913639", "postDate": "08/25/2022 12:28:07", "content": "Wonderful job!
\nUnfortunately in the end we failed to merge.
Maybe next time!
", "rawMarkdown": "Maybe next time!", "votes": null }, { "id": "1913750", "postDate": "08/25/2022 13:39:22", "content": "Wishing you hearty congratulations @joseantonioalatorre for the innovative approach and the fantastic result it gave you!
", "rawMarkdown": "Wishing you hearty congratulations @joseantonioalatorre for the innovative approach and the fantastic result it gave you!", "votes": null }, { "id": "1913791", "postDate": "08/25/2022 14:05:25", "content": "Wow！ good job！Thanks for sharing！
", "rawMarkdown": "Wow！ good job！Thanks for sharing！", "votes": null }, { "id": "1914074", "postDate": "08/25/2022 17:48:03", "content": "Congrats … THe hybrid approach is really interesting and seems the secret sauce we tried the two approaches separately and didn't bring fruits like the one you had merging the two as separate inputs. Hope if you could share the code for this Model architecture .
", "rawMarkdown": "Congrats ... THe hybrid approach is really interesting and seems the secret sauce we tried the two approaches separately and didn't bring fruits like the one you had merging the two as separate inputs. Hope if you could share the code for this Model architecture .", "votes": null }, { "id": "1915083", "postDate": "08/26/2022 16:40:48", "content": "Hi Gaurav, its on private wrappers so I cant share them
", "rawMarkdown": "Hi Gaurav, its on private wrappers so I cant share them", "votes": null }, { "id": "1915120", "postDate": "08/26/2022 17:17:34", "content": "np Thanks will try to make for my knowledge based on the notes you stated.. Think ensembles did at last achieve better scores ( but not selected I think this has been case for most :) ) ,but your model is great as it can be used in a more real world scenario .. 👍
", "rawMarkdown": "np Thanks will try to make for my knowledge based on the notes you stated.. Think ensembles did at last achieve better scores ( but not selected I think this has been case for most :) ) ,but your model is great as it can be used in a more real world scenario .. 👍", "votes": null }, { "id": "1915328", "postDate": "08/26/2022 21:50:59", "content": "Really impressive and innovative approach! Many congrats!
", "rawMarkdown": "Really impressive and innovative approach! Many congrats!", "votes": null }, { "id": "1916950", "postDate": "08/28/2022 09:48:32", "content": "Hi @joseantonioalatorre I wanted to ask you a question, regarding the differences (diff) in feature engineering did you also use the differences in absolute value?
", "rawMarkdown": "Hi @joseantonioalatorre I wanted to ask you a question, regarding the differences (diff) in feature engineering did you also use the differences in absolute value?", "votes": null }, { "id": "1917002", "postDate": "08/28/2022 10:29:26", "content": "Hi Raimondo, no I did not
", "rawMarkdown": "Hi Raimondo, no I did not", "votes": null }, { "id": "1928194", "postDate": "09/06/2022 10:11:40", "content": "Happy to see a spanish kaggler in the gold zone, congratulations and thanks for sharing!
\nIn would like to ask some questions:
\n1. Over which features did you compute the PCA?
\n2. Regarding the globally ranked features, \"globally\" means that you ranked them considering all customers or per customer?
\n3. How was the performance of the FCN compared with the GBDTs? Could you roughly share the architecture?
\n4. Did you experiment a boost after sequentially removing features by split importance?
\n5. The hybrid FCN + Transformer is very interesting, what was its CV/LB score? However, I think I didn't fully get the training procedure. For example, at first, you freeze Transformer weights and train the FCN: does this mean that during this step the 0.5FCN + 0.5Transformer is still carried out but with Transformer initial (not trained) weights?
Hi
\nCongrats ! The ensembling method is amzaing. By the way, I wonder the software you use to plot the first figrue.
", "rawMarkdown": "Congrats ! The ensembling method is amzaing. By the way, I wonder the software you use to plot the first figrue.", "votes": null } ], "comments": [ { "id": 1913358, "author_name": "santiagomota", "author_url": "", "post_date": "08/25/2022 09:38:31", "content": "Muchas felicidades José Antonio (y a Daniel)
", "votes": null, "replies": [ { "id": 1913360, "author_name": "joseantonioalatorre", "author_url": "", "post_date": "08/25/2022 09:40:07", "content": "Gracias Santiago
", "votes": null, "replies": [] } ] }, { "id": 1913369, "author_name": "raimondomelis", "author_url": "", "post_date": "08/25/2022 09:51:10", "content": "Congratulations on your achievement @joseantonioalatorre, it would be nice to get these notebooks
", "votes": null, "replies": [] }, { "id": 1913375, "author_name": "thedevastator", "author_url": "", "post_date": "08/25/2022 09:54:52", "content": "This is really really impressive!
\nDo you plan to release the code?
I am afraid I cant because I use custom wrappers for the models that are not public so will take me looooong to re-write the code :/
", "votes": null, "replies": [] } ] }, { "id": 1913507, "author_name": "lucasmorin", "author_url": "", "post_date": "08/25/2022 10:46:12", "content": "Thanks a lot for the mention. Did you use more than base aggregate ?
", "votes": null, "replies": [ { "id": 1913532, "author_name": "joseantonioalatorre", "author_url": "", "post_date": "08/25/2022 10:55:49", "content": "No only min,max,mean last, and diff last-first
", "votes": null, "replies": [] } ] }, { "id": 1913639, "author_name": "xiaowangiiiii", "author_url": "", "post_date": "08/25/2022 12:28:07", "content": "Wonderful job!
\nUnfortunately in the end we failed to merge.
Maybe next time!
", "votes": null, "replies": [] } ] }, { "id": 1913750, "author_name": "ravi20076", "author_url": "", "post_date": "08/25/2022 13:39:22", "content": "Wishing you hearty congratulations @joseantonioalatorre for the innovative approach and the fantastic result it gave you!
", "votes": null, "replies": [] }, { "id": 1913791, "author_name": "shanggangli", "author_url": "", "post_date": "08/25/2022 14:05:25", "content": "Wow！ good job！Thanks for sharing！
", "votes": null, "replies": [] }, { "id": 1914074, "author_name": "gauravbrills", "author_url": "", "post_date": "08/25/2022 17:48:03", "content": "Congrats … THe hybrid approach is really interesting and seems the secret sauce we tried the two approaches separately and didn't bring fruits like the one you had merging the two as separate inputs. Hope if you could share the code for this Model architecture .
", "votes": null, "replies": [ { "id": 1915083, "author_name": "joseantonioalatorre", "author_url": "", "post_date": "08/26/2022 16:40:48", "content": "Hi Gaurav, its on private wrappers so I cant share them
", "votes": null, "replies": [] }, { "id": 1915120, "author_name": "gauravbrills", "author_url": "", "post_date": "08/26/2022 17:17:34", "content": "np Thanks will try to make for my knowledge based on the notes you stated.. Think ensembles did at last achieve better scores ( but not selected I think this has been case for most :) ) ,but your model is great as it can be used in a more real world scenario .. 👍
", "votes": null, "replies": [] } ] }, { "id": 1915328, "author_name": "alberteinsten", "author_url": "", "post_date": "08/26/2022 21:50:59", "content": "Really impressive and innovative approach! Many congrats!
", "votes": null, "replies": [] }, { "id": 1916950, "author_name": "raimondomelis", "author_url": "", "post_date": "08/28/2022 09:48:32", "content": "Hi @joseantonioalatorre I wanted to ask you a question, regarding the differences (diff) in feature engineering did you also use the differences in absolute value?
", "votes": null, "replies": [ { "id": 1917002, "author_name": "joseantonioalatorre", "author_url": "", "post_date": "08/28/2022 10:29:26", "content": "Hi Raimondo, no I did not
", "votes": null, "replies": [] } ] }, { "id": 1928194, "author_name": "delai50", "author_url": "", "post_date": "09/06/2022 10:11:40", "content": "Happy to see a spanish kaggler in the gold zone, congratulations and thanks for sharing!
\nIn would like to ask some questions:
\n1. Over which features did you compute the PCA?
\n2. Regarding the globally ranked features, \"globally\" means that you ranked them considering all customers or per customer?
\n3. How was the performance of the FCN compared with the GBDTs? Could you roughly share the architecture?
\n4. Did you experiment a boost after sequentially removing features by split importance?
\n5. The hybrid FCN + Transformer is very interesting, what was its CV/LB score? However, I think I didn't fully get the training procedure. For example, at first, you freeze Transformer weights and train the FCN: does this mean that during this step the 0.5FCN + 0.5Transformer is still carried out but with Transformer initial (not trained) weights?
Hi
\nCongrats ! The ensembling method is amzaing. By the way, I wonder the software you use to plot the first figrue.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1913348": "Thank you Amex and Kaggle for hosting! Very fun competition with a lot of insights. \n\n## Solution\n\nThis is the first of two posts of our ensemble @gandagorn . My leg of the ensemble consisted of taking different preprocessing methods and passing them through different architectures (Thank you @jiweiliu @lucasmorin @raddar ! for your excellent notebooks) \nsecond part is [https://www.kaggle.com/competitions/amex-default-prediction/discussion/348058](url)\n\n\n\n\n## Preprocessing\n\n1. Used the @raddar integer features and raw features and then created separate pre-processing pipelines including standard aggregations (min, max means, last, diff) \n2. Added top 50 PCA Features. \n3. Globally ranked all the features and built products of features that shared the same trend. This boosted all the models. \n4. Used embedded layers for all categorical features. \n\n## PostProcessing\n\nSequentially removed features accordingly to split importance. \n\n## Models\n\n### LGBM with Focal Loss + logloss\n\nStandard lgbm trained with dart with one-sided focal loss added to the log-loss. The addition of the one-side focal loss was with the objective of having a better classification of the positive classes.\n\n### XGBoost\n\nNothing special.\n\n### FCN\n\nFully connected architecture using the same features that were used in the boosted trees. 20 folds CV.\n\n### FCN + Transformer Encoder\n\nProbably the most interesting model. A NN with double inputs: tabular features (from previous preprocessing) and full-time series of ranked raw features. The training process was as follows:\n\n1. Freeze Transformer weights and train the FCN architecture for ~30 epochs to produce 1d output [0,1]\n2. Freeze FCN weights and train the Transformer for ~30 epochs to produce 1d output [0,1]\n3. Freeze Transformer weights and train the FCN architecture for ~30 epochs to produce 1d output [0,1]\n\n\n\n\nFor the Transformer, I used the standard TransformerEncoderBlock with 8 heads and 2 layers", "1913358": "Muchas felicidades José Antonio (y a Daniel)", "1913360": "Gracias Santiago", "1913369": "Congratulations on your achievement @joseantonioalatorre, it would be nice to get these notebooks", "1913375": "This is really really impressive! \nDo you plan to release the code?", "1913437": "I am afraid I cant because I use custom wrappers for the models that are not public so will take me looooong to re-write the code :/", "1913507": "Thanks a lot for the mention. Did you use more than base aggregate ?", "1913532": "No only min,max,mean last, and diff last-first", "1913639": "Wonderful job! \nUnfortunately in the end we failed to merge.", "1913657": "Maybe next time!", "1913750": "Wishing you hearty congratulations @joseantonioalatorre for the innovative approach and the fantastic result it gave you!", "1913791": "Wow！ good job！Thanks for sharing！", "1914074": "Congrats ... THe hybrid approach is really interesting and seems the secret sauce we tried the two approaches separately and didn't bring fruits like the one you had merging the two as separate inputs. Hope if you could share the code for this Model architecture .", "1915083": "Hi Gaurav, its on private wrappers so I cant share them", "1915120": "np Thanks will try to make for my knowledge based on the notes you stated.. Think ensembles did at last achieve better scores ( but not selected I think this has been case for most :) ) ,but your model is great as it can be used in a more real world scenario .. 👍", "1915328": "Really impressive and innovative approach! Many congrats!", "1916950": "Hi @joseantonioalatorre I wanted to ask you a question, regarding the differences (diff) in feature engineering did you also use the differences in absolute value?", "1917002": "Hi Raimondo, no I did not", "1928194": "Happy to see a spanish kaggler in the gold zone, congratulations and thanks for sharing! \n\nIn would like to ask some questions:\n\n**1.** Over which features did you compute the PCA?\n**2.** Regarding the globally ranked features, \"globally\" means that you ranked them considering all customers or per customer?\n**3.** How was the performance of the FCN compared with the GBDTs? Could you roughly share the architecture?\n**4.** Did you experiment a boost after sequentially removing features by split importance? \n**5.** The hybrid FCN + Transformer is very interesting, what was its CV/LB score? However, I think I didn't fully get the training procedure. For example, at first, you freeze Transformer weights and train the FCN: does this mean that during this step the `0.5FCN + 0.5Transformer` is still carried out but with Transformer initial (not trained) weights?", "1928210": "Hi \n1. All of them.\n2. I considered all consumers.\n3. FCN and all my DL approaches perform ~.002 worst than other methods.\n4. Yes.\n5. (CV like in 3) Correct as you described it.", "1929711": "Congrats ! The ensembling method is amzaing. By the way, I wonder the software you use to plot the first figrue." }, "source": "meta" }