{ "id": 508124, "title": "57th place, 0.528. Without hack.", "url": "/competitions/home-credit-credit-risk-model-stability/writeups/andrey-nesterov-57th-place-0-528-without-hack", "author_name": "", "post_date": "2024-06-05T12:18:26.263Z", "votes": 24, "comment_count": 14, "views": 0, "content": "

Here is my efforts and final ensemble for this competition.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
model*Features Qt.CVPublicLBPrivateLB
catboost5510.6889720.5890.509
lgbm6970.6921200.5870.505
xgb7180.7026610.5810.504
lgbm_no-dates6010.6488290.5680.494
catboost_no-dates3200.6326680.5660.490
lama_mlp4330.6710070.5730.488
1dcnn5700.6874780.5620.476
L3 ensemble-0.6892440.6050.528
\n

* All models are 5-fold ensembles using the same StratifiedGroupKFold.
\n

\nMy initial sub-goal was to create a good enough diversity of models and datasets for the final ensemble. It's a common approach and I worked on it during the whole competition. Unfortunately, I didn't spend enough time on metric hacking, so my final position is not so good (only 62nd), although I was in top-10 before starting metric hacking.
\nMy final ensemble without metric hacking has a score of 0.528. The public metric hacking approach with adv classifier simply didn't work for PrivateLB.

\n



\nWhat helped improve the models:

\n
    \n
  1. Sorting by numgroups before aggregations - at least it worked for \"first\" and \"last\" aggregators. For some models it was used with combination of aggregation by numgroup2 and then by numgroup1 (after that some tables can also be concatenated, for example applprev, credit_bureau_a, credit_bureau_b).
    2. \n
  2. Using a 10-fold meta-classifier (LinearRegression) instead of a simple weighted ensemble improved the score to about +0.002. Also, while training meta_cls, I added some noise to the OOF predictions of some models to match the LB scores.
    3. \n
  3. Models using datasets without date columns - the scores of these models weren't as big, but they complemented the ensemble well enough.
    4. \n
  4. Using \"Coefficient of Variations\" aggregator - it helped a lot to improve the score of the CatBoost model.
    5. \n
  5. LAMA-MLP model improved initial ensemble in early stages of competition, and I spent some time to select features to overcome OOM errors and overfitting.
    6. \n
  6. One place where I directly incorporated the stability approach was using the metric for ReduceLRonPlateau for training MLP and 1DCNN models.
    7. \n
  7. Count encoding of categorical columns (from public notebooks)
    8. \n
  8. Filtering of correlated columns (from public notebooks).
    9. \n
  9. Collapse rare categories in one group.
    10. \n
  10. To overcome OOM errors, creating separate datasets and then merging them helped. Also memory issue was a big problem for 1DCNN model, but I overcame it by training model on TPU and then inferencing on GPU. Also TPU was used for permutation importance.
    11. \n
  11. CatBoost model has best private and public scores among other models. Also in one of the experiments 5-fold CatBoost achieved score 0.520 (difference was on using early stopping 100 and lambda 10) but unfortunately I didn't include that model in the final ensemble (LB was the same and CV I didn't know because that was inference training).
    12. \n
  12. Tuning some models using Optuna.
    13. \n
  13. The inclusion of the XGB model was relatively important. I also had problems training this model, because (as I understand later) it is not so good at categorical features (unlike LGBM and CatBoost), and preprocessing was critical for this model. More specifically, after using TargetEncoder (and then CatBoostEncoder), the XGB score increased to good enough values. The same approach was also used for 1DCNN model.
    14. \n
  14. Using slightly different features for all the models (some tables, aggregators, types of columns, filtering approaches were not included for all the models).
    15. \n
  15. Replacing NaN in num columns with 0 helped for 1DCNN model.
    16. \n
  16. Aggregators used: max, min, first, last, mean, var, sum, std plus coef_var (distributed differently among all models, column types, or used only for certain columns).
    17. \n
\n



\nWhat didn't work (at least in my experiments):

\n
    \n
  1. Feature selection based on adversarial validation (within train ds), permutation importance (including stability metric).
    2. \n
  2. I tried to create some loss functions (based on functions simpler than initial metric - from topics about metrics; incorporating weights and so on) for GBDT models, but without improvements.
    3. \n
  3. CV strategies based on splitting train ds on week numbers using distributions of some statistics.
    4. \n
  4. Tuning GBDT models using some specific parameters like dart for LGBM, ordered boosting_type for CatBoost and so on.
    5. \n
  5. Using IsolationForest with SMOTENC
    6. \n
  6. Different scalers for 1DCNN model like PowerTransformer, QuantileTransformer. Finally I used Log1p transformation only for columns with max_min-mean ratio greater than 100.
    7. \n
  7. Replace nums NaN with mean (or constant values like -99). BTW LAMA pipeline uses this approach, and I tried to use 0s, but LB didn't improve and I rejected this idea, but after deadline I saw that private score was improved for that submission.
    8. \n
  8. Compress some categories (e.g. by template P0_111_121 to P0 or P0_111)
    9. \n
  9. Some non-standard, custom aggregators such as entropy, mean-to-max, zero frequency, normalization within group, and so on. The goal was also to find aggragators with relative, not absolute, values.
    10. \n
  10. Additional feature engineering like ratios between some columns didn't improve models. As I understand after first competition was created very good features and squeezing out from dataset additional gains was not so simple task.
    11. \n
  11. By the results of previous competition I tried Regularized Greedy Forest (FastRGF version) but it has very poor performance and worsened final ensemble. Another idea from that competition was to train MLP only on some group of tables, but again without any results, so I discarded this idea as well. I also experimented with other models like DAE, TabTransformer, NN-cat_embeddings, packages like pytorch_tabular, but also I can't train them to good enough results.
    12. \n
  12. Combining some categorical features (similar to how CatBoost works).
    13. \n
  13. Balancing classes using parameters like scale_pos_weight.
    14. \n
  14. Include 1DCNN in the final ensemble. I struggled a lot to overcome the overfitting of this model, but eventually it is worsening the final ensemble from 0.529 to 0.528. Some variations of this model (with another ds) has relatively good private score (0.485) but LB was not so good (only 0.555), so I didn't include the best and maybe was digging in the wrong direction. Besides, without metric hacking this model will not contribute much to the final ensemble/position.
    15. \n
  15. Include some additional data such as FedFunds.
    16. \n
  16. Filter some weeks.
    17. \n
\n



\nOf course ideas from the community, discussions, public notebooks helped a lot.

\n



\nUpdate. 29.05.2024
\nJust out of curiosity, I tried adding the CatBoost model with score 0.520 to the final ensemble and the total private score increased to 0.536.

\n

Update. 5.06.2024
\nFeature importance by gain I used in selecting top-20 features from credit_bureau_a tables for lama-mlp model. This helped to avoid overfitting this model.

", "messages": [ { "id": "2840979", "postDate": "05/28/2024 10:39:34", "content": "

Here is my efforts and final ensemble for this competition.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
model*Features Qt.CVPublicLBPrivateLB
catboost5510.6889720.5890.509
lgbm6970.6921200.5870.505
xgb7180.7026610.5810.504
lgbm_no-dates6010.6488290.5680.494
catboost_no-dates3200.6326680.5660.490
lama_mlp4330.6710070.5730.488
1dcnn5700.6874780.5620.476
L3 ensemble-0.6892440.6050.528
\n

* All models are 5-fold ensembles using the same StratifiedGroupKFold.
\n

\nMy initial sub-goal was to create a good enough diversity of models and datasets for the final ensemble. It's a common approach and I worked on it during the whole competition. Unfortunately, I didn't spend enough time on metric hacking, so my final position is not so good (only 62nd), although I was in top-10 before starting metric hacking.
\nMy final ensemble without metric hacking has a score of 0.528. The public metric hacking approach with adv classifier simply didn't work for PrivateLB.

\n



\nWhat helped improve the models:

\n
    \n
  1. Sorting by numgroups before aggregations - at least it worked for \"first\" and \"last\" aggregators. For some models it was used with combination of aggregation by numgroup2 and then by numgroup1 (after that some tables can also be concatenated, for example applprev, credit_bureau_a, credit_bureau_b).
    2. \n
  2. Using a 10-fold meta-classifier (LinearRegression) instead of a simple weighted ensemble improved the score to about +0.002. Also, while training meta_cls, I added some noise to the OOF predictions of some models to match the LB scores.
    3. \n
  3. Models using datasets without date columns - the scores of these models weren't as big, but they complemented the ensemble well enough.
    4. \n
  4. Using \"Coefficient of Variations\" aggregator - it helped a lot to improve the score of the CatBoost model.
    5. \n
  5. LAMA-MLP model improved initial ensemble in early stages of competition, and I spent some time to select features to overcome OOM errors and overfitting.
    6. \n
  6. One place where I directly incorporated the stability approach was using the metric for ReduceLRonPlateau for training MLP and 1DCNN models.
    7. \n
  7. Count encoding of categorical columns (from public notebooks)
    8. \n
  8. Filtering of correlated columns (from public notebooks).
    9. \n
  9. Collapse rare categories in one group.
    10. \n
  10. To overcome OOM errors, creating separate datasets and then merging them helped. Also memory issue was a big problem for 1DCNN model, but I overcame it by training model on TPU and then inferencing on GPU. Also TPU was used for permutation importance.
    11. \n
  11. CatBoost model has best private and public scores among other models. Also in one of the experiments 5-fold CatBoost achieved score 0.520 (difference was on using early stopping 100 and lambda 10) but unfortunately I didn't include that model in the final ensemble (LB was the same and CV I didn't know because that was inference training).
    12. \n
  12. Tuning some models using Optuna.
    13. \n
  13. The inclusion of the XGB model was relatively important. I also had problems training this model, because (as I understand later) it is not so good at categorical features (unlike LGBM and CatBoost), and preprocessing was critical for this model. More specifically, after using TargetEncoder (and then CatBoostEncoder), the XGB score increased to good enough values. The same approach was also used for 1DCNN model.
    14. \n
  14. Using slightly different features for all the models (some tables, aggregators, types of columns, filtering approaches were not included for all the models).
    15. \n
  15. Replacing NaN in num columns with 0 helped for 1DCNN model.
    16. \n
  16. Aggregators used: max, min, first, last, mean, var, sum, std plus coef_var (distributed differently among all models, column types, or used only for certain columns).
    17. \n
\n



\nWhat didn't work (at least in my experiments):

\n
    \n
  1. Feature selection based on adversarial validation (within train ds), permutation importance (including stability metric).
    2. \n
  2. I tried to create some loss functions (based on functions simpler than initial metric - from topics about metrics; incorporating weights and so on) for GBDT models, but without improvements.
    3. \n
  3. CV strategies based on splitting train ds on week numbers using distributions of some statistics.
    4. \n
  4. Tuning GBDT models using some specific parameters like dart for LGBM, ordered boosting_type for CatBoost and so on.
    5. \n
  5. Using IsolationForest with SMOTENC
    6. \n
  6. Different scalers for 1DCNN model like PowerTransformer, QuantileTransformer. Finally I used Log1p transformation only for columns with max_min-mean ratio greater than 100.
    7. \n
  7. Replace nums NaN with mean (or constant values like -99). BTW LAMA pipeline uses this approach, and I tried to use 0s, but LB didn't improve and I rejected this idea, but after deadline I saw that private score was improved for that submission.
    8. \n
  8. Compress some categories (e.g. by template P0_111_121 to P0 or P0_111)
    9. \n
  9. Some non-standard, custom aggregators such as entropy, mean-to-max, zero frequency, normalization within group, and so on. The goal was also to find aggragators with relative, not absolute, values.
    10. \n
  10. Additional feature engineering like ratios between some columns didn't improve models. As I understand after first competition was created very good features and squeezing out from dataset additional gains was not so simple task.
    11. \n
  11. By the results of previous competition I tried Regularized Greedy Forest (FastRGF version) but it has very poor performance and worsened final ensemble. Another idea from that competition was to train MLP only on some group of tables, but again without any results, so I discarded this idea as well. I also experimented with other models like DAE, TabTransformer, NN-cat_embeddings, packages like pytorch_tabular, but also I can't train them to good enough results.
    12. \n
  12. Combining some categorical features (similar to how CatBoost works).
    13. \n
  13. Balancing classes using parameters like scale_pos_weight.
    14. \n
  14. Include 1DCNN in the final ensemble. I struggled a lot to overcome the overfitting of this model, but eventually it is worsening the final ensemble from 0.529 to 0.528. Some variations of this model (with another ds) has relatively good private score (0.485) but LB was not so good (only 0.555), so I didn't include the best and maybe was digging in the wrong direction. Besides, without metric hacking this model will not contribute much to the final ensemble/position.
    15. \n
  15. Include some additional data such as FedFunds.
    16. \n
  16. Filter some weeks.
    17. \n
\n



\nOf course ideas from the community, discussions, public notebooks helped a lot.

\n



\nUpdate. 29.05.2024
\nJust out of curiosity, I tried adding the CatBoost model with score 0.520 to the final ensemble and the total private score increased to 0.536.

\n

Update. 5.06.2024
\nFeature importance by gain I used in selecting top-20 features from credit_bureau_a tables for lama-mlp model. This helped to avoid overfitting this model.

", "rawMarkdown": "Here is my efforts and final ensemble for this competition.\n\n| model* | Features Qt.\t| CV\t| PublicLB\t| PrivateLB |\n| ---| --- | --- | --- | --- |\n| catboost\t| 551\t| 0.688972\t| 0.589\t| 0.509 |\n| lgbm\t| 697\t\t| 0.692120\t| 0.587\t| 0.505 |\n| xgb\t\t| 718\t\t| 0.702661\t| 0.581\t| 0.504 |\n| lgbm_no-dates\t| 601\t| 0.648829\t| 0.568\t| 0.494 |\n| catboost_no-dates\t| 320\t\t| 0.632668\t| 0.566\t | 0.490 |\n| lama_mlp\t| 433\t\t| 0.671007\t| 0.573\t| 0.488 |\n| 1dcnn\t| 570\t\t| 0.687478\t| 0.562\t| 0.476 |\n| L3 ensemble\t\t| -\t| 0.689244\t| 0.605\t| 0.528 |\n\n\\* All models are 5-fold ensembles using the same StratifiedGroupKFold.\n
\nMy initial sub-goal was to create a good enough diversity of models and datasets for the final ensemble. It's a common approach and I worked on it during the whole competition. Unfortunately, I didn't spend enough time on metric hacking, so my final position is not so good (only 62nd), although I was in top-10 before starting metric hacking.\nMy [final ensemble](https://www.kaggle.com/code/andreynesterov/home-credit-inference-final/notebook) without metric hacking has a score of 0.528. The public metric hacking approach with adv classifier simply didn't work for PrivateLB.\n\n
\nWhat helped improve the models:\n1. Sorting by numgroups before aggregations - at least it worked for \"first\" and \"last\" aggregators. For some models it was used with combination of aggregation by numgroup2 and then by numgroup1 (after that some tables can also be concatenated, for example applprev, credit_bureau_a, credit_bureau_b).\n1. Using a 10-fold meta-classifier (LinearRegression) instead of a simple weighted ensemble improved the score to about +0.002. Also, while training meta_cls, I added some noise to the OOF predictions of some models to match the LB scores.\n1. Models using datasets without date columns - the scores of these models weren't as big, but they complemented the ensemble well enough.\n1. Using \"Coefficient of Variations\" aggregator - it helped a lot to improve the score of the CatBoost model.\n1. LAMA-MLP model improved initial ensemble in early stages of competition, and I spent some time to select features to overcome OOM errors and overfitting.\n1. One place where I directly incorporated the stability approach was using the metric for ReduceLRonPlateau for training MLP and 1DCNN models.\n1. Count encoding of categorical columns (from public notebooks)\n1. Filtering of correlated columns (from public notebooks).\n1. Collapse rare categories in one group.\n1. To overcome OOM errors, creating separate datasets and then merging them helped. Also memory issue was a big problem for 1DCNN model, but I overcame it by training model on TPU and then inferencing on GPU. Also TPU was used for permutation importance.\n1. CatBoost model has best private and public scores among other models. Also in one of the experiments 5-fold CatBoost achieved score 0.520 (difference was on using early stopping 100 and lambda 10) but unfortunately I didn't include that model in the final ensemble (LB was the same and CV I didn't know because that was inference training).\n1. Tuning some models using Optuna.\n1. The inclusion of the XGB model was relatively important. I also had problems training this model, because (as I understand later) it is not so good at categorical features (unlike LGBM and CatBoost), and preprocessing was critical for this model. More specifically, after using TargetEncoder (and then CatBoostEncoder), the XGB score increased to good enough values. The same approach was also used for 1DCNN model.\n1. Using slightly different features for all the models (some tables, aggregators, types of columns, filtering approaches were not included for all the models).\n1. Replacing NaN in num columns with 0 helped for 1DCNN model.\n1. Aggregators used: max, min, first, last, mean, var, sum, std plus coef_var (distributed differently among all models, column types, or used only for certain columns).\n\n
\nWhat didn't work (at least in my experiments):\n1. Feature selection based on adversarial validation (within train ds), permutation importance (including stability metric).\n1. I tried to create some loss functions (based on functions simpler than initial metric - from topics about metrics; incorporating weights and so on) for GBDT models, but without improvements.\n1. CV strategies based on splitting train ds on week numbers using distributions of some statistics.\n1. Tuning GBDT models using some specific parameters like dart for LGBM, ordered boosting_type for CatBoost and so on.\n1. Using IsolationForest with SMOTENC\n1. Different scalers for 1DCNN model like PowerTransformer, QuantileTransformer. Finally I used Log1p transformation only for columns with max_min-mean ratio greater than 100.\n1. Replace nums NaN with mean (or constant values like -99). BTW LAMA pipeline uses this approach, and I tried to use 0s, but LB didn't improve and I rejected this idea, but after deadline I saw that private score was improved for that submission.\n1. Compress some categories (e.g. by template P0_111_121 to P0 or P0_111)\n1. Some non-standard, custom aggregators such as entropy, mean-to-max, zero frequency, normalization within group, and so on. The goal was also to find aggragators with relative, not absolute, values.\n1. Additional feature engineering like ratios between some columns didn't improve models. As I understand after first competition was created very good features and squeezing out from dataset additional gains was not so simple task.\n1. By the results of previous competition I tried Regularized Greedy Forest (FastRGF version) but it has very poor performance and worsened final ensemble. Another idea from that competition was to train MLP only on some group of tables, but again without any results, so I discarded this idea as well. I also experimented with other models like DAE, TabTransformer, NN-cat_embeddings, packages like pytorch_tabular, but also I can't train them to good enough results.\n1. Combining some categorical features (similar to how CatBoost works).\n1. Balancing classes using parameters like scale_pos_weight.\n1. Include 1DCNN in the final ensemble. I struggled a lot to overcome the overfitting of this model, but eventually it is worsening the final ensemble from 0.529 to 0.528. Some variations of this model (with another ds) has relatively good private score (0.485) but LB was not so good (only 0.555), so I didn't include the best and maybe was digging in the wrong direction. Besides, without metric hacking this model will not contribute much to the final ensemble/position.\n1. Include some additional data such as FedFunds.\n1. Filter some weeks.\n\n
\nOf course ideas from the community, discussions, public notebooks helped a lot.\n\n
\nUpdate. 29.05.2024\nJust out of curiosity, I tried adding the CatBoost model with score 0.520 to the final ensemble and the total private score increased to 0.536.\n\nUpdate. 5.06.2024\nFeature importance by gain I used in selecting top-20 features from credit_bureau_a tables for lama-mlp model. This helped to avoid overfitting this model.", "votes": null }, { "id": "2840985", "postDate": "05/28/2024 10:50:06", "content": "

What aggregates did you use for numeric columns?

", "rawMarkdown": "What aggregates did you use for numeric columns?", "votes": null }, { "id": "2841016", "postDate": "05/28/2024 11:04:38", "content": "

I used pretty common aggregators: max, min, first, last, mean, var, sum, std plus coef_var. But again, each dataset has slightly different aggregators, for example, I didn't include var method to ds with coef_var. Sum aggregator I used only in some datasets and only for certain columns (with enough diversity - I selected them manually).

", "rawMarkdown": "I used pretty common aggregators: max, min, first, last, mean, var, sum, std plus coef_var. But again, each dataset has slightly different aggregators, for example, I didn't include var method to ds with coef_var. Sum aggregator I used only in some datasets and only for certain columns (with enough diversity - I selected them manually).", "votes": null }, { "id": "2841041", "postDate": "05/28/2024 11:24:44", "content": "

Thanks! Have you experimented with diff features, namely last-mean, last-first?

", "rawMarkdown": "Thanks! Have you experimented with diff features, namely last-mean, last-first?", "votes": null }, { "id": "2841062", "postDate": "05/28/2024 11:34:01", "content": "

No, unfortunately.

", "rawMarkdown": "No, unfortunately.", "votes": null }, { "id": "2841208", "postDate": "05/28/2024 12:57:24", "content": "

Thank u for share . You mentioned \"Feature selection based on importance by gain\" didn't work. Did importance by split work?

", "rawMarkdown": "Thank u for share . You mentioned \"Feature selection based on importance by gain\" didn't work. Did importance by split work?", "votes": null }, { "id": "2841214", "postDate": "05/28/2024 13:05:21", "content": "

Yes, I selected features for MLP based on importance by split of LGBM, but importance by gain in my cases led to overfitting.

", "rawMarkdown": "Yes, I selected features for MLP based on importance by split of LGBM, but importance by gain in my cases led to overfitting.", "votes": null }, { "id": "2841222", "postDate": "05/28/2024 13:16:01", "content": "

Hi Andrey, congrats with the 62nd place! A lot of work is done!

\n

There are a few things that helped me to boost my score quite a bit are the following:

\n\n

Just curious, have you tried/considered such tail/diff aggregations to add to your data?

\n

I am still very puzzled to why the class balance does not work in this competition. When training my models, I even balanced the classes versus week number trying to exclude the possibility for models to learn any discrimination from this dimension ( as the WEEK_NUM is correlated with some inputs) with the hope this can enhance the model stability.

", "rawMarkdown": "Hi Andrey, congrats with the 62nd place! A lot of work is done!\n\nThere are a few things that helped me to boost my score quite a bit are the following:\n - Aggregation of numerical columns: last 3 rows tail(3).mean(); as well as diff() with mean and/or std\n - Many features created by hand using logical assumptions\n - The addition of stability metric had a minor effect\n - Max aggregations of string columns had a big effect. However, I rejected this due to a problem of interpretability with the hope that this is an artifact of the public LB test sample.\n\nJust curious, have you tried/considered such tail/diff aggregations to add to your data?\n\nI am still very puzzled to why the class balance does not work in this competition. When training my models, I even balanced the classes versus week number trying to exclude the possibility for models to learn any discrimination from this dimension ( as the WEEK_NUM is correlated with some inputs) with the hope this can enhance the model stability.", "votes": null }, { "id": "2841270", "postDate": "05/28/2024 13:48:16", "content": "

Hi Oleh. Thanks.
\nI was thinking about some kind of tail(n).some_agg() (I also read mention about it in previous competition solutions and discussions), but I think this idea got lost somewhere in my notes. I didn't think about diff.
\nAbout max agg for string columns - it may work because it contains some hierarchy in the some categories encoding (e.g. P0_111_121).

", "rawMarkdown": "Hi Oleh. Thanks.\nI was thinking about some kind of tail(n).some_agg() (I also read mention about it in previous competition solutions and discussions), but I think this idea got lost somewhere in my notes. I didn't think about diff.\nAbout max agg for string columns - it may work because it contains some hierarchy in the some categories encoding (e.g. P0_111_121).", "votes": null }, { "id": "2841285", "postDate": "05/28/2024 13:55:45", "content": "

You gain a lot from ensembling.
\nI have 0.598 single model and only 0.6 ensemble

", "rawMarkdown": "You gain a lot from ensembling. \nI have 0.598 single model and only 0.6 ensemble", "votes": null }, { "id": "2841298", "postDate": "05/28/2024 14:06:42", "content": "

Yes, that was a kind of exercise from my previous competitions. Of course, that's only one of the approaches, and a strong enough single model can beat a good ensemble.

", "rawMarkdown": "Yes, that was a kind of exercise from my previous competitions. Of course, that's only one of the approaches, and a strong enough single model can beat a good ensemble.", "votes": null }, { "id": "2841342", "postDate": "05/28/2024 14:36:27", "content": "

Hi @andreynesterov, thank you for sharing. I read that DART did not work well, but in my case it has improved the result and reach almost the catboost performance.

", "rawMarkdown": "Hi @andreynesterov, thank you for sharing. I read that DART did not work well, but in my case it has improved the result and reach almost the catboost performance.", "votes": null }, { "id": "2841364", "postDate": "05/28/2024 14:56:47", "content": "

That's great. I also thought dart was very promising, maybe I didn't pay much attention to tuning it or preparing features.

", "rawMarkdown": "That's great. I also thought dart was very promising, maybe I didn't pay much attention to tuning it or preparing features.", "votes": null }, { "id": "2841461", "postDate": "05/28/2024 15:31:22", "content": "

Thank you for the detailed insights on your findings. I have been following your work from the beginning of this competition and a lot of my work is inspired from your notebooks. Thank you for generously sharing those. Beginners like me learn a lot from your notebooks.

\n

I especially referred to the notebook where you ensembled xgb, cat, lgb model. I took the lessons from that notebook and tried ensembling 2 lgb and 2 cat - one of each with class balancing and other without the balancing and that did well. But, my best one which i did not submit was the one which i added a post-processing layer of manual implementation of isotonic calibration. it was a 0.593 on public LB and 0.527 on private LB.

", "rawMarkdown": "Thank you for the detailed insights on your findings. I have been following your work from the beginning of this competition and a lot of my work is inspired from your notebooks. Thank you for generously sharing those. Beginners like me learn a lot from your notebooks.\n\nI especially referred to the notebook where you ensembled xgb, cat, lgb model. I took the lessons from that notebook and tried ensembling 2 lgb and 2 cat - one of each with class balancing and other without the balancing and that did well. But, my best one which i did not submit was the one which i added a post-processing layer of manual implementation of isotonic calibration. it was a 0.593 on public LB and 0.527 on private LB.", "votes": null }, { "id": "2841501", "postDate": "05/28/2024 15:42:12", "content": "

That's interesting.

", "rawMarkdown": "That's interesting.", "votes": null } ], "comments": [ { "id": 2840985, "author_name": "andreasbis", "author_url": "", "post_date": "05/28/2024 10:50:06", "content": "

What aggregates did you use for numeric columns?

", "votes": null, "replies": [ { "id": 2841016, "author_name": "andreynesterov", "author_url": "", "post_date": "05/28/2024 11:04:38", "content": "

I used pretty common aggregators: max, min, first, last, mean, var, sum, std plus coef_var. But again, each dataset has slightly different aggregators, for example, I didn't include var method to ds with coef_var. Sum aggregator I used only in some datasets and only for certain columns (with enough diversity - I selected them manually).

", "votes": null, "replies": [ { "id": 2841041, "author_name": "andreasbis", "author_url": "", "post_date": "05/28/2024 11:24:44", "content": "

Thanks! Have you experimented with diff features, namely last-mean, last-first?

", "votes": null, "replies": [ { "id": 2841062, "author_name": "andreynesterov", "author_url": "", "post_date": "05/28/2024 11:34:01", "content": "

No, unfortunately.

", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2841208, "author_name": "sequoia12", "author_url": "", "post_date": "05/28/2024 12:57:24", "content": "

Thank u for share . You mentioned \"Feature selection based on importance by gain\" didn't work. Did importance by split work?

", "votes": null, "replies": [ { "id": 2841214, "author_name": "andreynesterov", "author_url": "", "post_date": "05/28/2024 13:05:21", "content": "

Yes, I selected features for MLP based on importance by split of LGBM, but importance by gain in my cases led to overfitting.

", "votes": null, "replies": [] } ] }, { "id": 2841222, "author_name": "olehkivernyk", "author_url": "", "post_date": "05/28/2024 13:16:01", "content": "

Hi Andrey, congrats with the 62nd place! A lot of work is done!

\n

There are a few things that helped me to boost my score quite a bit are the following:

\n\n

Just curious, have you tried/considered such tail/diff aggregations to add to your data?

\n

I am still very puzzled to why the class balance does not work in this competition. When training my models, I even balanced the classes versus week number trying to exclude the possibility for models to learn any discrimination from this dimension ( as the WEEK_NUM is correlated with some inputs) with the hope this can enhance the model stability.

", "votes": null, "replies": [ { "id": 2841270, "author_name": "andreynesterov", "author_url": "", "post_date": "05/28/2024 13:48:16", "content": "

Hi Oleh. Thanks.
\nI was thinking about some kind of tail(n).some_agg() (I also read mention about it in previous competition solutions and discussions), but I think this idea got lost somewhere in my notes. I didn't think about diff.
\nAbout max agg for string columns - it may work because it contains some hierarchy in the some categories encoding (e.g. P0_111_121).

", "votes": null, "replies": [] } ] }, { "id": 2841285, "author_name": "skrrydg", "author_url": "", "post_date": "05/28/2024 13:55:45", "content": "

You gain a lot from ensembling.
\nI have 0.598 single model and only 0.6 ensemble

", "votes": null, "replies": [ { "id": 2841298, "author_name": "andreynesterov", "author_url": "", "post_date": "05/28/2024 14:06:42", "content": "

Yes, that was a kind of exercise from my previous competitions. Of course, that's only one of the approaches, and a strong enough single model can beat a good ensemble.

", "votes": null, "replies": [] } ] }, { "id": 2841342, "author_name": "pourchot", "author_url": "", "post_date": "05/28/2024 14:36:27", "content": "

Hi @andreynesterov, thank you for sharing. I read that DART did not work well, but in my case it has improved the result and reach almost the catboost performance.

", "votes": null, "replies": [ { "id": 2841364, "author_name": "andreynesterov", "author_url": "", "post_date": "05/28/2024 14:56:47", "content": "

That's great. I also thought dart was very promising, maybe I didn't pay much attention to tuning it or preparing features.

", "votes": null, "replies": [] } ] }, { "id": 2841461, "author_name": "varuniraothumsi", "author_url": "", "post_date": "05/28/2024 15:31:22", "content": "

Thank you for the detailed insights on your findings. I have been following your work from the beginning of this competition and a lot of my work is inspired from your notebooks. Thank you for generously sharing those. Beginners like me learn a lot from your notebooks.

\n

I especially referred to the notebook where you ensembled xgb, cat, lgb model. I took the lessons from that notebook and tried ensembling 2 lgb and 2 cat - one of each with class balancing and other without the balancing and that did well. But, my best one which i did not submit was the one which i added a post-processing layer of manual implementation of isotonic calibration. it was a 0.593 on public LB and 0.527 on private LB.

", "votes": null, "replies": [ { "id": 2841501, "author_name": "andreynesterov", "author_url": "", "post_date": "05/28/2024 15:42:12", "content": "

That's interesting.

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2840979": "Here is my efforts and final ensemble for this competition.\n\n| model* | Features Qt.\t| CV\t| PublicLB\t| PrivateLB |\n| ---| --- | --- | --- | --- |\n| catboost\t| 551\t| 0.688972\t| 0.589\t| 0.509 |\n| lgbm\t| 697\t\t| 0.692120\t| 0.587\t| 0.505 |\n| xgb\t\t| 718\t\t| 0.702661\t| 0.581\t| 0.504 |\n| lgbm_no-dates\t| 601\t| 0.648829\t| 0.568\t| 0.494 |\n| catboost_no-dates\t| 320\t\t| 0.632668\t| 0.566\t | 0.490 |\n| lama_mlp\t| 433\t\t| 0.671007\t| 0.573\t| 0.488 |\n| 1dcnn\t| 570\t\t| 0.687478\t| 0.562\t| 0.476 |\n| L3 ensemble\t\t| -\t| 0.689244\t| 0.605\t| 0.528 |\n\n\\* All models are 5-fold ensembles using the same StratifiedGroupKFold.\n
\nMy initial sub-goal was to create a good enough diversity of models and datasets for the final ensemble. It's a common approach and I worked on it during the whole competition. Unfortunately, I didn't spend enough time on metric hacking, so my final position is not so good (only 62nd), although I was in top-10 before starting metric hacking.\nMy [final ensemble](https://www.kaggle.com/code/andreynesterov/home-credit-inference-final/notebook) without metric hacking has a score of 0.528. The public metric hacking approach with adv classifier simply didn't work for PrivateLB.\n\n
\nWhat helped improve the models:\n1. Sorting by numgroups before aggregations - at least it worked for \"first\" and \"last\" aggregators. For some models it was used with combination of aggregation by numgroup2 and then by numgroup1 (after that some tables can also be concatenated, for example applprev, credit_bureau_a, credit_bureau_b).\n1. Using a 10-fold meta-classifier (LinearRegression) instead of a simple weighted ensemble improved the score to about +0.002. Also, while training meta_cls, I added some noise to the OOF predictions of some models to match the LB scores.\n1. Models using datasets without date columns - the scores of these models weren't as big, but they complemented the ensemble well enough.\n1. Using \"Coefficient of Variations\" aggregator - it helped a lot to improve the score of the CatBoost model.\n1. LAMA-MLP model improved initial ensemble in early stages of competition, and I spent some time to select features to overcome OOM errors and overfitting.\n1. One place where I directly incorporated the stability approach was using the metric for ReduceLRonPlateau for training MLP and 1DCNN models.\n1. Count encoding of categorical columns (from public notebooks)\n1. Filtering of correlated columns (from public notebooks).\n1. Collapse rare categories in one group.\n1. To overcome OOM errors, creating separate datasets and then merging them helped. Also memory issue was a big problem for 1DCNN model, but I overcame it by training model on TPU and then inferencing on GPU. Also TPU was used for permutation importance.\n1. CatBoost model has best private and public scores among other models. Also in one of the experiments 5-fold CatBoost achieved score 0.520 (difference was on using early stopping 100 and lambda 10) but unfortunately I didn't include that model in the final ensemble (LB was the same and CV I didn't know because that was inference training).\n1. Tuning some models using Optuna.\n1. The inclusion of the XGB model was relatively important. I also had problems training this model, because (as I understand later) it is not so good at categorical features (unlike LGBM and CatBoost), and preprocessing was critical for this model. More specifically, after using TargetEncoder (and then CatBoostEncoder), the XGB score increased to good enough values. The same approach was also used for 1DCNN model.\n1. Using slightly different features for all the models (some tables, aggregators, types of columns, filtering approaches were not included for all the models).\n1. Replacing NaN in num columns with 0 helped for 1DCNN model.\n1. Aggregators used: max, min, first, last, mean, var, sum, std plus coef_var (distributed differently among all models, column types, or used only for certain columns).\n\n
\nWhat didn't work (at least in my experiments):\n1. Feature selection based on adversarial validation (within train ds), permutation importance (including stability metric).\n1. I tried to create some loss functions (based on functions simpler than initial metric - from topics about metrics; incorporating weights and so on) for GBDT models, but without improvements.\n1. CV strategies based on splitting train ds on week numbers using distributions of some statistics.\n1. Tuning GBDT models using some specific parameters like dart for LGBM, ordered boosting_type for CatBoost and so on.\n1. Using IsolationForest with SMOTENC\n1. Different scalers for 1DCNN model like PowerTransformer, QuantileTransformer. Finally I used Log1p transformation only for columns with max_min-mean ratio greater than 100.\n1. Replace nums NaN with mean (or constant values like -99). BTW LAMA pipeline uses this approach, and I tried to use 0s, but LB didn't improve and I rejected this idea, but after deadline I saw that private score was improved for that submission.\n1. Compress some categories (e.g. by template P0_111_121 to P0 or P0_111)\n1. Some non-standard, custom aggregators such as entropy, mean-to-max, zero frequency, normalization within group, and so on. The goal was also to find aggragators with relative, not absolute, values.\n1. Additional feature engineering like ratios between some columns didn't improve models. As I understand after first competition was created very good features and squeezing out from dataset additional gains was not so simple task.\n1. By the results of previous competition I tried Regularized Greedy Forest (FastRGF version) but it has very poor performance and worsened final ensemble. Another idea from that competition was to train MLP only on some group of tables, but again without any results, so I discarded this idea as well. I also experimented with other models like DAE, TabTransformer, NN-cat_embeddings, packages like pytorch_tabular, but also I can't train them to good enough results.\n1. Combining some categorical features (similar to how CatBoost works).\n1. Balancing classes using parameters like scale_pos_weight.\n1. Include 1DCNN in the final ensemble. I struggled a lot to overcome the overfitting of this model, but eventually it is worsening the final ensemble from 0.529 to 0.528. Some variations of this model (with another ds) has relatively good private score (0.485) but LB was not so good (only 0.555), so I didn't include the best and maybe was digging in the wrong direction. Besides, without metric hacking this model will not contribute much to the final ensemble/position.\n1. Include some additional data such as FedFunds.\n1. Filter some weeks.\n\n
\nOf course ideas from the community, discussions, public notebooks helped a lot.\n\n
\nUpdate. 29.05.2024\nJust out of curiosity, I tried adding the CatBoost model with score 0.520 to the final ensemble and the total private score increased to 0.536.\n\nUpdate. 5.06.2024\nFeature importance by gain I used in selecting top-20 features from credit_bureau_a tables for lama-mlp model. This helped to avoid overfitting this model.", "2840985": "What aggregates did you use for numeric columns?", "2841016": "I used pretty common aggregators: max, min, first, last, mean, var, sum, std plus coef_var. But again, each dataset has slightly different aggregators, for example, I didn't include var method to ds with coef_var. Sum aggregator I used only in some datasets and only for certain columns (with enough diversity - I selected them manually).", "2841041": "Thanks! Have you experimented with diff features, namely last-mean, last-first?", "2841062": "No, unfortunately.", "2841208": "Thank u for share . You mentioned \"Feature selection based on importance by gain\" didn't work. Did importance by split work?", "2841214": "Yes, I selected features for MLP based on importance by split of LGBM, but importance by gain in my cases led to overfitting.", "2841222": "Hi Andrey, congrats with the 62nd place! A lot of work is done!\n\nThere are a few things that helped me to boost my score quite a bit are the following:\n - Aggregation of numerical columns: last 3 rows tail(3).mean(); as well as diff() with mean and/or std\n - Many features created by hand using logical assumptions\n - The addition of stability metric had a minor effect\n - Max aggregations of string columns had a big effect. However, I rejected this due to a problem of interpretability with the hope that this is an artifact of the public LB test sample.\n\nJust curious, have you tried/considered such tail/diff aggregations to add to your data?\n\nI am still very puzzled to why the class balance does not work in this competition. When training my models, I even balanced the classes versus week number trying to exclude the possibility for models to learn any discrimination from this dimension ( as the WEEK_NUM is correlated with some inputs) with the hope this can enhance the model stability.", "2841270": "Hi Oleh. Thanks.\nI was thinking about some kind of tail(n).some_agg() (I also read mention about it in previous competition solutions and discussions), but I think this idea got lost somewhere in my notes. I didn't think about diff.\nAbout max agg for string columns - it may work because it contains some hierarchy in the some categories encoding (e.g. P0_111_121).", "2841285": "You gain a lot from ensembling. \nI have 0.598 single model and only 0.6 ensemble", "2841298": "Yes, that was a kind of exercise from my previous competitions. Of course, that's only one of the approaches, and a strong enough single model can beat a good ensemble.", "2841342": "Hi @andreynesterov, thank you for sharing. I read that DART did not work well, but in my case it has improved the result and reach almost the catboost performance.", "2841364": "That's great. I also thought dart was very promising, maybe I didn't pay much attention to tuning it or preparing features.", "2841461": "Thank you for the detailed insights on your findings. I have been following your work from the beginning of this competition and a lot of my work is inspired from your notebooks. Thank you for generously sharing those. Beginners like me learn a lot from your notebooks.\n\nI especially referred to the notebook where you ensembled xgb, cat, lgb model. I took the lessons from that notebook and tried ensembling 2 lgb and 2 cat - one of each with class balancing and other without the balancing and that did well. But, my best one which i did not submit was the one which i added a post-processing layer of manual implementation of isotonic calibration. it was a 0.593 on public LB and 0.527 on private LB.", "2841501": "That's interesting." }, "source": "meta" }