![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2099265%2F54e2ab7b4623137a7c2cbb00aaacf063%2FScreenshot%202024-02-18%20at%2010.19.08%20PM.png?generation=1708323569689791&alt=media\")
In this competition we have to deal with a highly imbalanced class distribution, as there are only about 3% of positive targets representing defaults:
\n| \n | [k] | \n[%] | \n
|---|---|---|
| Positive targets | \n48 | \n3 | \n
| Negative targets | \n1,479 | \n97 | \n
| Total cases | \n1,527 | \n100 | \n
After grouping data by WEEK_NUM, one can see that the default ratio doesn't really change much over the time, even though the total number of cases per week could change dramatically:
It means that when we stratify our folds by weeks, each of our models trains on 2-5% of positive targets and on 95-98% of negative ones. As a result, such an imbalance can affect model's ability to properly predict defaults (positive targets). Luckily, in this competition we mostly care about AUC, and not about actual probabilities, so can give a try to built-in class balancing features from popular gradient boosting libraries.
\nFor XGBClassifier we can use scale_pos_weight option, that scales the positive class weight. According to documentation, a typical value for this option is sum(negative targets) / sum(positive targets), i.e. in our case it is around 31. This value may need to be adjusted for each fold separately, as the default ratio varies over different time periods.
UPD: There is also a way to assign weight to each training sample separately. For that, one needs to convert training data to DMatrix object passing appropriate weight vector when creating it. Thanks to @thomasmeiner for pointing this out.
For LGBMClassifier there is a similar scale_pos_weight option to scale weight of positive class. In addition, there is a boolean flag is_unbalanced, that automatically assigns weights based on the ratio between the numbers of negative and positive classes. According to official documentation you can only use one of these options at a time, that makes sense as there is only one set of weights the model can use for training.
Just like the other libraries, CatBoostClassifier supports scale_pos_weight option. There is also a more advanced auto_class_weights feature, that can automatically adjust weights in Balanced and SqrtBalanced modes. You can use only one of these options to balance training.
Class weights, as all the other model parameters, may need to be tuned. In my experiments, setting scale_pos_weight to recommended typical values improved AUC by 0.01-0.02 for all the three libraries. The best LB improvement, however, was at the level of more humble 0.005 for LightGBM, but this is yet to be seen when the metric is fixed.
Besides adjusting class weights to deal with imbalance, one can also employ other techniques like down-/upsampling of data or postprocessing. For more information on this please refer to the following notebooks and discussions:
\nHope it helps and happy kaggling!
", "messages": [ { "id": "2658363", "postDate": "02/19/2024 06:30:28", "content": "In this competition we have to deal with a highly imbalanced class distribution, as there are only about 3% of positive targets representing defaults:
\n| \n | [k] | \n[%] | \n
|---|---|---|
| Positive targets | \n48 | \n3 | \n
| Negative targets | \n1,479 | \n97 | \n
| Total cases | \n1,527 | \n100 | \n
After grouping data by WEEK_NUM, one can see that the default ratio doesn't really change much over the time, even though the total number of cases per week could change dramatically:
It means that when we stratify our folds by weeks, each of our models trains on 2-5% of positive targets and on 95-98% of negative ones. As a result, such an imbalance can affect model's ability to properly predict defaults (positive targets). Luckily, in this competition we mostly care about AUC, and not about actual probabilities, so can give a try to built-in class balancing features from popular gradient boosting libraries.
\nFor XGBClassifier we can use scale_pos_weight option, that scales the positive class weight. According to documentation, a typical value for this option is sum(negative targets) / sum(positive targets), i.e. in our case it is around 31. This value may need to be adjusted for each fold separately, as the default ratio varies over different time periods.
UPD: There is also a way to assign weight to each training sample separately. For that, one needs to convert training data to DMatrix object passing appropriate weight vector when creating it. Thanks to @thomasmeiner for pointing this out.
For LGBMClassifier there is a similar scale_pos_weight option to scale weight of positive class. In addition, there is a boolean flag is_unbalanced, that automatically assigns weights based on the ratio between the numbers of negative and positive classes. According to official documentation you can only use one of these options at a time, that makes sense as there is only one set of weights the model can use for training.
Just like the other libraries, CatBoostClassifier supports scale_pos_weight option. There is also a more advanced auto_class_weights feature, that can automatically adjust weights in Balanced and SqrtBalanced modes. You can use only one of these options to balance training.
Class weights, as all the other model parameters, may need to be tuned. In my experiments, setting scale_pos_weight to recommended typical values improved AUC by 0.01-0.02 for all the three libraries. The best LB improvement, however, was at the level of more humble 0.005 for LightGBM, but this is yet to be seen when the metric is fixed.
Besides adjusting class weights to deal with imbalance, one can also employ other techniques like down-/upsampling of data or postprocessing. For more information on this please refer to the following notebooks and discussions:
\nHope it helps and happy kaggling!
", "rawMarkdown": "In this competition we have to deal with a highly imbalanced class distribution, as there are only about 3% of positive targets representing defaults:\n\n| | [k] | [%] |\n|------------------|--------|-----|\n| Positive targets | 48 | 3 |\n| Negative targets | 1,479 | 97 |\n| Total cases | 1,527 | 100 |\n\nThanks, that's interesting insights. Just curious, will results be similar if data is grouped not by weeks but months? Another thought is about whether ensembling different methods (e.g., XGBoost and Catboost) can imporove performance somehow.
", "rawMarkdown": "Thanks, that's interesting insights. Just curious, will results be similar if data is grouped not by weeks but months? Another thought is about whether ensembling different methods (e.g., XGBoost and Catboost) can imporove performance somehow.", "votes": null }, { "id": "2665768", "postDate": "02/23/2024 20:22:10", "content": "I guess grouping by months should result in a similar performance, but it is good to double check that on real data;
Combining different models is almost always beneficial, when you ensemble diverse models. I didn’t work on ensembling in this competition yet, mostly trying to build a set of good single models first.
What auc roc did you get after using scale_pos_weight? My score is about 0.84 in cv, but weight scaling does not work so well in my case. @kononenko
", "rawMarkdown": "What auc roc did you get after using scale_pos_weight? My score is about 0.84 in cv, but weight scaling does not work so well in my case. @kononenko", "votes": null }, { "id": "2674465", "postDate": "02/29/2024 10:51:08", "content": "Check out the first paragraph in the Conclusions section. I didn’t get better than that, but I only tested it for depth 1 features. From your CV it seems you are already using depth 2, right?
", "rawMarkdown": "Check out the first paragraph in the Conclusions section. I didn’t get better than that, but I only tested it for depth 1 features. From your CV it seems you are already using depth 2, right?", "votes": null }, { "id": "2674483", "postDate": "02/29/2024 11:00:29", "content": "Yes, I applied depth 2.
", "rawMarkdown": "Yes, I applied depth 2.", "votes": null }, { "id": "2674496", "postDate": "02/29/2024 11:07:54", "content": "Right, so for depth 1 it roughly changed AUC from 0.79-0.8 to 0.81-0.82 depending on a model.
", "rawMarkdown": "Right, so for depth 1 it roughly changed AUC from 0.79-0.8 to 0.81-0.82 depending on a model.", "votes": null } ], "comments": [ { "id": 2665271, "author_name": "ablaydosmaganbetov", "author_url": "", "post_date": "02/23/2024 15:13:12", "content": "Thanks, that's interesting insights. Just curious, will results be similar if data is grouped not by weeks but months? Another thought is about whether ensembling different methods (e.g., XGBoost and Catboost) can imporove performance somehow.
", "votes": null, "replies": [ { "id": 2665768, "author_name": "kononenko", "author_url": "", "post_date": "02/23/2024 20:22:10", "content": "I guess grouping by months should result in a similar performance, but it is good to double check that on real data;
Combining different models is almost always beneficial, when you ensemble diverse models. I didn’t work on ensembling in this competition yet, mostly trying to build a set of good single models first.
What auc roc did you get after using scale_pos_weight? My score is about 0.84 in cv, but weight scaling does not work so well in my case. @kononenko
", "votes": null, "replies": [ { "id": 2674465, "author_name": "kononenko", "author_url": "", "post_date": "02/29/2024 10:51:08", "content": "Check out the first paragraph in the Conclusions section. I didn’t get better than that, but I only tested it for depth 1 features. From your CV it seems you are already using depth 2, right?
", "votes": null, "replies": [ { "id": 2674483, "author_name": "jankowalski2000", "author_url": "", "post_date": "02/29/2024 11:00:29", "content": "Yes, I applied depth 2.
", "votes": null, "replies": [ { "id": 2674496, "author_name": "kononenko", "author_url": "", "post_date": "02/29/2024 11:07:54", "content": "Right, so for depth 1 it roughly changed AUC from 0.79-0.8 to 0.81-0.82 depending on a model.
", "votes": null, "replies": [] } ] } ] } ] } ], "raw_markdown_by_id": { "2658363": "In this competition we have to deal with a highly imbalanced class distribution, as there are only about 3% of positive targets representing defaults:\n\n| | [k] | [%] |\n|------------------|--------|-----|\n| Positive targets | 48 | 3 |\n| Negative targets | 1,479 | 97 |\n| Total cases | 1,527 | 100 |\n\n