First of all, thanks to Kaggle and the host of the competition. To be honest, we don't fully understand what was effective, let me briefly describe our solution.
\nThanks @kentookumura @pegasus27 for the collaboration.
\nBusiness context: https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/overview
\nData context: https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/data
\nIn feature engineering, we created 772 handcrafted features using tables other than credit_bureau_b1, other_1, deposit_1, debitcard_1. We performed manual and correlation-based feature selection and reduced the number of features to 411.
\nIn modeling, we built 10 models using LightGBM, XGBoost, CatBoost, and HistGradientBoostingClassifier. I stacked these outputs with a RidgeClassifier and applied probability calibration to create predicted values. Then, we created the final predicted values using random seed averaging.
\nIn postprocess, we added a simple process to take the maximum value of pmts_year_1139T in the credit_bureau_a_2 table and apply a negative correction to the score for each year.
\nThe feature engineering was mainly done by @kentookumura. Initially, we decided not to use the credit_bureau_b_2, other_1, deposit_1, debitcard_1 tables which had a high rate of missing case_id. We created handcrafted features based on the results of EDA and the solutions from past competitions. We reduced the number of features from 772 to 411 through manual and correlation-based feature selection. Below, we will describe the points that we believe were effective.
\nEra
\ndf_base = df_base.with_columns(\n ((pl.col() / ).floor() * ).alias().cast(pl.Int32),\n)\nAge at Start of Employment
\ndf = df.with_columns(\n ((pl.col() - pl.col()).dt.total_days() // ).cast(pl.Int32).alias(), \n)\nEmployment Period
\ndf_base = df_base.with_columns(\n (pl.col() - pl.col()).alias(),\n)\nDate processing other than suffix D
\n ():\n colin df.columns:\n col[-] (,):\n df = df.with_columns(pl.col(col) - pl.col())\n df = df.with_columns(pl.col(col).dt.total_days())\n df = df.with_columns(pl.col(col).cast(pl.Float32))\n\n col:\n df = df.with_columns(pl.col(col) - pl.col().dt.year())\n df = df.with_columns(pl.col(col).cast(pl.Int32))\nMerge tax_registry tables
\nFrom some case_id with multiple provider information, we inferred the correspondence of each table column and made it into one table.
\nAggregation of String type (mode and n_unique)
\nWe used the process pl.col(col).drop_nans().drop_nulls().mode().sort().first() for reproducibility in polars.
Removal of features that fluctuate greatly during the training data period
\nWe manually checked and removed features that fluctuate greatly with each WEEK_NUM.
\n@uplus26e7 mainly handled the modeling. We used StratifiedGroupKFold(k=5) based on WEEK_NUM for CV. We tried multiple GBDT models that do not require scaling of features or missing value completion, as these did not go well. In order to create diverse models, we created 10 models with multiple parameters and stacked them with RidgeClassifier. Finally, we corrected the predicted values using Scikit-Learn's CalibratedClassifierCV.
\nThe above model performed random seed averaging (5 seeds) and used it for the final inference.
\n| Model | \nLocal CV AUC (average 5 seeds) | \nMain Parameters | \n
|---|---|---|
| XGBoost | \n0.8569388045 | \n\n |
| CatBoost | \n0.8543810988 | \n\n |
| LightGBM | \n0.8576141787 | \nboosting=”gbdt”, extra_tree=True | \n
| LightGBM | \n0.8569003931 | \nboosting=”gbdt” | \n
| LightGBM | \n0.8068982316 | \nboosting=”rf” | \n
| LightGBM | \n0.7993733596 | \nboosting=”rf”, extra_tree=True | \n
| LightGBM | \n0.8546620954 | \nboosting=”dart” | \n
| LightGBM | \n0.8517172791 | \nboosting=”dart”, extra_tree=True | \n
| HistGradientBoostingClassifier | \n0.8496922895 | \n\n |
| LightGBM | \n0.8574351195 | \nboosting=”gbdt”, extra_tree=True, data_sample_strategy=”goss” | \n
| CalibratedClassifierCV (RidgeClassifier) | \n0.859322 | \n\n |
@kentookumura's thorough EDA and experiments revealed that the 'pmts_year_1139T' in the 'credit_bureau_a_2' table is likely the most recent 'date_decision' year. It was also observed that no date column transformations were added in data changes. Based on these findings, we implemented post-processing to decrease the predicted value based on the maximum 'pmts_year_1139T' value.
\nsubmission = pd.read_csv()\npmts_year = ... \n submission.loc[pmts_year == , ] = (submission.loc[pmts_year == , ] - ).clip()\n submission.loc[pmts_year == , ] = (submission.loc[pmts_year == , ] - ).clip()\n submission.loc[pmts_year == , ] = (submission.loc[pmts_year==, ] - ).clip()\n submission.to_csv(, index=)\nFirst of all, thanks to Kaggle and the host of the competition. To be honest, we don't fully understand what was effective, let me briefly describe our solution.
\nThanks @kentookumura @pegasus27 for the collaboration.
\nBusiness context: https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/overview
\nData context: https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/data
\nIn feature engineering, we created 772 handcrafted features using tables other than credit_bureau_b1, other_1, deposit_1, debitcard_1. We performed manual and correlation-based feature selection and reduced the number of features to 411.
\nIn modeling, we built 10 models using LightGBM, XGBoost, CatBoost, and HistGradientBoostingClassifier. I stacked these outputs with a RidgeClassifier and applied probability calibration to create predicted values. Then, we created the final predicted values using random seed averaging.
\nIn postprocess, we added a simple process to take the maximum value of pmts_year_1139T in the credit_bureau_a_2 table and apply a negative correction to the score for each year.
\nThe feature engineering was mainly done by @kentookumura. Initially, we decided not to use the credit_bureau_b_2, other_1, deposit_1, debitcard_1 tables which had a high rate of missing case_id. We created handcrafted features based on the results of EDA and the solutions from past competitions. We reduced the number of features from 772 to 411 through manual and correlation-based feature selection. Below, we will describe the points that we believe were effective.
\nEra
\ndf_base = df_base.with_columns(\n ((pl.col() / ).floor() * ).alias().cast(pl.Int32),\n)\nAge at Start of Employment
\ndf = df.with_columns(\n ((pl.col() - pl.col()).dt.total_days() // ).cast(pl.Int32).alias(), \n)\nEmployment Period
\ndf_base = df_base.with_columns(\n (pl.col() - pl.col()).alias(),\n)\nDate processing other than suffix D
\n ():\n colin df.columns:\n col[-] (,):\n df = df.with_columns(pl.col(col) - pl.col())\n df = df.with_columns(pl.col(col).dt.total_days())\n df = df.with_columns(pl.col(col).cast(pl.Float32))\n\n col:\n df = df.with_columns(pl.col(col) - pl.col().dt.year())\n df = df.with_columns(pl.col(col).cast(pl.Int32))\nMerge tax_registry tables
\nFrom some case_id with multiple provider information, we inferred the correspondence of each table column and made it into one table.
\nAggregation of String type (mode and n_unique)
\nWe used the process pl.col(col).drop_nans().drop_nulls().mode().sort().first() for reproducibility in polars.
Removal of features that fluctuate greatly during the training data period
\nWe manually checked and removed features that fluctuate greatly with each WEEK_NUM.
\n@uplus26e7 mainly handled the modeling. We used StratifiedGroupKFold(k=5) based on WEEK_NUM for CV. We tried multiple GBDT models that do not require scaling of features or missing value completion, as these did not go well. In order to create diverse models, we created 10 models with multiple parameters and stacked them with RidgeClassifier. Finally, we corrected the predicted values using Scikit-Learn's CalibratedClassifierCV.
\nThe above model performed random seed averaging (5 seeds) and used it for the final inference.
\n| Model | \nLocal CV AUC (average 5 seeds) | \nMain Parameters | \n
|---|---|---|
| XGBoost | \n0.8569388045 | \n\n |
| CatBoost | \n0.8543810988 | \n\n |
| LightGBM | \n0.8576141787 | \nboosting=”gbdt”, extra_tree=True | \n
| LightGBM | \n0.8569003931 | \nboosting=”gbdt” | \n
| LightGBM | \n0.8068982316 | \nboosting=”rf” | \n
| LightGBM | \n0.7993733596 | \nboosting=”rf”, extra_tree=True | \n
| LightGBM | \n0.8546620954 | \nboosting=”dart” | \n
| LightGBM | \n0.8517172791 | \nboosting=”dart”, extra_tree=True | \n
| HistGradientBoostingClassifier | \n0.8496922895 | \n\n |
| LightGBM | \n0.8574351195 | \nboosting=”gbdt”, extra_tree=True, data_sample_strategy=”goss” | \n
| CalibratedClassifierCV (RidgeClassifier) | \n0.859322 | \n\n |
@kentookumura's thorough EDA and experiments revealed that the 'pmts_year_1139T' in the 'credit_bureau_a_2' table is likely the most recent 'date_decision' year. It was also observed that no date column transformations were added in data changes. Based on these findings, we implemented post-processing to decrease the predicted value based on the maximum 'pmts_year_1139T' value.
\nsubmission = pd.read_csv()\npmts_year = ... \n submission.loc[pmts_year == , ] = (submission.loc[pmts_year == , ] - ).clip()\n submission.loc[pmts_year == , ] = (submission.loc[pmts_year == , ] - ).clip()\n submission.loc[pmts_year == , ] = (submission.loc[pmts_year==, ] - ).clip()\n submission.to_csv(, index=)\nNice writeup! However, I have some questions:
\nThank you for your comment.
\n@uplus26e7
\nHow did you implement pseudo-labels? As OOF predictions on the validation or train set?
In each fold, we used the predictions of the base model to the validation data as targets.
", "rawMarkdown": "In each fold, we used the predictions of the base model to the validation data as targets.", "votes": null }, { "id": "2841344", "postDate": "05/28/2024 14:37:14", "content": "Glad to see you have used Calibrated results. My mistake was that I did not select the calibrated response for the final submission, else it would have boosted me up by at least 50. I learned a good lesson in this first competition - to trust my instincts. I manually implemented the isotonic calibration on the final results of my LGB and CatBoost ensemble and it gave me a 0.09 point boost on the private LB but did not select it as final submission because it did not give me any boost in public LB.
", "rawMarkdown": "Glad to see you have used Calibrated results. My mistake was that I did not select the calibrated response for the final submission, else it would have boosted me up by at least 50. I learned a good lesson in this first competition - to trust my instincts. I manually implemented the isotonic calibration on the final results of my LGB and CatBoost ensemble and it gave me a 0.09 point boost on the private LB but did not select it as final submission because it did not give me any boost in public LB.", "votes": null }, { "id": "2841741", "postDate": "05/28/2024 17:15:21", "content": "Thank you for your sharing. I learned a lot from your post.
\nHowever, apart from using pseudo-labeling, are there any techniques I can use?
In my opinion, this competition is not suitable for learning materials because the postprocess effects are too big.
", "rawMarkdown": "In my opinion, this competition is not suitable for learning materials because the postprocess effects are too big.", "votes": null }, { "id": "2844419", "postDate": "05/30/2024 04:25:25", "content": "Congratulations on securing 14th place in this competition. Thanks for sharing details of your solution.
", "rawMarkdown": "Congratulations on securing 14th place in this competition. Thanks for sharing details of your solution.", "votes": null }, { "id": "2885338", "postDate": "06/23/2024 03:55:04", "content": "Thank you for your sharing，
\nhowever i have one question： in Postprocess stage,how do you determine the decrease value
\nThank you for your response in advance.
Nice writeup! However, I have some questions:
\nThank you for your comment.
\n@uplus26e7
\nHow did you implement pseudo-labels? As OOF predictions on the validation or train set?
In each fold, we used the predictions of the base model to the validation data as targets.
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2841344, "author_name": "varuniraothumsi", "author_url": "", "post_date": "05/28/2024 14:37:14", "content": "Glad to see you have used Calibrated results. My mistake was that I did not select the calibrated response for the final submission, else it would have boosted me up by at least 50. I learned a good lesson in this first competition - to trust my instincts. I manually implemented the isotonic calibration on the final results of my LGB and CatBoost ensemble and it gave me a 0.09 point boost on the private LB but did not select it as final submission because it did not give me any boost in public LB.
", "votes": null, "replies": [] }, { "id": 2841741, "author_name": "nicholasyong216", "author_url": "", "post_date": "05/28/2024 17:15:21", "content": "Thank you for your sharing. I learned a lot from your post.
\nHowever, apart from using pseudo-labeling, are there any techniques I can use?
In my opinion, this competition is not suitable for learning materials because the postprocess effects are too big.
", "votes": null, "replies": [] } ] }, { "id": 2844419, "author_name": "crsuthikshnkumar", "author_url": "", "post_date": "05/30/2024 04:25:25", "content": "Congratulations on securing 14th place in this competition. Thanks for sharing details of your solution.
", "votes": null, "replies": [] }, { "id": 2885338, "author_name": "cauchemare", "author_url": "", "post_date": "06/23/2024 03:55:04", "content": "Thank you for your sharing，
\nhowever i have one question： in Postprocess stage,how do you determine the decrease value
\nThank you for your response in advance.