![\"image_\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6960421%2F92721e389d44b29d04f931d244fed316%2F.png?generation=1716967303955938&alt=media\")
Hello, this is yuuniee.
\nThank you to the competition host Home Credit and staff, Kaggle staff and everyone involved.
The image below is a summary of my solution.
\n
The maximum Private LB Score for my model pool without Metric Hacking is close to about 0.53.
\nAs everyone knows, this competition takes place in two phases.
\nPhase 1 was ML (Machine Learning), and Phase 2 was MH (Metric Hack).
\nI will briefly introduce each part.
\n(If you are only curious about ML, please read only Phase 1.)
Thank you for writing a great notebook for EDA and solution construction at the beginning of the competition.
\n@sergiosaharovskiy - https://www.kaggle.com/code/sergiosaharovskiy/home-credit-crms-2024-eda-and-submission
\n@greysky - https://www.kaggle.com/code/greysky/home-credit-baseline
\nAdditionally, I would like to thank the many participants who provided various insights.
The CV strategy is StratifiedGroupKFold, and has been tested and trained in various forms: No Shuffle and Shuffle. In my case, differences in CV of around 0.001 to 0.005 had a low correlation with LB, while differences above 0.01 had a high correlation with LB. In addition, CV improvement through model parameter adjustment had a relatively low correlation with LB, and CV improvement through FE had a relatively high correlation with LB.
There's nothing special about Feature Engineering. Max, Min, Avg, Var, First, Last and Max-Min Difference of each item were utilized. (I will release the code after cleaning it up.)
LGBM was a bit lacking compared to Catboost, but was good for the ensemble. The reason why the number of features is smaller is because some items (mainly categorical types) were excluded because they caused performance degradation and overfitting in LGBM, and other slightly different types of features were added and deleted.
DNN is a Denselight model and utilizes the LightAutoML library. There were some bugs, so I had to fix some of them myself, but overall, I think it is a good library. At first, I planned to lightly test with Denselight and then build the final model with a larger model (FT-Transformer, etc.). Surprisingly, I haven't created or found a model that beats Denselight performance. Perhaps I made a mistake, or the nature of the data is sensitive to overfitting, so I guess a simpler model worked better.
Catboost was the model that performed best in this competition and I guess it performed well on a large number of categorical features (about 117).
During this period, my public LB was close to top 5 and the competition seemed to be going very smoothly.
\nBut…
\n\n+ What didn't work
\n\n
\n- Transformer type models such as Tabnet, TabTransformer, FT-Transformer, etc.
\n- Income, expenditure, and tax statistics by period (month, week, etc.).
\n- Differences between individual income, expenses, and taxes.
\n- K-means clustering
\n
Pointed out problems with MH at the beginning of the competition,
\n@at7459 - https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/476449
\nAnnounced the revival of MH in the latter half of the competition,
\n@johnpateha - https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/497167
Also, thanks to the many participants who suggested various other comments and solutions. Thanks to you, my knowledge has expanded.
\nIn particular, the two people above are true scientists and journalists who could have hidden this discovery and profited from it, but pursued the public interest by exposing it to everyone. (Why not consider Kaggle giving a special award for this sharing?)
\nAs a result, the problem was not solved, but we learned something(?) from it and at least I was able to take minimal measures.
After @johnpateha's \"Metric hack again, sorry\" post, I tried Data Reversing and found that the difference between date_decision and min_refreshdate_3813885D had a high correlation (about 0.9 or more) with WEEK_NUM.
\n(More information can be found at https://www.kaggle.com/code/eivolkova/how-to-restore-the-dates?scriptVersionId=180157891, published by @eivolkova immediately after the competition.)
\nAnd the way suggested by @at7459:
DEVIDE = /\nREDUCE = \ncondition = df[] < (df[].()-df[].())*DEVIDE+df[].()\ndf.loc[condition, ] = (df.loc[condition, ] - REDUCE).clip()\nBased on this, the results of examining the score details submitted by changing the REDUCE and DEVIDE values were as follows.
\nThe difference in pure model performance between top competitors was 0.00X,
\nSince the difference due to the above adjustment value was 0.0X,
\nAt this stage it became gambling for me.
Some will bet low, others will bet high.
\nHere, I trusted my model and chose neutral, and chose DEVIDE=1/2, REDUCE=0.03, which is the median of the expected distribution.
\nI expected someone who bet high or low to take the prize, and I expected about a 50% chance of finishing in the top 10.
The first selected submission is the No Hack model with CV Best Score created in Phase 1, and the second selected submission is the application of Metric Hack to it. (I thought there was a high probability that Metric Hack would work, but I also submitted No Hack just in case)
Rather than simply sharing solutions, I thought it would be better to share my experience in detail. To improve the bad parts that happened in this competition.
My ranking was due to luck, and although it is the Best Private Score submitted, it probably won't be Best Solution.
Although the competition ended on a gloomy note due to the metric hacking issue, the host and staff worked hard to make it a meaningful and successful competition.
Based on this, we expect a more developed system in the future.
thank you
\n\n", "messages": [ { "id": "2842729", "postDate": "05/29/2024 08:24:57", "content": "Edit : I posted the FE code here.
\n
\n 644(for lgbm) - https://www.kaggle.com/code/yuuniekiri/fork-of-home-credit-risk-lightgbm
\n 661(for cat & dnn) - https://www.kaggle.com/code/yuuniekiri/fork-of-home-credit-catboost-inference
Hello, this is yuuniee.
\nThank you to the competition host Home Credit and staff, Kaggle staff and everyone involved.
The image below is a summary of my solution.
\n
The maximum Private LB Score for my model pool without Metric Hacking is close to about 0.53.
\nAs everyone knows, this competition takes place in two phases.
\nPhase 1 was ML (Machine Learning), and Phase 2 was MH (Metric Hack).
\nI will briefly introduce each part.
\n(If you are only curious about ML, please read only Phase 1.)
Thank you for writing a great notebook for EDA and solution construction at the beginning of the competition.
\n@sergiosaharovskiy - https://www.kaggle.com/code/sergiosaharovskiy/home-credit-crms-2024-eda-and-submission
\n@greysky - https://www.kaggle.com/code/greysky/home-credit-baseline
\nAdditionally, I would like to thank the many participants who provided various insights.
The CV strategy is StratifiedGroupKFold, and has been tested and trained in various forms: No Shuffle and Shuffle. In my case, differences in CV of around 0.001 to 0.005 had a low correlation with LB, while differences above 0.01 had a high correlation with LB. In addition, CV improvement through model parameter adjustment had a relatively low correlation with LB, and CV improvement through FE had a relatively high correlation with LB.
There's nothing special about Feature Engineering. Max, Min, Avg, Var, First, Last and Max-Min Difference of each item were utilized. (I will release the code after cleaning it up.)
LGBM was a bit lacking compared to Catboost, but was good for the ensemble. The reason why the number of features is smaller is because some items (mainly categorical types) were excluded because they caused performance degradation and overfitting in LGBM, and other slightly different types of features were added and deleted.
DNN is a Denselight model and utilizes the LightAutoML library. There were some bugs, so I had to fix some of them myself, but overall, I think it is a good library. At first, I planned to lightly test with Denselight and then build the final model with a larger model (FT-Transformer, etc.). Surprisingly, I haven't created or found a model that beats Denselight performance. Perhaps I made a mistake, or the nature of the data is sensitive to overfitting, so I guess a simpler model worked better.
Catboost was the model that performed best in this competition and I guess it performed well on a large number of categorical features (about 117).
During this period, my public LB was close to top 5 and the competition seemed to be going very smoothly.
\nBut…
\n\n+ What didn't work
\n\n
\n- Transformer type models such as Tabnet, TabTransformer, FT-Transformer, etc.
\n- Income, expenditure, and tax statistics by period (month, week, etc.).
\n- Differences between individual income, expenses, and taxes.
\n- K-means clustering
\n
Pointed out problems with MH at the beginning of the competition,
\n@at7459 - https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/476449
\nAnnounced the revival of MH in the latter half of the competition,
\n@johnpateha - https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/497167
Also, thanks to the many participants who suggested various other comments and solutions. Thanks to you, my knowledge has expanded.
\nIn particular, the two people above are true scientists and journalists who could have hidden this discovery and profited from it, but pursued the public interest by exposing it to everyone. (Why not consider Kaggle giving a special award for this sharing?)
\nAs a result, the problem was not solved, but we learned something(?) from it and at least I was able to take minimal measures.
After @johnpateha's \"Metric hack again, sorry\" post, I tried Data Reversing and found that the difference between date_decision and min_refreshdate_3813885D had a high correlation (about 0.9 or more) with WEEK_NUM.
\n(More information can be found at https://www.kaggle.com/code/eivolkova/how-to-restore-the-dates?scriptVersionId=180157891, published by @eivolkova immediately after the competition.)
\nAnd the way suggested by @at7459:
DEVIDE = /\nREDUCE = \ncondition = df[] < (df[].()-df[].())*DEVIDE+df[].()\ndf.loc[condition, ] = (df.loc[condition, ] - REDUCE).clip()\nBased on this, the results of examining the score details submitted by changing the REDUCE and DEVIDE values were as follows.
\nThe difference in pure model performance between top competitors was 0.00X,
\nSince the difference due to the above adjustment value was 0.0X,
\nAt this stage it became gambling for me.
Some will bet low, others will bet high.
\nHere, I trusted my model and chose neutral, and chose DEVIDE=1/2, REDUCE=0.03, which is the median of the expected distribution.
\nI expected someone who bet high or low to take the prize, and I expected about a 50% chance of finishing in the top 10.
The first selected submission is the No Hack model with CV Best Score created in Phase 1, and the second selected submission is the application of Metric Hack to it. (I thought there was a high probability that Metric Hack would work, but I also submitted No Hack just in case)
Rather than simply sharing solutions, I thought it would be better to share my experience in detail. To improve the bad parts that happened in this competition.
My ranking was due to luck, and although it is the Best Private Score submitted, it probably won't be Best Solution.
Although the competition ended on a gloomy note due to the metric hacking issue, the host and staff worked hard to make it a meaningful and successful competition.
Based on this, we expect a more developed system in the future.
thank you
\n\n", "rawMarkdown": "##Foreword\n\nHello, this is yuuniee.\nThank you to the competition host Home Credit and staff, Kaggle staff and everyone involved.\n\nThe image below is a summary of my solution.\n\n\n\nThe maximum Private LB Score for my model pool without Metric Hacking is close to about 0.53.\n\nAs everyone knows, this competition takes place in two phases.\nPhase 1 was ML (Machine Learning), and Phase 2 was MH (Metric Hack).\nI will briefly introduce each part.\n(If you are only curious about ML, please read only Phase 1.)\n\n\n## I. Phase 1 - ML (Machine Learning)\n\nThank you for writing a great notebook for EDA and solution construction at the beginning of the competition.\n@sergiosaharovskiy - https://www.kaggle.com/code/sergiosaharovskiy/home-credit-crms-2024-eda-and-submission\n@greysky - https://www.kaggle.com/code/greysky/home-credit-baseline\nAdditionally, I would like to thank the many participants who provided various insights.\n\n1. The CV strategy is StratifiedGroupKFold, and has been tested and trained in various forms: No Shuffle and Shuffle. In my case, differences in CV of around 0.001 to 0.005 had a low correlation with LB, while differences above 0.01 had a high correlation with LB. In addition, CV improvement through model parameter adjustment had a relatively low correlation with LB, and CV improvement through FE had a relatively high correlation with LB.\n\n2. There's nothing special about Feature Engineering. Max, Min, Avg, Var, First, Last and Max-Min Difference of each item were utilized. (I will release the code after cleaning it up.)\n\n3. LGBM was a bit lacking compared to Catboost, but was good for the ensemble. The reason why the number of features is smaller is because some items (mainly categorical types) were excluded because they caused performance degradation and overfitting in LGBM, and other slightly different types of features were added and deleted.\n\n4. DNN is a Denselight model and utilizes the LightAutoML library. There were some bugs, so I had to fix some of them myself, but overall, I think it is a good library. At first, I planned to lightly test with Denselight and then build the final model with a larger model (FT-Transformer, etc.). Surprisingly, I haven't created or found a model that beats Denselight performance. Perhaps I made a mistake, or the nature of the data is sensitive to overfitting, so I guess a simpler model worked better.\n\n5. Catboost was the model that performed best in this competition and I guess it performed well on a large number of categorical features (about 117).\n\nDuring this period, my public LB was close to top 5 and the competition seemed to be going very smoothly.\nBut...\n\n\n>### + What didn't work\n>1. Transformer type models such as Tabnet, TabTransformer, FT-Transformer, etc.\n>2. Income, expenditure, and tax statistics by period (month, week, etc.).\n>3. Differences between individual income, expenses, and taxes.\n>4. K-means clustering\n\n\n## II. Phase 2 - MH(Metric Hack)\n\nPointed out problems with MH at the beginning of the competition,\n@at7459 - https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/476449\nAnnounced the revival of MH in the latter half of the competition,\n@johnpateha - https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/497167\n\nAlso, thanks to the many participants who suggested various other comments and solutions. Thanks to you, my knowledge has expanded.\nIn particular, the two people above are true scientists and journalists who could have hidden this discovery and profited from it, but pursued the public interest by exposing it to everyone. (Why not consider Kaggle giving a special award for this sharing?)\nAs a result, the problem was not solved, but we learned something(?) from it and at least I was able to take minimal measures.\n\nAfter @johnpateha's \"Metric hack again, sorry\" post, I tried Data Reversing and found that the difference between date_decision and min_refreshdate_3813885D had a high correlation (about 0.9 or more) with WEEK_NUM.\n(More information can be found at https://www.kaggle.com/code/eivolkova/how-to-restore-the-dates?scriptVersionId=180157891, published by @eivolkova immediately after the competition.)\nAnd the way suggested by @at7459:\n\n```python\nDEVIDE = 1/2\nREDUCE = 0.02\ncondition = df['WEEK_NUM'] < (df['WEEK_NUM'].max()-df['WEEK_NUM'].min())*DEVIDE+df['WEEK_NUM'].min()\ndf.loc[condition, 'score'] = (df.loc[condition, 'score'] - REDUCE).clip(0)\n```\n\nBased on this, the results of examining the score details submitted by changing the REDUCE and DEVIDE values were as follows.\n\n1. The public/private division of test data spans the entire period and it is difficult to identify a specific distribution, and the optimal value in private will probably be distributed around 1/4 to 3/4.\n2. The optimal public LB value of REDUCE is 0.03 (at least in my model), and the optimal value in private is probably distributed around 0.02 to 0.04.\n\nThe difference in pure model performance between top competitors was 0.00X,\nSince the difference due to the above adjustment value was 0.0X,\nAt this stage it became gambling for me.\n\nSome will bet low, others will bet high.\nHere, I trusted my model and chose neutral, and chose DEVIDE=1/2, REDUCE=0.03, which is the median of the expected distribution.\nI expected someone who bet high or low to take the prize, and I expected about a 50% chance of finishing in the top 10.\n\n\n## III. Conclusion\n\n+ The first selected submission is the No Hack model with CV Best Score created in Phase 1, and the second selected submission is the application of Metric Hack to it. (I thought there was a high probability that Metric Hack would work, but I also submitted No Hack just in case)\n\n+ Rather than simply sharing solutions, I thought it would be better to share my experience in detail. To improve the bad parts that happened in this competition.\n\n+ My ranking was due to luck, and although it is the Best Private Score submitted, it probably won't be Best Solution.\n\n+ Although the competition ended on a gloomy note due to the metric hacking issue, the host and staff worked hard to make it a meaningful and successful competition. \n\n+ Based on this, we expect a more developed system in the future.\n\n\nthank you\n\n\n>Edit : I posted the FE code here.\n>644(for lgbm) - https://www.kaggle.com/code/yuuniekiri/fork-of-home-credit-risk-lightgbm\n>661(for cat & dnn) - https://www.kaggle.com/code/yuuniekiri/fork-of-home-credit-catboost-inference", "votes": null }, { "id": "2842917", "postDate": "05/29/2024 10:37:11", "content": "Edit : I posted the FE code here.
\n
\n 644(for lgbm) - https://www.kaggle.com/code/yuuniekiri/fork-of-home-credit-risk-lightgbm
\n 661(for cat & dnn) - https://www.kaggle.com/code/yuuniekiri/fork-of-home-credit-catboost-inference
Congratulations on the placement @yuuniekiri , how did you choose the features for each model?
\nIf possible, share your notebook with us.
Thank you, @thiagomantuani. A notebook link has been added.
", "rawMarkdown": "Thank you, @thiagomantuani. A notebook link has been added.", "votes": null }, { "id": "2843263", "postDate": "05/29/2024 13:35:53", "content": "Congratulations on your second #1 rank competition @yuuniekiri! Even though I didn't participate in the competition, I was following along since the beginning after my initial post some months ago. After reading your straightforward write-up and both notebooks, I totally agree with your choice of models, such as CatBoost, LGBM, and DNN (eventually NaiveBayes would've helped a bit). My only question that I have left is, did you try to use rankdata() after making the predictions? That gave me a huge boost in my first competition with almost exactly the same ensemble models. I'm just curious, if you don't mind, whenever you have time, could you please try a \"Late submission\" with the following updated part of your \"Fork of Home Credit Risk (LightGBM)\" at the very end:
Edit after @yuuniekiri comment:
\n scipy.stats rankdata \n\n reader\ngc.collect()\n\nranked_preds = rankdata(y_pred) / (y_pred) \n\ndf_subm = pd.read_csv()\ndf_subm = df_subm.set_index()\ndf_subm[] = ranked_preds \ndf_subm.to_csv()\n(df_subm)\nThanks a lot for your try-out and response. Enjoy your preliminary win!
\nBest regards,
\nEtienne
Thank you for your congratulations.
\nHowever, the submissions resulting from the code above are different from the competition format.
\n(Only two columns are allowed: case_id and score.)
\nIt will probably cause an error.
You're welcome. That's right, I had a logical mistake. We don't need an additional column, I updated my code above (Edit). I looked into my old competition and that's the way how I made it there. Now the predictions should be averaged ranked. Give it a shot, if it doesn't work it's also fine - just for quick testing purpose :) Thanks!
", "rawMarkdown": "You're welcome. That's right, I had a logical mistake. We don't need an additional column, I updated my code above (Edit). I looked into my old competition and that's the way how I made it there. Now the predictions should be averaged ranked. Give it a shot, if it doesn't work it's also fine - just for quick testing purpose :) Thanks!", "votes": null }, { "id": "2844027", "postDate": "05/29/2024 20:57:16", "content": "Congratulations! Thanks for the detailed explanation. Finally, the winning solution used the metric hack as was expected for some Kaggle experts.
", "rawMarkdown": "Congratulations! Thanks for the detailed explanation. Finally, the winning solution used the metric hack as was expected for some Kaggle experts.", "votes": null }, { "id": "2844034", "postDate": "05/29/2024 21:02:01", "content": "This is awesome! I'll be reading this in detail soon! :) Great work @yuuniekiri!
", "rawMarkdown": "This is awesome! I'll be reading this in detail soon! :) Great work @yuuniekiri!", "votes": null }, { "id": "2844081", "postDate": "05/29/2024 22:38:52", "content": "Hi! May i know what's the logic of it? Isn't all the scores equal to the mean df_subm[\"score\"] = final_preds_ranked? Thank you!
", "rawMarkdown": "Hi! May i know what's the logic of it? Isn't all the scores equal to the mean df_subm[\"score\"] = final_preds_ranked? Thank you!", "votes": null }, { "id": "2844111", "postDate": "05/29/2024 23:59:54", "content": "Congratulations! This notebook is very impressive for me.
", "rawMarkdown": "Congratulations! This notebook is very impressive for me.", "votes": null }, { "id": "2844391", "postDate": "05/30/2024 04:19:02", "content": "Congratulations on winning this competition. Thanks for detailed explanation of your solution and sharing valuable comments on topic of Metric Hacking. Can you comment on execution time of your notebook and share best practices for the competition participants.
", "rawMarkdown": "Congratulations on winning this competition. Thanks for detailed explanation of your solution and sharing valuable comments on topic of Metric Hacking. Can you comment on execution time of your notebook and share best practices for the competition participants.", "votes": null }, { "id": "2844427", "postDate": "05/30/2024 04:30:15", "content": "Congratulations! I learn a lot~
", "rawMarkdown": "Congratulations! I learn a lot~", "votes": null }, { "id": "2844545", "postDate": "05/30/2024 06:03:01", "content": "Thank you very much.
\nThe execution time of my notes is approximately 2 to 3 hours. (I have never measured it accurately.)
Best practice - I don't know if I'm qualified to comment on this. It's difficult to pinpoint something, but since financial data has a lot of noise and is vulnerable to overfitting, I think we need to pay special attention to generalization. For example, you should be very careful about randomly inserting unexplained data (but this may not always be accurate). In addition, we explore various methods to prevent overfitting.
Your idea is so amazing. Thank you for sharing awesome idea.
", "rawMarkdown": "Your idea is so amazing. Thank you for sharing awesome idea.", "votes": null }, { "id": "2844613", "postDate": "05/30/2024 06:49:12", "content": "@mengjingyang Thanks for your question. Before using rankdata(), the score for each test case is just the raw probability. These probabilities are averaged across all models, resulting in a single score for each test case. However, without rankdata(), these scores can still be skewed or affected by outliers.
By applying rankdata(), the predictions are converted into ranks and then normalized. This process makes the scores more robust by reducing the influence of outliers and ensures a uniform distribution (relative) instead of an absolute distribution (which can be skewed). Hope that clarifies to understand why I thought of that.
In addition, CV improvement through model parameter adjustment had a relatively low correlation with LB, and CV improvement through FE had a relatively high correlation with LB.
This is consistently my experience as well on various problems (not only this competition).
", "rawMarkdown": "```In addition, CV improvement through model parameter adjustment had a relatively low correlation with LB, and CV improvement through FE had a relatively high correlation with LB.```\n\nThis is consistently my experience as well on various problems (not only this competition).", "votes": null }, { "id": "2844679", "postDate": "05/30/2024 07:29:12", "content": "Thanks for sharing - this is great. One question out of curiosity: have you managed to use credit bureau A data successfully?
\n(for context - I haven't - my post with details here https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/478360)
Congratulations , thanks for sharing your wonderfull journay!!
", "rawMarkdown": "Congratulations , thanks for sharing your wonderfull journay!!", "votes": null }, { "id": "2844897", "postDate": "05/30/2024 09:28:31", "content": "Congrats Yuuniee!! From your shared code I assume you used several models for each model category (LGBM, DNN, Catboost). Were those models the result of each folod of the CV process or were models result of training on the whole train dataset but with different params?
", "rawMarkdown": "Congrats Yuuniee!! From your shared code I assume you used several models for each model category (LGBM, DNN, Catboost). Were those models the result of each folod of the CV process or were models result of training on the whole train dataset but with different params?", "votes": null }, { "id": "2844941", "postDate": "05/30/2024 09:55:52", "content": "Thanks a lot.
\nThe CV score above is the score for all out of fold(OOF) of 5folds at StratifiedGroupKFold (no shuffle).
This is quite a mystery to me too.
\nAs you can see from the code and comments for LGBM posted above, I had quite a bit of trouble with it.
\nAs a result, many features were deleted from bureau A and only some were retained.
\n(And I experienced something similar with person data.)
\nBecause test data cannot be seen, it is difficult to determine the cause.
From Credit Bureau data I just used the feature related to the financial institution that approved the loan. Other features degraded the stability metric. I also thought it was better not to use debit card data. On the other hand, the best way I found to use tax data, was the sum of all deduction tax of each client, it makes sense because it represents the tax deductions in the last year. That feature boosted my model's performance.
", "rawMarkdown": "From Credit Bureau data I just used the feature related to the financial institution that approved the loan. Other features degraded the stability metric. I also thought it was better not to use debit card data. On the other hand, the best way I found to use tax data, was the sum of all deduction tax of each client, it makes sense because it represents the tax deductions in the last year. That feature boosted my model's performance.", "votes": null }, { "id": "2845269", "postDate": "05/30/2024 13:42:48", "content": "Hi @yuuniekiri,
\nCongrats with the great performance and amazing solution! Also I would like to thank you for using LightAutoML models in your solution 😎
\n\n\nDNN is a Denselight model and utilizes the Lightautoml library. There were some bugs, so I had to fix some of them myself, but overall, I think it is a good library.
\n
Could you please share the problems you had with our library using GitHub issue mechanism? It will help us to make LightAutoML better 😊
\nP.S. If you already have fixes for the problems and you would like to share them, please use the Pull Request mechanism
\nAlex
\n(Head of LightAutoML team)
Amazing @yuuniekiri ! Congrats!
\nHow did you manage OOM errors?
", "rawMarkdown": "Amazing @yuuniekiri ! Congrats!\n\nHow did you manage OOM errors?", "votes": null }, { "id": "2845622", "postDate": "05/30/2024 15:52:55", "content": "Wow! Quite a nice job! Congrats!!!
", "rawMarkdown": "Wow! Quite a nice job! Congrats!!!", "votes": null }, { "id": "2845692", "postDate": "05/30/2024 16:28:55", "content": "My interpretation is that hosts downsampled this source of data heavily for the test set, and hence it led to overfitting.
", "rawMarkdown": "My interpretation is that hosts downsampled this source of data heavily for the test set, and hence it led to overfitting.", "votes": null }, { "id": "2845807", "postDate": "05/30/2024 17:34:24", "content": "Let me add my 5 cents. With MLP model I also had problems with overfitting, so I used only top 20 features (by gain importance of LGBM). But GBDT models worked pretty well with cred_b_a_* tables, especially one version of CatBoost with a private score of 0.520. Of course some features were filtered, but nothing special (maybe with the exception of additional aggregators that are applied to some num cols). In ensemble, these models (using cred_b_a_* tables) worked well.
\nAnd also in early stages I filtered out some rows with null cols greater then some threshold, but then I think I just found out combination of aggregators that worked well with all tables and rows of cred_b_a_* data.
", "rawMarkdown": "Let me add my 5 cents. With MLP model I also had problems with overfitting, so I used only top 20 features (by gain importance of LGBM). But GBDT models worked pretty well with cred_b_a_* tables, especially one version of CatBoost with a private score of 0.520. Of course some features were filtered, but nothing special (maybe with the exception of additional aggregators that are applied to some num cols). In ensemble, these models (using cred_b_a_* tables) worked well.\n\nAnd also in early stages I filtered out some rows with null cols greater then some threshold, but then I think I just found out combination of aggregators that worked well with all tables and rows of cred_b_a_* data.", "votes": null }, { "id": "2845822", "postDate": "05/30/2024 17:48:55", "content": "I use credit_bureau_a_1_3 for training instead of credit_bureau_a_1_*.
", "rawMarkdown": "I use credit_bureau_a_1_3 for training instead of credit_bureau_a_1_*.", "votes": null }, { "id": "2845992", "postDate": "05/30/2024 19:45:05", "content": "Can you elaborate?
", "rawMarkdown": "Can you elaborate?", "votes": null }, { "id": "2846139", "postDate": "05/31/2024 00:03:40", "content": "Congratulations on your award!
\nThank you for the solution! I will keep this in mind for future reference.
Congratulations, thanks for sharing. @yuuniekiri
", "rawMarkdown": "Congratulations, thanks for sharing. @yuuniekiri", "votes": null }, { "id": "2846194", "postDate": "05/31/2024 02:27:56", "content": "Thank you so much.
\nLightAutoML is a really convenient and useful library.
\nThe problem I had was not fatal, but I will share it there soon. :)
Thank you for your congrats.
\nExcept when there are more than 1000 features, I have not experienced OOM (Out of memory). But if you're having trouble, the following methods may help:
\n\n", "rawMarkdown": "Thank you for your congrats. \n\nExcept when there are more than 1000 features, I have not experienced OOM (Out of memory). But if you're having trouble, the following methods may help:\n\n>1. Capture as many useless features as possible and remove them all.\n>2. Use reduce_memory_useage in the step of reading each file.\n>3. In addition to RAM, GPU RAM is used together. (A representative example is the cuDF library created by NVIDIA's RAPIDS team.)\n>4. When data processing is complete, save the data locally as a file. Then, delete the DataFrame and reset the memory with commands such as gc.collect() and reset.\n>5. In the prediction step, locally stored files are read in chunks and each is predicted.", "votes": null }, { "id": "2846220", "postDate": "05/31/2024 03:03:17", "content": "\n
\n- Capture as many useless features as possible and remove them all.
\n- Use reduce_memory_useage in the step of reading each file.
\n- In addition to RAM, GPU RAM is used together. (A representative example is the cuDF library created by NVIDIA's RAPIDS team.)
\n- When data processing is complete, save the data locally as a file. Then, delete the DataFrame and reset the memory with commands such as gc.collect() and reset.
\n- In the prediction step, locally stored files are read in chunks and each is predicted.
\n
Congratulations. Thank you for sharing.
", "rawMarkdown": "Congratulations. Thank you for sharing.", "votes": null }, { "id": "2846286", "postDate": "05/31/2024 04:01:16", "content": "Thank you for clear cut explanation
", "rawMarkdown": "Thank you for clear cut explanation", "votes": null }, { "id": "2846477", "postDate": "05/31/2024 06:00:01", "content": "great hacking man
", "rawMarkdown": "great hacking man", "votes": null }, { "id": "2846618", "postDate": "05/31/2024 07:08:27", "content": "Hi, we will have to look into it, in production models credit bureau contributes quite significantly…
\nin dataset should be data as they are in our real processes (except for anonymization/masking), and there were no adjustments like downsampling in test set.
Thank you! This logic makes sense. So ranked_preds is a 2d array(N * model_number) here?
", "rawMarkdown": "Thank you! This logic makes sense. So ranked_preds is a 2d array(N * model_number) here?", "votes": null }, { "id": "2846673", "postDate": "05/31/2024 07:39:50", "content": "You're welcome! No, ranked_preds is not a 2D array. It's a 1D array that represents the normalized ranks of the combined predictions from all models.
Interesting
", "rawMarkdown": "Interesting", "votes": null }, { "id": "2847765", "postDate": "05/31/2024 17:14:49", "content": "Congratulations! My team had similar idea - Ensemble model of LightGBM, CatBoost and Tensorflow. We were really close to developing the ensemble model. However after the change of rules and allowing the metric hack, our focus shifted on the metric hack.
", "rawMarkdown": "Congratulations! My team had similar idea - Ensemble model of LightGBM, CatBoost and Tensorflow. We were really close to developing the ensemble model. However after the change of rules and allowing the metric hack, our focus shifted on the metric hack.", "votes": null }, { "id": "2847982", "postDate": "05/31/2024 18:37:21", "content": "I see. So final_preds_ranked is a number. I am still confused as to why you set the entire score column to the same number. Thank you again!
", "rawMarkdown": "I see. So final_preds_ranked is a number. I am still confused as to why you set the entire score column to the same number. Thank you again!", "votes": null }, { "id": "2848059", "postDate": "05/31/2024 19:05:04", "content": "Great approach
", "rawMarkdown": "Great approach", "votes": null }, { "id": "2848245", "postDate": "05/31/2024 23:36:47", "content": "helpful for me!
", "rawMarkdown": "helpful for me!", "votes": null }, { "id": "2849145", "postDate": "06/01/2024 12:33:17", "content": "Congratulations on your award!
\nThank you for the solution! Very interesting approach
Great work man!
", "rawMarkdown": "Great work man!", "votes": null }, { "id": "2850116", "postDate": "06/02/2024 00:10:08", "content": "Smart way of using the hacks!
", "rawMarkdown": "Smart way of using the hacks!", "votes": null }, { "id": "2851303", "postDate": "06/02/2024 17:02:24", "content": "Thanks! Very useful article for fresh man
", "rawMarkdown": "Thanks! Very useful article for fresh man", "votes": null }, { "id": "2851969", "postDate": "06/03/2024 04:24:18", "content": "a very good guide ,thanks
", "rawMarkdown": "a very good guide ,thanks", "votes": null }, { "id": "2852381", "postDate": "06/03/2024 08:58:11", "content": "Thanks! Very useful article for fresh man.
", "rawMarkdown": "Thanks! Very useful article for fresh man.", "votes": null }, { "id": "2852723", "postDate": "06/03/2024 12:29:02", "content": "선생님처럼 할려면 얼마나 해야할까요..
", "rawMarkdown": "선생님처럼 할려면 얼마나 해야할까요..", "votes": null }, { "id": "2852941", "postDate": "06/03/2024 14:24:27", "content": "Participate actively in the competition.
\nI think it's best to have fun!
Keep it in English, please.
", "rawMarkdown": "Keep it in English, please.", "votes": null }, { "id": "2853669", "postDate": "06/03/2024 21:18:45", "content": "thank you for the summary. Very inspiring
", "rawMarkdown": "thank you for the summary. Very inspiring", "votes": null }, { "id": "2856429", "postDate": "06/05/2024 09:14:15", "content": "Congrats @yuuniekiri for the winning. May I ask you what approach did you choose to find the best ensemble weights?
", "rawMarkdown": "Congrats @yuuniekiri for the winning. May I ask you what approach did you choose to find the best ensemble weights?", "votes": null }, { "id": "2856467", "postDate": "06/05/2024 10:01:55", "content": "Thank you for your congrats.
\nMy approach is linear regression and weighted testing in OOF.
\nInterestingly, the optimal weights of OOF score was close to the optimal weights of LB.
Thank you for sharing such a detailed and insightful breakdown of your solution, yuuniee! Your approach to combining LGBM, DNN, and Catboost models with a weighted ensemble clearly highlights the importance of diversity in modeling techniques.
\nI particularly appreciate the transparency in discussing both phases of the competition—Machine Learning and Metric Hacking. Your use of StratifiedGroupKFold and various feature engineering techniques, along with your observations about the correlation between CV improvements and LB performance, are incredibly valuable for the community.
\nThe post-processing strategy and the thoughtful discussion on the impact of WEEK_NUM adjustments and the choices around DEVIDE and REDUCE values offer practical insights into handling similar challenges.
\nIt's also commendable how you acknowledged the contributions and insights from other participants like @sergiosaharovskiy and @greysky for their foundational notebooks, and @at7459 and @johnpateha for their critical discussions around Metric Hacking. The collaboration and knowledge sharing within this community are what make these competitions so enriching.
\nDespite the challenges posed by Metric Hacking, your balanced approach to selecting submission strategies demonstrates a clear understanding of risk management in competition settings.
", "rawMarkdown": "Thank you for sharing such a detailed and insightful breakdown of your solution, yuuniee! Your approach to combining LGBM, DNN, and Catboost models with a weighted ensemble clearly highlights the importance of diversity in modeling techniques.\n\nI particularly appreciate the transparency in discussing both phases of the competition—Machine Learning and Metric Hacking. Your use of StratifiedGroupKFold and various feature engineering techniques, along with your observations about the correlation between CV improvements and LB performance, are incredibly valuable for the community.\n\nThe post-processing strategy and the thoughtful discussion on the impact of WEEK_NUM adjustments and the choices around DEVIDE and REDUCE values offer practical insights into handling similar challenges.\n\nIt's also commendable how you acknowledged the contributions and insights from other participants like @sergiosaharovskiy and @greysky for their foundational notebooks, and @at7459 and @johnpateha for their critical discussions around Metric Hacking. The collaboration and knowledge sharing within this community are what make these competitions so enriching.\n\nDespite the challenges posed by Metric Hacking, your balanced approach to selecting submission strategies demonstrates a clear understanding of risk management in competition settings.", "votes": null }, { "id": "2865334", "postDate": "06/10/2024 17:01:02", "content": "Congratulations
", "rawMarkdown": "Congratulations", "votes": null }, { "id": "2875243", "postDate": "06/17/2024 01:10:07", "content": "Congrats @yuuniekiri for the winning. How do we filter out effective features and toxic features?Thank you for your response in advance.
", "rawMarkdown": "Congrats @yuuniekiri for the winning. How do we filter out effective features and toxic features?Thank you for your response in advance.", "votes": null }, { "id": "2875270", "postDate": "06/17/2024 02:01:37", "content": "As you can see in the article above, features that have a significant advantage in the CV score are kept, and features that have no advantage are removed.
\nAdditionally, even if there is no significant gain in score, logically meaningful and stable features can be kept. This can also apply in the opposite case.
Nice share, Congratulations for your first solo gold.
", "rawMarkdown": "Nice share, Congratulations for your first solo gold.", "votes": null }, { "id": "2889766", "postDate": "06/25/2024 16:53:07", "content": "Congratulations
", "rawMarkdown": "Congratulations", "votes": null }, { "id": "2904468", "postDate": "07/04/2024 11:27:27", "content": "Hi @yuuniekiri, I cannot find the final submission notebook for the first place. Can you please share it?
", "rawMarkdown": "Hi @yuuniekiri, I cannot find the final submission notebook for the first place. Can you please share it?", "votes": null }, { "id": "3012991", "postDate": "10/09/2024 14:54:37", "content": "Thanks for your share!
", "rawMarkdown": "Thanks for your share!", "votes": null }, { "id": "3135086", "postDate": "02/27/2025 03:49:04", "content": "Hi @yuuniekiri, Congratulations on your win. I love the way you approach Feature Engineering, especially Aggregation. I was wondering—if all feature names were hidden, how would you do in Aggregation?
", "rawMarkdown": "Hi @yuuniekiri, Congratulations on your win. I love the way you approach Feature Engineering, especially Aggregation. I was wondering—if all feature names were hidden, how would you do in Aggregation?", "votes": null }, { "id": "3312920", "postDate": "11/08/2025 08:47:50", "content": "Nice, keep doing well
", "rawMarkdown": "Nice, keep doing well", "votes": null } ], "comments": [ { "id": 2842917, "author_name": "thiagomantuani", "author_url": "", "post_date": "05/29/2024 10:37:11", "content": "Congratulations on the placement @yuuniekiri , how did you choose the features for each model?
\nIf possible, share your notebook with us.
Thank you, @thiagomantuani. A notebook link has been added.
", "votes": null, "replies": [] } ] }, { "id": 2843263, "author_name": "etiennekaiser", "author_url": "", "post_date": "05/29/2024 13:35:53", "content": "Congratulations on your second #1 rank competition @yuuniekiri! Even though I didn't participate in the competition, I was following along since the beginning after my initial post some months ago. After reading your straightforward write-up and both notebooks, I totally agree with your choice of models, such as CatBoost, LGBM, and DNN (eventually NaiveBayes would've helped a bit). My only question that I have left is, did you try to use rankdata() after making the predictions? That gave me a huge boost in my first competition with almost exactly the same ensemble models. I'm just curious, if you don't mind, whenever you have time, could you please try a \"Late submission\" with the following updated part of your \"Fork of Home Credit Risk (LightGBM)\" at the very end:
Edit after @yuuniekiri comment:
\n scipy.stats rankdata \n\n reader\ngc.collect()\n\nranked_preds = rankdata(y_pred) / (y_pred) \n\ndf_subm = pd.read_csv()\ndf_subm = df_subm.set_index()\ndf_subm[] = ranked_preds \ndf_subm.to_csv()\n(df_subm)\nThanks a lot for your try-out and response. Enjoy your preliminary win!
\nBest regards,
\nEtienne
Thank you for your congratulations.
\nHowever, the submissions resulting from the code above are different from the competition format.
\n(Only two columns are allowed: case_id and score.)
\nIt will probably cause an error.
You're welcome. That's right, I had a logical mistake. We don't need an additional column, I updated my code above (Edit). I looked into my old competition and that's the way how I made it there. Now the predictions should be averaged ranked. Give it a shot, if it doesn't work it's also fine - just for quick testing purpose :) Thanks!
", "votes": null, "replies": [ { "id": 2844081, "author_name": "mengjingyang", "author_url": "", "post_date": "05/29/2024 22:38:52", "content": "Hi! May i know what's the logic of it? Isn't all the scores equal to the mean df_subm[\"score\"] = final_preds_ranked? Thank you!
", "votes": null, "replies": [ { "id": 2844613, "author_name": "etiennekaiser", "author_url": "", "post_date": "05/30/2024 06:49:12", "content": "@mengjingyang Thanks for your question. Before using rankdata(), the score for each test case is just the raw probability. These probabilities are averaged across all models, resulting in a single score for each test case. However, without rankdata(), these scores can still be skewed or affected by outliers.
By applying rankdata(), the predictions are converted into ranks and then normalized. This process makes the scores more robust by reducing the influence of outliers and ensures a uniform distribution (relative) instead of an absolute distribution (which can be skewed). Hope that clarifies to understand why I thought of that.
Thank you! This logic makes sense. So ranked_preds is a 2d array(N * model_number) here?
", "votes": null, "replies": [ { "id": 2846673, "author_name": "etiennekaiser", "author_url": "", "post_date": "05/31/2024 07:39:50", "content": "You're welcome! No, ranked_preds is not a 2D array. It's a 1D array that represents the normalized ranks of the combined predictions from all models.
I see. So final_preds_ranked is a number. I am still confused as to why you set the entire score column to the same number. Thank you again!
", "votes": null, "replies": [] } ] } ] } ] } ] } ] } ] } ] }, { "id": 2844027, "author_name": "eduardotoloza", "author_url": "", "post_date": "05/29/2024 20:57:16", "content": "Congratulations! Thanks for the detailed explanation. Finally, the winning solution used the metric hack as was expected for some Kaggle experts.
", "votes": null, "replies": [] }, { "id": 2844034, "author_name": "vishrutgrover", "author_url": "", "post_date": "05/29/2024 21:02:01", "content": "This is awesome! I'll be reading this in detail soon! :) Great work @yuuniekiri!
", "votes": null, "replies": [] }, { "id": 2844111, "author_name": "chrischan0204", "author_url": "", "post_date": "05/29/2024 23:59:54", "content": "Congratulations! This notebook is very impressive for me.
", "votes": null, "replies": [] }, { "id": 2844391, "author_name": "crsuthikshnkumar", "author_url": "", "post_date": "05/30/2024 04:19:02", "content": "Congratulations on winning this competition. Thanks for detailed explanation of your solution and sharing valuable comments on topic of Metric Hacking. Can you comment on execution time of your notebook and share best practices for the competition participants.
", "votes": null, "replies": [ { "id": 2844545, "author_name": "yuuniekiri", "author_url": "", "post_date": "05/30/2024 06:03:01", "content": "Thank you very much.
\nThe execution time of my notes is approximately 2 to 3 hours. (I have never measured it accurately.)
Best practice - I don't know if I'm qualified to comment on this. It's difficult to pinpoint something, but since financial data has a lot of noise and is vulnerable to overfitting, I think we need to pay special attention to generalization. For example, you should be very careful about randomly inserting unexplained data (but this may not always be accurate). In addition, we explore various methods to prevent overfitting.
Congratulations! I learn a lot~
", "votes": null, "replies": [] }, { "id": 2844603, "author_name": "ishlove7", "author_url": "", "post_date": "05/30/2024 06:39:27", "content": "Your idea is so amazing. Thank you for sharing awesome idea.
", "votes": null, "replies": [] }, { "id": 2844674, "author_name": "narsil", "author_url": "", "post_date": "05/30/2024 07:26:14", "content": "In addition, CV improvement through model parameter adjustment had a relatively low correlation with LB, and CV improvement through FE had a relatively high correlation with LB.
This is consistently my experience as well on various problems (not only this competition).
", "votes": null, "replies": [] }, { "id": 2844679, "author_name": "narsil", "author_url": "", "post_date": "05/30/2024 07:29:12", "content": "Thanks for sharing - this is great. One question out of curiosity: have you managed to use credit bureau A data successfully?
\n(for context - I haven't - my post with details here https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/478360)
This is quite a mystery to me too.
\nAs you can see from the code and comments for LGBM posted above, I had quite a bit of trouble with it.
\nAs a result, many features were deleted from bureau A and only some were retained.
\n(And I experienced something similar with person data.)
\nBecause test data cannot be seen, it is difficult to determine the cause.
From Credit Bureau data I just used the feature related to the financial institution that approved the loan. Other features degraded the stability metric. I also thought it was better not to use debit card data. On the other hand, the best way I found to use tax data, was the sum of all deduction tax of each client, it makes sense because it represents the tax deductions in the last year. That feature boosted my model's performance.
", "votes": null, "replies": [] }, { "id": 2845692, "author_name": "narsil", "author_url": "", "post_date": "05/30/2024 16:28:55", "content": "My interpretation is that hosts downsampled this source of data heavily for the test set, and hence it led to overfitting.
", "votes": null, "replies": [ { "id": 2846618, "author_name": "tomasjeline2", "author_url": "", "post_date": "05/31/2024 07:08:27", "content": "Hi, we will have to look into it, in production models credit bureau contributes quite significantly…
\nin dataset should be data as they are in our real processes (except for anonymization/masking), and there were no adjustments like downsampling in test set.
Let me add my 5 cents. With MLP model I also had problems with overfitting, so I used only top 20 features (by gain importance of LGBM). But GBDT models worked pretty well with cred_b_a_* tables, especially one version of CatBoost with a private score of 0.520. Of course some features were filtered, but nothing special (maybe with the exception of additional aggregators that are applied to some num cols). In ensemble, these models (using cred_b_a_* tables) worked well.
\nAnd also in early stages I filtered out some rows with null cols greater then some threshold, but then I think I just found out combination of aggregators that worked well with all tables and rows of cred_b_a_* data.
", "votes": null, "replies": [] }, { "id": 2845822, "author_name": "thunderthunder", "author_url": "", "post_date": "05/30/2024 17:48:55", "content": "I use credit_bureau_a_1_3 for training instead of credit_bureau_a_1_*.
", "votes": null, "replies": [ { "id": 2845992, "author_name": "eduardotoloza", "author_url": "", "post_date": "05/30/2024 19:45:05", "content": "Can you elaborate?
", "votes": null, "replies": [] } ] } ] }, { "id": 2844882, "author_name": "kunduruanil", "author_url": "", "post_date": "05/30/2024 09:13:23", "content": "Congratulations , thanks for sharing your wonderfull journay!!
", "votes": null, "replies": [] }, { "id": 2844897, "author_name": "diegoiglesias", "author_url": "", "post_date": "05/30/2024 09:28:31", "content": "Congrats Yuuniee!! From your shared code I assume you used several models for each model category (LGBM, DNN, Catboost). Were those models the result of each folod of the CV process or were models result of training on the whole train dataset but with different params?
", "votes": null, "replies": [ { "id": 2844941, "author_name": "yuuniekiri", "author_url": "", "post_date": "05/30/2024 09:55:52", "content": "Thanks a lot.
\nThe CV score above is the score for all out of fold(OOF) of 5folds at StratifiedGroupKFold (no shuffle).
Hi @yuuniekiri,
\nCongrats with the great performance and amazing solution! Also I would like to thank you for using LightAutoML models in your solution 😎
\n\n\nDNN is a Denselight model and utilizes the Lightautoml library. There were some bugs, so I had to fix some of them myself, but overall, I think it is a good library.
\n
Could you please share the problems you had with our library using GitHub issue mechanism? It will help us to make LightAutoML better 😊
\nP.S. If you already have fixes for the problems and you would like to share them, please use the Pull Request mechanism
\nAlex
\n(Head of LightAutoML team)
Thank you so much.
\nLightAutoML is a really convenient and useful library.
\nThe problem I had was not fatal, but I will share it there soon. :)
Amazing @yuuniekiri ! Congrats!
\nHow did you manage OOM errors?
", "votes": null, "replies": [ { "id": 2846201, "author_name": "yuuniekiri", "author_url": "", "post_date": "05/31/2024 02:40:09", "content": "Thank you for your congrats.
\nExcept when there are more than 1000 features, I have not experienced OOM (Out of memory). But if you're having trouble, the following methods may help:
\n\n", "votes": null, "replies": [] } ] }, { "id": 2845622, "author_name": "barbosajaf", "author_url": "", "post_date": "05/30/2024 15:52:55", "content": "\n
\n- Capture as many useless features as possible and remove them all.
\n- Use reduce_memory_useage in the step of reading each file.
\n- In addition to RAM, GPU RAM is used together. (A representative example is the cuDF library created by NVIDIA's RAPIDS team.)
\n- When data processing is complete, save the data locally as a file. Then, delete the DataFrame and reset the memory with commands such as gc.collect() and reset.
\n- In the prediction step, locally stored files are read in chunks and each is predicted.
\n
Wow! Quite a nice job! Congrats!!!
", "votes": null, "replies": [] }, { "id": 2846139, "author_name": "hoshinoyumemi", "author_url": "", "post_date": "05/31/2024 00:03:40", "content": "Congratulations on your award!
\nThank you for the solution! I will keep this in mind for future reference.
Congratulations, thanks for sharing. @yuuniekiri
", "votes": null, "replies": [] }, { "id": 2846220, "author_name": "clovisdalmolinvieira", "author_url": "", "post_date": "05/31/2024 03:03:17", "content": "Congratulations. Thank you for sharing.
", "votes": null, "replies": [] }, { "id": 2846286, "author_name": "masopf", "author_url": "", "post_date": "05/31/2024 04:01:16", "content": "Thank you for clear cut explanation
", "votes": null, "replies": [] }, { "id": 2846477, "author_name": "antonygithinji", "author_url": "", "post_date": "05/31/2024 06:00:01", "content": "great hacking man
", "votes": null, "replies": [] }, { "id": 2847445, "author_name": "ineerav", "author_url": "", "post_date": "05/31/2024 14:39:11", "content": "Interesting
", "votes": null, "replies": [] }, { "id": 2847765, "author_name": "matousfamera", "author_url": "", "post_date": "05/31/2024 17:14:49", "content": "Congratulations! My team had similar idea - Ensemble model of LightGBM, CatBoost and Tensorflow. We were really close to developing the ensemble model. However after the change of rules and allowing the metric hack, our focus shifted on the metric hack.
", "votes": null, "replies": [] }, { "id": 2848059, "author_name": "tomaraman", "author_url": "", "post_date": "05/31/2024 19:05:04", "content": "Great approach
", "votes": null, "replies": [] }, { "id": 2848245, "author_name": "aa000098", "author_url": "", "post_date": "05/31/2024 23:36:47", "content": "helpful for me!
", "votes": null, "replies": [] }, { "id": 2849145, "author_name": "arjunm97", "author_url": "", "post_date": "06/01/2024 12:33:17", "content": "Congratulations on your award!
\nThank you for the solution! Very interesting approach
Great work man!
", "votes": null, "replies": [] }, { "id": 2850116, "author_name": "manavtrivedi", "author_url": "", "post_date": "06/02/2024 00:10:08", "content": "Smart way of using the hacks!
", "votes": null, "replies": [] }, { "id": 2851303, "author_name": "larrylin666", "author_url": "", "post_date": "06/02/2024 17:02:24", "content": "Thanks! Very useful article for fresh man
", "votes": null, "replies": [] }, { "id": 2851969, "author_name": "mubashirsidiki", "author_url": "", "post_date": "06/03/2024 04:24:18", "content": "a very good guide ,thanks
", "votes": null, "replies": [] }, { "id": 2852381, "author_name": "kerenshang", "author_url": "", "post_date": "06/03/2024 08:58:11", "content": "Thanks! Very useful article for fresh man.
", "votes": null, "replies": [] }, { "id": 2852723, "author_name": "jidaengwarrior", "author_url": "", "post_date": "06/03/2024 12:29:02", "content": "선생님처럼 할려면 얼마나 해야할까요..
", "votes": null, "replies": [ { "id": 2852941, "author_name": "yuuniekiri", "author_url": "", "post_date": "06/03/2024 14:24:27", "content": "Participate actively in the competition.
\nI think it's best to have fun!
Keep it in English, please.
", "votes": null, "replies": [] } ] }, { "id": 2853669, "author_name": "milkyway0809", "author_url": "", "post_date": "06/03/2024 21:18:45", "content": "thank you for the summary. Very inspiring
", "votes": null, "replies": [] }, { "id": 2856429, "author_name": "rram12", "author_url": "", "post_date": "06/05/2024 09:14:15", "content": "Congrats @yuuniekiri for the winning. May I ask you what approach did you choose to find the best ensemble weights?
", "votes": null, "replies": [ { "id": 2856467, "author_name": "yuuniekiri", "author_url": "", "post_date": "06/05/2024 10:01:55", "content": "Thank you for your congrats.
\nMy approach is linear regression and weighted testing in OOF.
\nInterestingly, the optimal weights of OOF score was close to the optimal weights of LB.
Thank you for sharing such a detailed and insightful breakdown of your solution, yuuniee! Your approach to combining LGBM, DNN, and Catboost models with a weighted ensemble clearly highlights the importance of diversity in modeling techniques.
\nI particularly appreciate the transparency in discussing both phases of the competition—Machine Learning and Metric Hacking. Your use of StratifiedGroupKFold and various feature engineering techniques, along with your observations about the correlation between CV improvements and LB performance, are incredibly valuable for the community.
\nThe post-processing strategy and the thoughtful discussion on the impact of WEEK_NUM adjustments and the choices around DEVIDE and REDUCE values offer practical insights into handling similar challenges.
\nIt's also commendable how you acknowledged the contributions and insights from other participants like @sergiosaharovskiy and @greysky for their foundational notebooks, and @at7459 and @johnpateha for their critical discussions around Metric Hacking. The collaboration and knowledge sharing within this community are what make these competitions so enriching.
\nDespite the challenges posed by Metric Hacking, your balanced approach to selecting submission strategies demonstrates a clear understanding of risk management in competition settings.
", "votes": null, "replies": [] }, { "id": 2865334, "author_name": "troostiticgh", "author_url": "", "post_date": "06/10/2024 17:01:02", "content": "Congratulations
", "votes": null, "replies": [] }, { "id": 2875243, "author_name": "wajyjpku", "author_url": "", "post_date": "06/17/2024 01:10:07", "content": "Congrats @yuuniekiri for the winning. How do we filter out effective features and toxic features?Thank you for your response in advance.
", "votes": null, "replies": [ { "id": 2875270, "author_name": "yuuniekiri", "author_url": "", "post_date": "06/17/2024 02:01:37", "content": "As you can see in the article above, features that have a significant advantage in the CV score are kept, and features that have no advantage are removed.
\nAdditionally, even if there is no significant gain in score, logically meaningful and stable features can be kept. This can also apply in the opposite case.
Nice share, Congratulations for your first solo gold.
", "votes": null, "replies": [] }, { "id": 2889766, "author_name": "nonchokablexd", "author_url": "", "post_date": "06/25/2024 16:53:07", "content": "Congratulations
", "votes": null, "replies": [] }, { "id": 2904468, "author_name": "jeannkouagou", "author_url": "", "post_date": "07/04/2024 11:27:27", "content": "Hi @yuuniekiri, I cannot find the final submission notebook for the first place. Can you please share it?
", "votes": null, "replies": [] }, { "id": 3012991, "author_name": "outlierrr", "author_url": "", "post_date": "10/09/2024 14:54:37", "content": "Thanks for your share!
", "votes": null, "replies": [] }, { "id": 3135086, "author_name": "trannhattrinhk16hcm", "author_url": "", "post_date": "02/27/2025 03:49:04", "content": "Hi @yuuniekiri, Congratulations on your win. I love the way you approach Feature Engineering, especially Aggregation. I was wondering—if all feature names were hidden, how would you do in Aggregation?
", "votes": null, "replies": [] }, { "id": 3312920, "author_name": "afeedahamed", "author_url": "", "post_date": "11/08/2025 08:47:50", "content": "Nice, keep doing well
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2842729": "##Foreword\n\nHello, this is yuuniee.\nThank you to the competition host Home Credit and staff, Kaggle staff and everyone involved.\n\nThe image below is a summary of my solution.\n\n\n\nThe maximum Private LB Score for my model pool without Metric Hacking is close to about 0.53.\n\nAs everyone knows, this competition takes place in two phases.\nPhase 1 was ML (Machine Learning), and Phase 2 was MH (Metric Hack).\nI will briefly introduce each part.\n(If you are only curious about ML, please read only Phase 1.)\n\n\n## I. Phase 1 - ML (Machine Learning)\n\nThank you for writing a great notebook for EDA and solution construction at the beginning of the competition.\n@sergiosaharovskiy - https://www.kaggle.com/code/sergiosaharovskiy/home-credit-crms-2024-eda-and-submission\n@greysky - https://www.kaggle.com/code/greysky/home-credit-baseline\nAdditionally, I would like to thank the many participants who provided various insights.\n\n1. The CV strategy is StratifiedGroupKFold, and has been tested and trained in various forms: No Shuffle and Shuffle. In my case, differences in CV of around 0.001 to 0.005 had a low correlation with LB, while differences above 0.01 had a high correlation with LB. In addition, CV improvement through model parameter adjustment had a relatively low correlation with LB, and CV improvement through FE had a relatively high correlation with LB.\n\n2. There's nothing special about Feature Engineering. Max, Min, Avg, Var, First, Last and Max-Min Difference of each item were utilized. (I will release the code after cleaning it up.)\n\n3. LGBM was a bit lacking compared to Catboost, but was good for the ensemble. The reason why the number of features is smaller is because some items (mainly categorical types) were excluded because they caused performance degradation and overfitting in LGBM, and other slightly different types of features were added and deleted.\n\n4. DNN is a Denselight model and utilizes the LightAutoML library. There were some bugs, so I had to fix some of them myself, but overall, I think it is a good library. At first, I planned to lightly test with Denselight and then build the final model with a larger model (FT-Transformer, etc.). Surprisingly, I haven't created or found a model that beats Denselight performance. Perhaps I made a mistake, or the nature of the data is sensitive to overfitting, so I guess a simpler model worked better.\n\n5. Catboost was the model that performed best in this competition and I guess it performed well on a large number of categorical features (about 117).\n\nDuring this period, my public LB was close to top 5 and the competition seemed to be going very smoothly.\nBut...\n\n\n>### + What didn't work\n>1. Transformer type models such as Tabnet, TabTransformer, FT-Transformer, etc.\n>2. Income, expenditure, and tax statistics by period (month, week, etc.).\n>3. Differences between individual income, expenses, and taxes.\n>4. K-means clustering\n\n\n## II. Phase 2 - MH(Metric Hack)\n\nPointed out problems with MH at the beginning of the competition,\n@at7459 - https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/476449\nAnnounced the revival of MH in the latter half of the competition,\n@johnpateha - https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/497167\n\nAlso, thanks to the many participants who suggested various other comments and solutions. Thanks to you, my knowledge has expanded.\nIn particular, the two people above are true scientists and journalists who could have hidden this discovery and profited from it, but pursued the public interest by exposing it to everyone. (Why not consider Kaggle giving a special award for this sharing?)\nAs a result, the problem was not solved, but we learned something(?) from it and at least I was able to take minimal measures.\n\nAfter @johnpateha's \"Metric hack again, sorry\" post, I tried Data Reversing and found that the difference between date_decision and min_refreshdate_3813885D had a high correlation (about 0.9 or more) with WEEK_NUM.\n(More information can be found at https://www.kaggle.com/code/eivolkova/how-to-restore-the-dates?scriptVersionId=180157891, published by @eivolkova immediately after the competition.)\nAnd the way suggested by @at7459:\n\n```python\nDEVIDE = 1/2\nREDUCE = 0.02\ncondition = df['WEEK_NUM'] < (df['WEEK_NUM'].max()-df['WEEK_NUM'].min())*DEVIDE+df['WEEK_NUM'].min()\ndf.loc[condition, 'score'] = (df.loc[condition, 'score'] - REDUCE).clip(0)\n```\n\nBased on this, the results of examining the score details submitted by changing the REDUCE and DEVIDE values were as follows.\n\n1. The public/private division of test data spans the entire period and it is difficult to identify a specific distribution, and the optimal value in private will probably be distributed around 1/4 to 3/4.\n2. The optimal public LB value of REDUCE is 0.03 (at least in my model), and the optimal value in private is probably distributed around 0.02 to 0.04.\n\nThe difference in pure model performance between top competitors was 0.00X,\nSince the difference due to the above adjustment value was 0.0X,\nAt this stage it became gambling for me.\n\nSome will bet low, others will bet high.\nHere, I trusted my model and chose neutral, and chose DEVIDE=1/2, REDUCE=0.03, which is the median of the expected distribution.\nI expected someone who bet high or low to take the prize, and I expected about a 50% chance of finishing in the top 10.\n\n\n## III. Conclusion\n\n+ The first selected submission is the No Hack model with CV Best Score created in Phase 1, and the second selected submission is the application of Metric Hack to it. (I thought there was a high probability that Metric Hack would work, but I also submitted No Hack just in case)\n\n+ Rather than simply sharing solutions, I thought it would be better to share my experience in detail. To improve the bad parts that happened in this competition.\n\n+ My ranking was due to luck, and although it is the Best Private Score submitted, it probably won't be Best Solution.\n\n+ Although the competition ended on a gloomy note due to the metric hacking issue, the host and staff worked hard to make it a meaningful and successful competition. \n\n+ Based on this, we expect a more developed system in the future.\n\n\nthank you\n\n\n>Edit : I posted the FE code here.\n>644(for lgbm) - https://www.kaggle.com/code/yuuniekiri/fork-of-home-credit-risk-lightgbm\n>661(for cat & dnn) - https://www.kaggle.com/code/yuuniekiri/fork-of-home-credit-catboost-inference", "2842917": "Congratulations on the placement @yuuniekiri , how did you choose the features for each model?\nIf possible, share your notebook with us.", "2842974": "Thank you, @thiagomantuani. A notebook link has been added.", "2843263": "**Congratulations on your second #1 rank competition @yuuniekiri!** Even though I didn't participate in the competition, I was following along since the beginning after my initial post some months ago. After reading your straightforward write-up and both notebooks, I totally agree with your choice of models, such as CatBoost, LGBM, and DNN (eventually NaiveBayes would've helped a bit). My only question that I have left is, did you try to use `rankdata()` after making the predictions? That gave me a huge boost in my first competition with almost exactly the same ensemble models. I'm just curious, if you don't mind, whenever you have time, could you please try a \"Late submission\" with the following updated part of your \"Fork of Home Credit Risk (LightGBM)\" at the very end:\n\n**Edit after @yuuniekiri comment:**\n\n```python\nfrom scipy.stats import rankdata # <- newly added, import rankdata\n\ndel reader\ngc.collect()\n\nranked_preds = rankdata(y_pred) / len(y_pred) # <- newly added\n\ndf_subm = pd.read_csv(f\"{ROOT}/sample_submission.csv\")\ndf_subm = df_subm.set_index(\"case_id\")\ndf_subm[\"score\"] = ranked_preds # <- newly adjusted\ndf_subm.to_csv(\"submission.csv\")\nprint(df_subm)\n```\n\nThanks a lot for your try-out and response. Enjoy your preliminary win!\nBest regards,\nEtienne", "2843306": "Thank you for your congratulations. \nHowever, the submissions resulting from the code above are different from the competition format.\n(Only two columns are allowed: case_id and score.)\nIt will probably cause an error.", "2843340": "You're welcome. That's right, I had a logical mistake. We don't need an additional column, I updated my code above (Edit). I looked into my old competition and that's the way how I made it there. Now the predictions should be averaged ranked. Give it a shot, if it doesn't work it's also fine - just for quick testing purpose :) Thanks!", "2844027": "Congratulations! Thanks for the detailed explanation. Finally, the winning solution used the metric hack as was expected for some Kaggle experts.", "2844034": "This is awesome! I'll be reading this in detail soon! :) Great work @yuuniekiri!", "2844081": "Hi! May i know what's the logic of it? Isn't all the scores equal to the mean df_subm[\"score\"] = final_preds_ranked? Thank you!", "2844111": "Congratulations! This notebook is very impressive for me.", "2844391": "Congratulations on winning this competition. Thanks for detailed explanation of your solution and sharing valuable comments on topic of Metric Hacking. Can you comment on execution time of your notebook and share best practices for the competition participants.", "2844427": "Congratulations! I learn a lot~", "2844545": "Thank you very much. \n\n1. The execution time of my notes is approximately 2 to 3 hours. (I have never measured it accurately.)\n\n2. Best practice - I don't know if I'm qualified to comment on this. It's difficult to pinpoint something, but since financial data has a lot of noise and is vulnerable to overfitting, I think we need to pay special attention to generalization. For example, you should be very careful about randomly inserting unexplained data (but this may not always be accurate). In addition, we explore various methods to prevent overfitting.", "2844603": "Your idea is so amazing. Thank you for sharing awesome idea.", "2844613": "mengjingyang Thanks for your question. Before using `rankdata()`, the score for each test case is just the raw probability. These probabilities are averaged across all models, resulting in a single score for each test case. However, without `rankdata()`, these scores can still be skewed or affected by outliers.\n\nBy applying `rankdata()`, the predictions are converted into ranks and then normalized. This process makes the scores more robust by reducing the influence of outliers and ensures a uniform distribution (relative) instead of an absolute distribution (which can be skewed). Hope that clarifies to understand why I thought of that.", "2844674": "```In addition, CV improvement through model parameter adjustment had a relatively low correlation with LB, and CV improvement through FE had a relatively high correlation with LB.```\n\nThis is consistently my experience as well on various problems (not only this competition).", "2844679": "Thanks for sharing - this is great. One question out of curiosity: have you managed to use credit bureau A data successfully?\n(for context - I haven't - my post with details here https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/478360)", "2844882": "Congratulations , thanks for sharing your wonderfull journay!!", "2844897": "Congrats Yuuniee!! From your shared code I assume you used several models for each model category (LGBM, DNN, Catboost). Were those models the result of each folod of the CV process or were models result of training on the whole train dataset but with different params?", "2844941": "Thanks a lot. \nThe CV score above is the score for all out of fold(OOF) of 5folds at StratifiedGroupKFold (no shuffle).", "2844961": "This is quite a mystery to me too. \nAs you can see from the code and comments for LGBM posted above, I had quite a bit of trouble with it.\nAs a result, many features were deleted from bureau A and only some were retained. \n(And I experienced something similar with person data.)\nBecause test data cannot be seen, it is difficult to determine the cause.", "2845079": "From Credit Bureau data I just used the feature related to the financial institution that approved the loan. Other features degraded the stability metric. I also thought it was better not to use debit card data. On the other hand, the best way I found to use tax data, was the sum of all deduction tax of each client, it makes sense because it represents the tax deductions in the last year. That feature boosted my model's performance.", "2845269": "Hi @yuuniekiri,\n\nCongrats with the great performance and amazing solution! Also I would like to thank you for using LightAutoML models in your solution 😎\n\n>DNN is a Denselight model and utilizes the Lightautoml library. There were some bugs, so I had to fix some of them myself, but overall, I think it is a good library. \n\nCould you please share the problems you had with our library using [GitHub issue mechanism](https://github.com/sb-ai-lab/LightAutoML/issues)? It will help us to make LightAutoML better 😊\n\nP.S. If you already have fixes for the problems and you would like to share them, please use the [Pull Request mechanism](https://github.com/sb-ai-lab/LightAutoML/pulls)\n\nAlex\n(Head of LightAutoML team)", "2845308": "Amazing @yuuniekiri ! Congrats!\n\nHow did you manage OOM errors?", "2845622": "Wow! Quite a nice job! Congrats!!!", "2845692": "My interpretation is that hosts downsampled this source of data heavily for the test set, and hence it led to overfitting.", "2845807": "Let me add my 5 cents. With MLP model I also had problems with overfitting, so I used only top 20 features (by gain importance of LGBM). But GBDT models worked pretty well with cred_b_a_* tables, especially one version of CatBoost with a private score of 0.520. Of course some features were filtered, but nothing special (maybe with the exception of additional aggregators that are applied to some num cols). In ensemble, these models (using cred_b_a_* tables) worked well.\n\nAnd also in early stages I filtered out some rows with null cols greater then some threshold, but then I think I just found out combination of aggregators that worked well with all tables and rows of cred_b_a_* data.", "2845822": "I use credit_bureau_a_1_3 for training instead of credit_bureau_a_1_*.", "2845992": "Can you elaborate?", "2846139": "Congratulations on your award! \nThank you for the solution! I will keep this in mind for future reference.", "2846187": "Congratulations, thanks for sharing. @yuuniekiri", "2846194": "Thank you so much.\n\nLightAutoML is a really convenient and useful library.\nThe problem I had was not fatal, but I will share it there soon. :)", "2846201": "Thank you for your congrats. \n\nExcept when there are more than 1000 features, I have not experienced OOM (Out of memory). But if you're having trouble, the following methods may help:\n\n>1. Capture as many useless features as possible and remove them all.\n>2. Use reduce_memory_useage in the step of reading each file.\n>3. In addition to RAM, GPU RAM is used together. (A representative example is the cuDF library created by NVIDIA's RAPIDS team.)\n>4. When data processing is complete, save the data locally as a file. Then, delete the DataFrame and reset the memory with commands such as gc.collect() and reset.\n>5. In the prediction step, locally stored files are read in chunks and each is predicted.", "2846220": "Congratulations. Thank you for sharing.", "2846286": "Thank you for clear cut explanation", "2846477": "great hacking man", "2846618": "Hi, we will have to look into it, in production models credit bureau contributes quite significantly...\nin dataset should be data as they are in our real processes (except for anonymization/masking), and there were no adjustments like downsampling in test set.", "2846640": "Thank you! This logic makes sense. So ranked_preds is a 2d array(N * model_number) here?", "2846673": "You're welcome! No, `ranked_preds` is not a 2D array. It's a 1D array that represents the normalized ranks of the combined predictions from **all** models.", "2847445": "Interesting", "2847765": "Congratulations! My team had similar idea - Ensemble model of LightGBM, CatBoost and Tensorflow. We were really close to developing the ensemble model. However after the change of rules and allowing the metric hack, our focus shifted on the metric hack.", "2847982": "I see. So final_preds_ranked is a number. I am still confused as to why you set the entire score column to the same number. Thank you again!", "2848059": "Great approach", "2848245": "helpful for me!", "2849145": "Congratulations on your award!\nThank you for the solution! Very interesting approach", "2849248": "Great work man!", "2850116": "Smart way of using the hacks!", "2851303": "Thanks! Very useful article for fresh man", "2851969": "a very good guide ,thanks", "2852381": "Thanks! Very useful article for fresh man.", "2852723": "선생님처럼 할려면 얼마나 해야할까요..", "2852941": "Participate actively in the competition.\nI think it's best to have fun!", "2852944": "Keep it in English, please.", "2853669": "thank you for the summary. Very inspiring", "2856429": "Congrats @yuuniekiri for the winning. May I ask you what approach did you choose to find the best ensemble weights?", "2856467": "Thank you for your congrats.\nMy approach is linear regression and weighted testing in OOF.\nInterestingly, the optimal weights of OOF score was close to the optimal weights of LB.", "2858305": "Thank you for sharing such a detailed and insightful breakdown of your solution, yuuniee! Your approach to combining LGBM, DNN, and Catboost models with a weighted ensemble clearly highlights the importance of diversity in modeling techniques.\n\nI particularly appreciate the transparency in discussing both phases of the competition—Machine Learning and Metric Hacking. Your use of StratifiedGroupKFold and various feature engineering techniques, along with your observations about the correlation between CV improvements and LB performance, are incredibly valuable for the community.\n\nThe post-processing strategy and the thoughtful discussion on the impact of WEEK_NUM adjustments and the choices around DEVIDE and REDUCE values offer practical insights into handling similar challenges.\n\nIt's also commendable how you acknowledged the contributions and insights from other participants like @sergiosaharovskiy and @greysky for their foundational notebooks, and @at7459 and @johnpateha for their critical discussions around Metric Hacking. The collaboration and knowledge sharing within this community are what make these competitions so enriching.\n\nDespite the challenges posed by Metric Hacking, your balanced approach to selecting submission strategies demonstrates a clear understanding of risk management in competition settings.", "2865334": "Congratulations", "2875243": "Congrats @yuuniekiri for the winning. How do we filter out effective features and toxic features?Thank you for your response in advance.", "2875270": "As you can see in the article above, features that have a significant advantage in the CV score are kept, and features that have no advantage are removed.\nAdditionally, even if there is no significant gain in score, logically meaningful and stable features can be kept. This can also apply in the opposite case.", "2880567": "Nice share, Congratulations for your first solo gold.", "2889766": "Congratulations", "2904468": "Hi @yuuniekiri, I cannot find the final submission notebook for the first place. Can you please share it?", "3012991": "Thanks for your share!", "3135086": "Hi @yuuniekiri, Congratulations on your win. I love the way you approach Feature Engineering, especially Aggregation. I was wondering—if all feature names were hidden, how would you do in Aggregation?", "3312920": "Nice, keep doing well" }, "source": "meta" }