{ "id": 506552, "title": "Metric hacking doesn't work in my solution.", "url": "/competitions/home-credit-credit-risk-model-stability/discussion/506552", "author_name": "Takumi Mukaiyama", "post_date": "2024-05-22T09:53:48.939000", "votes": 5, "comment_count": 14, "views": 0, "content": "

In my notebook. metric hack like below doesn't work.
\nIs anyone else experiencing the same ?

\n

Public notebooks using metric hacking don't specify lgbm model parameters
\nThe simpler the model, the better the metric hacking works ？

\n

If you have any ideas, please share me.

\n
  \n\n  y_pred < \ny_pred[condition]  (y_pred[condition] - SHIFT).clip()\n
", "messages": [ { "id": 2828839, "postDate": "2024-05-22T09:53:48.940Z", "content": "

In my notebook. metric hack like below doesn't work.
\nIs anyone else experiencing the same ?

\n

Public notebooks using metric hacking don't specify lgbm model parameters
\nThe simpler the model, the better the metric hacking works ？

\n

If you have any ideas, please share me.

\n
  \n\n  y_pred < \ny_pred[condition]  (y_pred[condition] - SHIFT).clip()\n
", "rawMarkdown": "In my notebook. metric hack like below doesn't work.\nIs anyone else experiencing the same ?\n\nPublic notebooks using metric hacking don't specify lgbm model parameters\nThe simpler the model, the better the metric hacking works ？\n\nIf you have any ideas, please share me.\n\n```\nSHIFT = 0.073\n\ncondition = y_pred < 0.977\ny_pred[condition] = (y_pred[condition] - SHIFT).clip(0)\n```", "votes": 5 }, { "id": 2829059, "postDate": "2024-05-22T11:51:49.383Z", "content": "

I also had some problems integrating mh into my solution. In my case the problem was related to using my own dataset with full feature set for the final lgbm classifier. After recreating the dataset with only max aggregators (as in public notebooks; and with almost the same features) the problem was solved and LB increased.

", "rawMarkdown": "I also had some problems integrating mh into my solution. In my case the problem was related to using my own dataset with full feature set for the final lgbm classifier. After recreating the dataset with only max aggregators (as in public notebooks; and with almost the same features) the problem was solved and LB increased.", "votes": 3, "replies": [ { "id": 2829244, "postDate": "2024-05-22T13:54:29.023Z", "content": "

Did you change thereshold values ?

", "rawMarkdown": "Did you change thereshold values ?", "replies": [ { "id": 2829334, "postDate": "2024-05-22T14:34:41.537Z", "content": "

With my extended feature set, the metric hack didn't work with either public or custom thresholds, and in some experiments I even got scores equal to 0.

", "rawMarkdown": "With my extended feature set, the metric hack didn't work with either public or custom thresholds, and in some experiments I even got scores equal to 0." }, { "id": 2829339, "postDate": "2024-05-22T14:41:11.140Z", "content": "

I see….
\nThen, the metric can be effective only under certain conditions (simple data set?)
\nI'm starting to feel like it's over-fitting.

", "rawMarkdown": "I see....\nThen, the metric can be effective only under certain conditions (simple data set?)\nI'm starting to feel like it's over-fitting." }, { "id": 2829347, "postDate": "2024-05-22T14:50:59.997Z", "content": "

It seems some of the features (passed to lgbm adv classifier) are very poor at distinguishing between training and test items, moreover with default lgbm params like feature_fraction=1. So this can also be used as a hyperparameter to tune. But this does not apply to the main models which can use any features and public thresholds should work.

", "rawMarkdown": "It seems some of the features (passed to lgbm adv classifier) are very poor at distinguishing between training and test items, moreover with default lgbm params like feature_fraction=1. So this can also be used as a hyperparameter to tune. But this does not apply to the main models which can use any features and public thresholds should work." } ] } ] }, { "id": 2828955, "postDate": "2024-05-22T10:24:39.697Z", "content": "

Hello, I think I can help with your question on \"Using Metric Hacking to Boost Your Model's LB Score\". However, I'd like to know, what is the LB score of your model without using metric hacking?

", "rawMarkdown": "Hello, I think I can help with your question on \"Using Metric Hacking to Boost Your Model's LB Score\". However, I'd like to know, what is the LB score of your model without using metric hacking?", "replies": [ { "id": 2828980, "postDate": "2024-05-22T10:41:47.627Z", "content": "

@sunandray
\nHi.
\nHighest LB score without metric hacking is 0.591 in my model.

", "rawMarkdown": "@sunandray \nHi.\nHighest LB score without metric hacking is 0.591 in my model.\n", "replies": [ { "id": 2829020, "postDate": "2024-05-22T11:12:40.183Z", "content": "

When you say metric hacking doesn't work, you mean after metric hacking you score is still 0.591?

", "rawMarkdown": "When you say metric hacking doesn't work, you mean after metric hacking you score is still 0.591?" }, { "id": 2829031, "postDate": "2024-05-22T11:22:35.340Z", "content": "

after hacking, my score is worse than original(not hacking).

", "rawMarkdown": "after hacking, my score is worse than original(not hacking)." }, { "id": 2829039, "postDate": "2024-05-22T11:34:41.510Z", "content": "

Your approach isn't correct. You need to integrate the \"This is the way\" method, rather than just making minor adjustments to the prediction results at the end of your code.

", "rawMarkdown": "Your approach isn't correct. You need to integrate the \"This is the way\" method, rather than just making minor adjustments to the prediction results at the end of your code.", "votes": -1 }, { "id": 2829060, "postDate": "2024-05-22T11:52:07.043Z", "content": "

I think you may need to adjust thresholds that works for your model

", "rawMarkdown": "I think you may need to adjust thresholds that works for your model" }, { "id": 2829283, "postDate": "2024-05-22T14:08:33.953Z", "content": "

does \"Integrate\" means edit script1.py and 0.585.py or replace them to original code ?

\n

I changed script1.py to my code to use my data.pkl.

", "rawMarkdown": "does \"Integrate\" means edit script1.py and 0.585.py or replace them to original code ? \n\nI changed script1.py to my code to use my data.pkl." }, { "id": 2829688, "postDate": "2024-05-22T18:29:02.067Z", "content": "

Just the 585.py. Two issues: you’ll still need to calibrate the amount to detune from any matches relating to the training weeks, and, secondly, it is gambling to use the metric hack unless you have a very clever probing technique

", "rawMarkdown": "Just the 585.py. Two issues: you’ll still need to calibrate the amount to detune from any matches relating to the training weeks, and, secondly, it is gambling to use the metric hack unless you have a very clever probing technique" } ] } ] }, { "id": 2828998, "postDate": "2024-05-22T10:57:19.933Z", "rawMarkdown": "", "votes": -1, "isDeleted": true } ], "comments": [ { "id": 2829059, "author_name": "Andrey Nesterov", "author_url": "", "post_date": "2024-05-22T11:51:49.383000", "content": "

I also had some problems integrating mh into my solution. In my case the problem was related to using my own dataset with full feature set for the final lgbm classifier. After recreating the dataset with only max aggregators (as in public notebooks; and with almost the same features) the problem was solved and LB increased.

", "votes": 3, "replies": [ { "id": 2829244, "author_name": "Takumi Mukaiyama", "author_url": "", "post_date": "2024-05-22T13:54:29.023000", "content": "

Did you change thereshold values ?

", "votes": 0, "replies": [ { "id": 2829334, "author_name": "Andrey Nesterov", "author_url": "", "post_date": "2024-05-22T14:34:41.537000", "content": "

With my extended feature set, the metric hack didn't work with either public or custom thresholds, and in some experiments I even got scores equal to 0.

", "votes": 0, "replies": [] }, { "id": 2829339, "author_name": "Takumi Mukaiyama", "author_url": "", "post_date": "2024-05-22T14:41:11.140000", "content": "

I see….
\nThen, the metric can be effective only under certain conditions (simple data set?)
\nI'm starting to feel like it's over-fitting.

", "votes": 0, "replies": [] }, { "id": 2829347, "author_name": "Andrey Nesterov", "author_url": "", "post_date": "2024-05-22T14:50:59.997000", "content": "

It seems some of the features (passed to lgbm adv classifier) are very poor at distinguishing between training and test items, moreover with default lgbm params like feature_fraction=1. So this can also be used as a hyperparameter to tune. But this does not apply to the main models which can use any features and public thresholds should work.

", "votes": 0, "replies": [] } ] } ] }, { "id": 2828955, "author_name": "Yiyang Fu & Ruyi Ni", "author_url": "", "post_date": "2024-05-22T10:24:39.697000", "content": "

Hello, I think I can help with your question on \"Using Metric Hacking to Boost Your Model's LB Score\". However, I'd like to know, what is the LB score of your model without using metric hacking?

", "votes": 0, "replies": [ { "id": 2828980, "author_name": "Takumi Mukaiyama", "author_url": "", "post_date": "2024-05-22T10:41:47.627000", "content": "

@sunandray
\nHi.
\nHighest LB score without metric hacking is 0.591 in my model.

", "votes": 0, "replies": [ { "id": 2829020, "author_name": "Evan", "author_url": "", "post_date": "2024-05-22T11:12:40.183000", "content": "

When you say metric hacking doesn't work, you mean after metric hacking you score is still 0.591?

", "votes": 0, "replies": [] }, { "id": 2829031, "author_name": "Takumi Mukaiyama", "author_url": "", "post_date": "2024-05-22T11:22:35.340000", "content": "

after hacking, my score is worse than original(not hacking).

", "votes": 0, "replies": [] }, { "id": 2829039, "author_name": "Yiyang Fu & Ruyi Ni", "author_url": "", "post_date": "2024-05-22T11:34:41.510000", "content": "

Your approach isn't correct. You need to integrate the \"This is the way\" method, rather than just making minor adjustments to the prediction results at the end of your code.

", "votes": -1, "replies": [] }, { "id": 2829060, "author_name": "Evan", "author_url": "", "post_date": "2024-05-22T11:52:07.043000", "content": "

I think you may need to adjust thresholds that works for your model

", "votes": 0, "replies": [] }, { "id": 2829283, "author_name": "Takumi Mukaiyama", "author_url": "", "post_date": "2024-05-22T14:08:33.953000", "content": "

does \"Integrate\" means edit script1.py and 0.585.py or replace them to original code ?

\n

I changed script1.py to my code to use my data.pkl.

", "votes": 0, "replies": [] }, { "id": 2829688, "author_name": "Rob Freeman", "author_url": "", "post_date": "2024-05-22T18:29:02.067000", "content": "

Just the 585.py. Two issues: you’ll still need to calibrate the amount to detune from any matches relating to the training weeks, and, secondly, it is gambling to use the metric hack unless you have a very clever probing technique

", "votes": 0, "replies": [] } ] } ] }, { "id": 2828998, "author_name": "", "author_url": "", "post_date": "2024-05-22T10:57:19.933000", "content": "", "votes": -1, "replies": [] } ], "raw_markdown_by_id": { "2828839": "In my notebook. metric hack like below doesn't work.\nIs anyone else experiencing the same ?\n\nPublic notebooks using metric hacking don't specify lgbm model parameters\nThe simpler the model, the better the metric hacking works ？\n\nIf you have any ideas, please share me.\n\n```\nSHIFT = 0.073\n\ncondition = y_pred < 0.977\ny_pred[condition] = (y_pred[condition] - SHIFT).clip(0)\n```", "2829059": "I also had some problems integrating mh into my solution. In my case the problem was related to using my own dataset with full feature set for the final lgbm classifier. After recreating the dataset with only max aggregators (as in public notebooks; and with almost the same features) the problem was solved and LB increased.", "2828955": "Hello, I think I can help with your question on \"Using Metric Hacking to Boost Your Model's LB Score\". However, I'd like to know, what is the LB score of your model without using metric hacking?", "2828998": "" } }