I shared in a previous competition how to improve mcc by selecting the right threshold, see https://www.kaggle.com/cpmpml/optimizing-probabilities-for-best-mcc
\n\nThe code is fast enough to be used as an evaluation function in XGBoost. Adapting to LightGBM is straightforwxard.
", "messages": [ { "id": "450711", "postDate": "01/05/2019 15:03:44", "content": "I shared in a previous competition how to improve mcc by selecting the right threshold, see https://www.kaggle.com/cpmpml/optimizing-probabilities-for-best-mcc
\n\nThe code is fast enough to be used as an evaluation function in XGBoost. Adapting to LightGBM is straightforwxard.
", "rawMarkdown": "I shared in a previous competition how to improve mcc by selecting the right threshold, see https://www.kaggle.com/cpmpml/optimizing-probabilities-for-best-mcc\n\nThe code is fast enough to be used as an evaluation function in XGBoost. Adapting to LightGBM is straightforwxard.", "votes": null }, { "id": "450777", "postDate": "01/05/2019 18:57:13", "content": "Thanks @CPMP, let me ask: how can I use this for my test predictions?
", "rawMarkdown": "Thanks @CPMP, let me ask: how can I use this for my test predictions?", "votes": null }, { "id": "450808", "postDate": "01/05/2019 20:09:27", "content": "You can use the threshold for which mcc is maximal on train.
", "rawMarkdown": "You can use the threshold for which mcc is maximal on train.", "votes": null }, { "id": "451378", "postDate": "01/07/2019 01:27:35", "content": "Hi @cpmpml, will you join this competition?
", "rawMarkdown": "Hi @cpmpml, will you join this competition?", "votes": null }, { "id": "451793", "postDate": "01/07/2019 16:56:23", "content": "I hope so.
", "rawMarkdown": "I hope so.", "votes": null }, { "id": "451970", "postDate": "01/08/2019 01:42:18", "content": "Nobody is giving good tips much more like you (except host), so this competition is hard for newbie.
", "rawMarkdown": "Nobody is giving good tips much more like you (except host), so this competition is hard for newbie.", "votes": null }, { "id": "452174", "postDate": "01/08/2019 10:18:12", "content": "I tried to use a custom evaluation function with LightGBM but it performed worse. I also tried to search for the optimal threshold on the out-of-fold data and use it on the test but it was equal to 0.5 so there was no added benefit.
\n\nThanks though! I'll look back into this further down the road.
", "rawMarkdown": "I tried to use a custom evaluation function with LightGBM but it performed worse. I also tried to search for the optimal threshold on the out-of-fold data and use it on the test but it was equal to 0.5 so there was no added benefit.\n\nThanks though! I'll look back into this further down the road.", "votes": null }, { "id": "452322", "postDate": "01/08/2019 15:12:43", "content": "\n\n\nI tried to use a custom evaluation function with LightGBM but it performed worse
\n
What do you mean? Did you use the code I share or something else? Also, what do you mean by: \"it performed worse\"?
", "rawMarkdown": "> I tried to use a custom evaluation function with LightGBM but it performed worse\n\nWhat do you mean? Did you use the code I share or something else? Also, what do you mean by: \"it performed worse\"?", "votes": null }, { "id": "452357", "postDate": "01/08/2019 16:05:07", "content": "Yes I used the code you proposed. Specifically I wrapped your eval_mcc function so that I could feed it to the eval_metric parameter of LightGBM.
By worse I mean that my local out-of-fold CV as well as my leaderboard score went down. I'm wondering if it this because there are too few rows... Maybe that a weighted average between the \"optimal\" threshold and a prior of 0.5 with some heuristic weighting could work. To be honest I haven't spent as much time as I would have liked on this. At the moment I'm mostly focused on extracting features!
", "rawMarkdown": "Yes I used the code you proposed. Specifically I wrapped your `eval_mcc` function so that I could feed it to the `eval_metric` parameter of LightGBM.\n\nBy worse I mean that my local out-of-fold CV as well as my leaderboard score went down. I'm wondering if it this because there are too few rows... Maybe that a weighted average between the \"optimal\" threshold and a prior of 0.5 with some heuristic weighting could work. To be honest I haven't spent as much time as I would have liked on this. At the moment I'm mostly focused on extracting features!", "votes": null }, { "id": "452397", "postDate": "01/08/2019 17:14:29", "content": "Thanks for the feedback.
", "rawMarkdown": "Thanks for the feedback.", "votes": null }, { "id": "452414", "postDate": "01/08/2019 17:33:54", "content": "You're setting high expectations on me here ;) I think I will enter it next week, this week I am traveling.
", "rawMarkdown": "You're setting high expectations on me here ;) I think I will enter it next week, this week I am traveling.", "votes": null }, { "id": "471950", "postDate": "02/15/2019 06:15:04", "content": "Thanks for sharing.
", "rawMarkdown": "Thanks for sharing.", "votes": null } ], "comments": [ { "id": 450777, "author_name": "joaopmpeinado", "author_url": "", "post_date": "01/05/2019 18:57:13", "content": "Thanks @CPMP, let me ask: how can I use this for my test predictions?
", "votes": null, "replies": [ { "id": 450808, "author_name": "cpmpml", "author_url": "", "post_date": "01/05/2019 20:09:27", "content": "You can use the threshold for which mcc is maximal on train.
", "votes": null, "replies": [] } ] }, { "id": 451378, "author_name": "onodera", "author_url": "", "post_date": "01/07/2019 01:27:35", "content": "Hi @cpmpml, will you join this competition?
", "votes": null, "replies": [ { "id": 451793, "author_name": "cpmpml", "author_url": "", "post_date": "01/07/2019 16:56:23", "content": "I hope so.
", "votes": null, "replies": [] }, { "id": 451970, "author_name": "onodera", "author_url": "", "post_date": "01/08/2019 01:42:18", "content": "Nobody is giving good tips much more like you (except host), so this competition is hard for newbie.
", "votes": null, "replies": [] }, { "id": 452414, "author_name": "cpmpml", "author_url": "", "post_date": "01/08/2019 17:33:54", "content": "You're setting high expectations on me here ;) I think I will enter it next week, this week I am traveling.
", "votes": null, "replies": [] }, { "id": 471950, "author_name": "longyin2", "author_url": "", "post_date": "02/15/2019 06:15:04", "content": "Thanks for sharing.
", "votes": null, "replies": [] } ] }, { "id": 452174, "author_name": "maxhalford", "author_url": "", "post_date": "01/08/2019 10:18:12", "content": "I tried to use a custom evaluation function with LightGBM but it performed worse. I also tried to search for the optimal threshold on the out-of-fold data and use it on the test but it was equal to 0.5 so there was no added benefit.
\n\nThanks though! I'll look back into this further down the road.
", "votes": null, "replies": [ { "id": 452322, "author_name": "cpmpml", "author_url": "", "post_date": "01/08/2019 15:12:43", "content": "\n\n\nI tried to use a custom evaluation function with LightGBM but it performed worse
\n
What do you mean? Did you use the code I share or something else? Also, what do you mean by: \"it performed worse\"?
", "votes": null, "replies": [] }, { "id": 452357, "author_name": "maxhalford", "author_url": "", "post_date": "01/08/2019 16:05:07", "content": "Yes I used the code you proposed. Specifically I wrapped your eval_mcc function so that I could feed it to the eval_metric parameter of LightGBM.
By worse I mean that my local out-of-fold CV as well as my leaderboard score went down. I'm wondering if it this because there are too few rows... Maybe that a weighted average between the \"optimal\" threshold and a prior of 0.5 with some heuristic weighting could work. To be honest I haven't spent as much time as I would have liked on this. At the moment I'm mostly focused on extracting features!
", "votes": null, "replies": [] }, { "id": 452397, "author_name": "cpmpml", "author_url": "", "post_date": "01/08/2019 17:14:29", "content": "Thanks for the feedback.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "450711": "I shared in a previous competition how to improve mcc by selecting the right threshold, see https://www.kaggle.com/cpmpml/optimizing-probabilities-for-best-mcc\n\nThe code is fast enough to be used as an evaluation function in XGBoost. Adapting to LightGBM is straightforwxard.", "450777": "Thanks @CPMP, let me ask: how can I use this for my test predictions?", "450808": "You can use the threshold for which mcc is maximal on train.", "451378": "Hi @cpmpml, will you join this competition?", "451793": "I hope so.", "451970": "Nobody is giving good tips much more like you (except host), so this competition is hard for newbie.", "452174": "I tried to use a custom evaluation function with LightGBM but it performed worse. I also tried to search for the optimal threshold on the out-of-fold data and use it on the test but it was equal to 0.5 so there was no added benefit.\n\nThanks though! I'll look back into this further down the road.", "452322": "> I tried to use a custom evaluation function with LightGBM but it performed worse\n\nWhat do you mean? Did you use the code I share or something else? Also, what do you mean by: \"it performed worse\"?", "452357": "Yes I used the code you proposed. Specifically I wrapped your `eval_mcc` function so that I could feed it to the `eval_metric` parameter of LightGBM.\n\nBy worse I mean that my local out-of-fold CV as well as my leaderboard score went down. I'm wondering if it this because there are too few rows... Maybe that a weighted average between the \"optimal\" threshold and a prior of 0.5 with some heuristic weighting could work. To be honest I haven't spent as much time as I would have liked on this. At the moment I'm mostly focused on extracting features!", "452397": "Thanks for the feedback.", "452414": "You're setting high expectations on me here ;) I think I will enter it next week, this week I am traveling.", "471950": "Thanks for sharing." }, "source": "meta" }