I was trying to implement a customer objective function for the competition. As far as I understand, both LightGBM and XGBoost requires the diagonal part of the Hessian matrix, which is \n\nwhere k is the row index, and i is the column index, and l_k is the loss from the k-th row. In the log loss, this term seems to be \n\nwithout considering weights. But when I check the implementation of LightGBM and XGBoost, they calculate the hessian part using 2.0*p*(1.0-p).
So there is a factor of 2 here. I must be missing something here. Could anyone kindly indicate where am I wrong? Thanks a lot!
", "messages": [ { "id": "419855", "postDate": "11/12/2018 16:26:08", "content": "I was trying to implement a customer objective function for the competition. As far as I understand, both LightGBM and XGBoost requires the diagonal part of the Hessian matrix, which is \n\nwhere k is the row index, and i is the column index, and l_k is the loss from the k-th row. In the log loss, this term seems to be \n\nwithout considering weights. But when I check the implementation of LightGBM and XGBoost, they calculate the hessian part using 2.0*p*(1.0-p).
So there is a factor of 2 here. I must be missing something here. Could anyone kindly indicate where am I wrong? Thanks a lot!
", "rawMarkdown": "I was trying to implement a customer objective function for the competition. As far as I understand, both LightGBM and XGBoost requires the diagonal part of the Hessian matrix, which is \nHi lucaskg, did you find out how to implement Hessian matrix in LightGBM, if you can guide me how to calculate and implement Hessian matrix for current loss function in LightGBM. \nThanks!!
", "rawMarkdown": "Hi lucaskg, did you find out how to implement Hessian matrix in LightGBM, if you can guide me how to calculate and implement Hessian matrix for current loss function in LightGBM. \nThanks!!", "votes": null }, { "id": "434024", "postDate": "12/05/2018 19:41:41", "content": "seems like theres a kernel already implemented that? https://www.kaggle.com/mithrillion/know-your-objective
", "rawMarkdown": "seems like theres a kernel already implemented that? https://www.kaggle.com/mithrillion/know-your-objective", "votes": null }, { "id": "434032", "postDate": "12/05/2018 19:56:48", "content": "This kernel sets the hessian to 1. However you can calculate the hessian yourself by taking the derivative of the gradient again. LGB and XGB only need the diagonal of the hessian, not the full hessian matrix.
", "rawMarkdown": "This kernel sets the hessian to 1. However you can calculate the hessian yourself by taking the derivative of the gradient again. LGB and XGB only need the diagonal of the hessian, not the full hessian matrix.", "votes": null }, { "id": "434165", "postDate": "12/06/2018 01:45:11", "content": "Did you ever find a reference for this? Multiplying my hessian by a factor of 2 improved my CV by a small amount
", "rawMarkdown": "Did you ever find a reference for this? Multiplying my hessian by a factor of 2 improved my CV by a small amount", "votes": null }, { "id": "434303", "postDate": "12/06/2018 07:16:03", "content": "Hi Jonny, does calculating Hessian Analytically is improving the CV score ?
", "rawMarkdown": "Hi Jonny, does calculating Hessian Analytically is improving the CV score ?", "votes": null }, { "id": "434327", "postDate": "12/06/2018 07:50:03", "content": "Maybe this link is the answer. \nhttps://github.com/Microsoft/LightGBM/issues/1052
\n\nWe used correct hessian, but CV score does'nt improve much... . We may be mistaking something, but We don't yet understand it...
", "rawMarkdown": "Maybe this link is the answer. \nhttps://github.com/Microsoft/LightGBM/issues/1052\n\nWe used correct hessian, but CV score does'nt improve much... . We may be mistaking something, but We don't yet understand it...", "votes": null }, { "id": "434425", "postDate": "12/06/2018 11:23:54", "content": "Yes, I found an improvement in both CV and LB score. I have improved my model a lot since then, I don't know if the difference would be bigger or smaller now.
", "rawMarkdown": "Yes, I found an improvement in both CV and LB score. I have improved my model a lot since then, I don't know if the difference would be bigger or smaller now.", "votes": null }, { "id": "434431", "postDate": "12/06/2018 11:37:30", "content": "Keep in mind parameters like min child weight and the L1/L2 regularization values need to be tuned again after changing the hessian calculation, which could increase your improvement a little
", "rawMarkdown": "Keep in mind parameters like min child weight and the L1/L2 regularization values need to be tuned again after changing the hessian calculation, which could increase your improvement a little", "votes": null }, { "id": "434452", "postDate": "12/06/2018 12:29:03", "content": "Thanks a lot! We tried parameter tuning again , and CV score improved. Thanks for the helpful advice!
", "rawMarkdown": "Thanks a lot! We tried parameter tuning again , and CV score improved. Thanks for the helpful advice!", "votes": null }, { "id": "434509", "postDate": "12/06/2018 14:20:49", "content": "Hi Jonny, did you finally got it working with hard-coded Hessian? I tried autograd as well as manually implemented Hessian, but it never worked properly as expected. In the kernel I said that min_sum_hessian_in_leaf affects how hessians will change your fitted trees, but I still frequently got nonsensical results even after disabling it, so the problem is not really solved for me. I'd be really interested to know if you have successfully implemented the full objective function because I cannot really figure out what went wrong for me.
\n\nAlso, the way the XGB paper describes how they used Hessian seem to suggest that scale of Hessian should not matter, but why is 2x hessian giving better results?
", "rawMarkdown": "Hi Jonny, did you finally got it working with hard-coded Hessian? I tried autograd as well as manually implemented Hessian, but it never worked properly as expected. In the kernel I said that min_sum_hessian_in_leaf affects how hessians will change your fitted trees, but I still frequently got nonsensical results even after disabling it, so the problem is not really solved for me. I'd be really interested to know if you have successfully implemented the full objective function because I cannot really figure out what went wrong for me.\n\nAlso, the way the XGB paper describes how they used Hessian seem to suggest that scale of Hessian should not matter, but why is 2x hessian giving better results?", "votes": null }, { "id": "434514", "postDate": "12/06/2018 14:28:20", "content": "i used your kernal's objective function
\n\nhttps://www.kaggle.com/mithrillion/know-your-objective
\n\nbut i am lost here, it would be great if you people can help me with objective function !!
", "rawMarkdown": "i used your kernal's objective function\n\nhttps://www.kaggle.com/mithrillion/know-your-objective\n\nbut i am lost here, it would be great if you people can help me with objective function !!", "votes": null }, { "id": "434527", "postDate": "12/06/2018 14:39:40", "content": "You have to be a bit careful with the objective function in my kernel. It works fairly well in basic stratified folds, but the normalisation factors for different classes are actually not constant. If you actually want a more stable objective function, you might have to pass in the class counts in the training set to the objective function instead of calculating the counts from the set being evaluated. I think it's actually more convenient if you only implement the metric function and use the weight trick in some other kernels together with multi_log_loss to achieve the same goal. If you want more control over the objective function, the objective in my kernel with the modification I mentioned above should do the trick. Unfortunately I still don't have a proper Hessian...
", "rawMarkdown": "You have to be a bit careful with the objective function in my kernel. It works fairly well in basic stratified folds, but the normalisation factors for different classes are actually not constant. If you actually want a more stable objective function, you might have to pass in the class counts in the training set to the objective function instead of calculating the counts from the set being evaluated. I think it's actually more convenient if you only implement the metric function and use the weight trick in some other kernels together with multi\\_log\\_loss to achieve the same goal. If you want more control over the objective function, the objective in my kernel with the modification I mentioned above should do the trick. Unfortunately I still don't have a proper Hessian...", "votes": null }, { "id": "434541", "postDate": "12/06/2018 14:58:50", "content": "Using your kernel as a guide I was able to manually implement the gradient and hessian calculations. For a long time I was stuck because my training loss didn't decrease at all, so I thought I did something wrong. Then I realized L1 regularization of 0.1 was too high relative to the new size of the hessian, so I set it to zero and my model improved. I think it is because I correctly implemented the objective, but maybe it's because L1 regularization was hurting.
\n\nI will try to make a small kernel about it when I get bored of feature engineering, maybe Saturday.
", "rawMarkdown": "Using your kernel as a guide I was able to manually implement the gradient and hessian calculations. For a long time I was stuck because my training loss didn't decrease at all, so I thought I did something wrong. Then I realized L1 regularization of 0.1 was too high relative to the new size of the hessian, so I set it to zero and my model improved. I think it is because I correctly implemented the objective, but maybe it's because L1 regularization was hurting.\n\nI will try to make a small kernel about it when I get bored of feature engineering, maybe Saturday.", "votes": null }, { "id": "434598", "postDate": "12/06/2018 16:57:58", "content": "I'm still new to xgb/lgbm, this is probably a silly question. Why do people still try to implement the customized loss function (which requires output a gradient and hessian), when you can just pass in the desired class weights to the 'class_weight' parameter in the (sklearn api) LGBMClassifier with 'multiclass' as the objective?
", "rawMarkdown": "I'm still new to xgb/lgbm, this is probably a silly question. Why do people still try to implement the customized loss function (which requires output a gradient and hessian), when you can just pass in the desired class weights to the 'class_weight' parameter in the (sklearn api) LGBMClassifier with 'multiclass' as the objective?", "votes": null }, { "id": "434612", "postDate": "12/06/2018 17:15:29", "content": "Principled reason: I don't think the objective function is exactly equivalent to the multiclass objective with sample weights (maybe I am wrong), and we should get a better score if we can directly minimize what we are evaluated by.
\n\nHonest reason: I will try anything if there is some hint on the discussion forum or in kernels that it improved someone's score, even if I don't really understand why it helps :)
", "rawMarkdown": "Principled reason: I don't think the objective function is exactly equivalent to the multiclass objective with sample weights (maybe I am wrong), and we should get a better score if we can directly minimize what we are evaluated by.\n\nHonest reason: I will try anything if there is some hint on the discussion forum or in kernels that it improved someone's score, even if I don't really understand why it helps :)", "votes": null }, { "id": "434623", "postDate": "12/06/2018 17:28:58", "content": "I guess there are some benefits to a custom objective that you cannot get from using the weights, like changing the cutoff point of probability clipping, using the logsumexp trick to stabilise the calculation, etc. Nothing major, but could be helpful.
", "rawMarkdown": "I guess there are some benefits to a custom objective that you cannot get from using the weights, like changing the cutoff point of probability clipping, using the logsumexp trick to stabilise the calculation, etc. Nothing major, but could be helpful.", "votes": null }, { "id": "434853", "postDate": "12/07/2018 03:52:50", "content": "A followup question, do you know how does lightgbm implement the multiclass log_loss? I did a quick search online and looked at its code on github but seems like didn't find a clear place.
", "rawMarkdown": "A followup question, do you know how does lightgbm implement the multiclass log_loss? I did a quick search online and looked at its code on github but seems like didn't find a clear place.", "votes": null }, { "id": "434952", "postDate": "12/07/2018 07:54:04", "content": "The code is in c++ but the relevant part is fairly easy to understand:\n\n Common::Softmax(&rec);\n for (int k = 0; k < num_class_; ++k) {\n auto p = rec[k];\n size_t idx = static_cast<size_t>(num_data_) * k + i;\n if (label_int_[i] == k) {\n gradients[idx] = static_cast<score_t>((p - 1.0f) * weights_[i]);\n } else {\n gradients[idx] = static_cast<score_t>((p) * weights_[i]);\n }\n hessians[idx] = static_cast<score_t>((2.0f * p * (1.0f - p))* weights_[i]);\n }\n
Thanks for all the replies, I have yet another late follower-up question. So I am trying some of the public kernels thats using sample weights (instead of implementing a customized loss function), the weights they gave is calculated as such:
\n\nw = y.value_counts()\nsample_weights = {i : np.sum(w) / w[i] for i in w.index}\nwhy there is a np.sum(w), shouldn't it just be 1/w[i] according to the evaluation formula provided?
\n\n ... where N is the number of objects in the class set, ...\nBut if I just pass in /w[i] as the sample weights to lgbm fit, the loss (evaluated by Oliver's script) is a lot higher than the one with np.sum(w), does anyone have an intuition of why is this?
", "rawMarkdown": "Thanks for all the replies, I have yet another late follower-up question. So I am trying some of the public kernels thats using sample weights (instead of implementing a customized loss function), the weights they gave is calculated as such:\n\n w = y.value_counts()\n sample_weights = {i : np.sum(w) / w[i] for i in w.index}\n\nwhy there is a *np.sum(w)*, shouldn't it just be *1/w[i]* according to the evaluation formula provided? \n\n ... where N is the number of objects in the class set, ...\n\nBut if I just pass in */w[i]* as the sample weights to lgbm fit, the loss (evaluated by Oliver's script) is a lot higher than the one with *np.sum(w)*, does anyone have an intuition of why is this?", "votes": null }, { "id": "437473", "postDate": "12/12/2018 01:36:18", "content": "I am missing something here, my model seems it is not learning when using my custom objective function
", "rawMarkdown": "I am missing something here, my model seems it is not learning when using my custom objective function", "votes": null }, { "id": "437627", "postDate": "12/12/2018 08:14:22", "content": "Happens the same to me. It performs worse when using the custom objective; I'm missing something as well...
", "rawMarkdown": "Happens the same to me. It performs worse when using the custom objective; I'm missing something as well...", "votes": null }, { "id": "437689", "postDate": "12/12/2018 10:10:30", "content": "There are many tricky issues when implementing custom objective. Here is a list of roadblocks that took away a few hours of my life:\n1. 'F(ortran)' order vs 'C' order regarding in input and output of objective functions\n2. loss function scale (mean vs sum and how they affect learning rate)\n3. Hessian scale and min_sum_hessian_in_leaf\n4. Per-batch N normalisation factor vs static N based on the training set (should only really matter when doing minibatch or unstratified folds)
i manually calculated gradient and hessian. the loss is decreasing when hessian is set to np.ones but when i implement the second derivative equation the loss is at 0 value always
", "rawMarkdown": "i manually calculated gradient and hessian. the loss is decreasing when hessian is set to np.ones but when i implement the second derivative equation the loss is at 0 value always", "votes": null } ], "comments": [ { "id": 433892, "author_name": "mks2192", "author_url": "", "post_date": "12/05/2018 16:09:34", "content": "Hi lucaskg, did you find out how to implement Hessian matrix in LightGBM, if you can guide me how to calculate and implement Hessian matrix for current loss function in LightGBM. \nThanks!!
", "votes": null, "replies": [ { "id": 434024, "author_name": "samshipengs", "author_url": "", "post_date": "12/05/2018 19:41:41", "content": "seems like theres a kernel already implemented that? https://www.kaggle.com/mithrillion/know-your-objective
", "votes": null, "replies": [] }, { "id": 434032, "author_name": "jlomond", "author_url": "", "post_date": "12/05/2018 19:56:48", "content": "This kernel sets the hessian to 1. However you can calculate the hessian yourself by taking the derivative of the gradient again. LGB and XGB only need the diagonal of the hessian, not the full hessian matrix.
", "votes": null, "replies": [] }, { "id": 434303, "author_name": "mks2192", "author_url": "", "post_date": "12/06/2018 07:16:03", "content": "Hi Jonny, does calculating Hessian Analytically is improving the CV score ?
", "votes": null, "replies": [] }, { "id": 434425, "author_name": "jlomond", "author_url": "", "post_date": "12/06/2018 11:23:54", "content": "Yes, I found an improvement in both CV and LB score. I have improved my model a lot since then, I don't know if the difference would be bigger or smaller now.
", "votes": null, "replies": [] }, { "id": 434509, "author_name": "mithrillion", "author_url": "", "post_date": "12/06/2018 14:20:49", "content": "Hi Jonny, did you finally got it working with hard-coded Hessian? I tried autograd as well as manually implemented Hessian, but it never worked properly as expected. In the kernel I said that min_sum_hessian_in_leaf affects how hessians will change your fitted trees, but I still frequently got nonsensical results even after disabling it, so the problem is not really solved for me. I'd be really interested to know if you have successfully implemented the full objective function because I cannot really figure out what went wrong for me.
\n\nAlso, the way the XGB paper describes how they used Hessian seem to suggest that scale of Hessian should not matter, but why is 2x hessian giving better results?
", "votes": null, "replies": [] }, { "id": 434514, "author_name": "mks2192", "author_url": "", "post_date": "12/06/2018 14:28:20", "content": "i used your kernal's objective function
\n\nhttps://www.kaggle.com/mithrillion/know-your-objective
\n\nbut i am lost here, it would be great if you people can help me with objective function !!
", "votes": null, "replies": [] }, { "id": 434527, "author_name": "mithrillion", "author_url": "", "post_date": "12/06/2018 14:39:40", "content": "You have to be a bit careful with the objective function in my kernel. It works fairly well in basic stratified folds, but the normalisation factors for different classes are actually not constant. If you actually want a more stable objective function, you might have to pass in the class counts in the training set to the objective function instead of calculating the counts from the set being evaluated. I think it's actually more convenient if you only implement the metric function and use the weight trick in some other kernels together with multi_log_loss to achieve the same goal. If you want more control over the objective function, the objective in my kernel with the modification I mentioned above should do the trick. Unfortunately I still don't have a proper Hessian...
", "votes": null, "replies": [] }, { "id": 434541, "author_name": "jlomond", "author_url": "", "post_date": "12/06/2018 14:58:50", "content": "Using your kernel as a guide I was able to manually implement the gradient and hessian calculations. For a long time I was stuck because my training loss didn't decrease at all, so I thought I did something wrong. Then I realized L1 regularization of 0.1 was too high relative to the new size of the hessian, so I set it to zero and my model improved. I think it is because I correctly implemented the objective, but maybe it's because L1 regularization was hurting.
\n\nI will try to make a small kernel about it when I get bored of feature engineering, maybe Saturday.
", "votes": null, "replies": [] }, { "id": 434598, "author_name": "samshipengs", "author_url": "", "post_date": "12/06/2018 16:57:58", "content": "I'm still new to xgb/lgbm, this is probably a silly question. Why do people still try to implement the customized loss function (which requires output a gradient and hessian), when you can just pass in the desired class weights to the 'class_weight' parameter in the (sklearn api) LGBMClassifier with 'multiclass' as the objective?
", "votes": null, "replies": [] }, { "id": 434612, "author_name": "jlomond", "author_url": "", "post_date": "12/06/2018 17:15:29", "content": "Principled reason: I don't think the objective function is exactly equivalent to the multiclass objective with sample weights (maybe I am wrong), and we should get a better score if we can directly minimize what we are evaluated by.
\n\nHonest reason: I will try anything if there is some hint on the discussion forum or in kernels that it improved someone's score, even if I don't really understand why it helps :)
", "votes": null, "replies": [] }, { "id": 434623, "author_name": "mithrillion", "author_url": "", "post_date": "12/06/2018 17:28:58", "content": "I guess there are some benefits to a custom objective that you cannot get from using the weights, like changing the cutoff point of probability clipping, using the logsumexp trick to stabilise the calculation, etc. Nothing major, but could be helpful.
", "votes": null, "replies": [] }, { "id": 434853, "author_name": "samshipengs", "author_url": "", "post_date": "12/07/2018 03:52:50", "content": "A followup question, do you know how does lightgbm implement the multiclass log_loss? I did a quick search online and looked at its code on github but seems like didn't find a clear place.
", "votes": null, "replies": [] }, { "id": 434952, "author_name": "mithrillion", "author_url": "", "post_date": "12/07/2018 07:54:04", "content": "The code is in c++ but the relevant part is fairly easy to understand:\n\n Common::Softmax(&rec);\n for (int k = 0; k < num_class_; ++k) {\n auto p = rec[k];\n size_t idx = static_cast<size_t>(num_data_) * k + i;\n if (label_int_[i] == k) {\n gradients[idx] = static_cast<score_t>((p - 1.0f) * weights_[i]);\n } else {\n gradients[idx] = static_cast<score_t>((p) * weights_[i]);\n }\n hessians[idx] = static_cast<score_t>((2.0f * p * (1.0f - p))* weights_[i]);\n }\n
Thanks for all the replies, I have yet another late follower-up question. So I am trying some of the public kernels thats using sample weights (instead of implementing a customized loss function), the weights they gave is calculated as such:
\n\nw = y.value_counts()\nsample_weights = {i : np.sum(w) / w[i] for i in w.index}\nwhy there is a np.sum(w), shouldn't it just be 1/w[i] according to the evaluation formula provided?
\n\n ... where N is the number of objects in the class set, ...\nBut if I just pass in /w[i] as the sample weights to lgbm fit, the loss (evaluated by Oliver's script) is a lot higher than the one with np.sum(w), does anyone have an intuition of why is this?
", "votes": null, "replies": [] } ] }, { "id": 434165, "author_name": "jlomond", "author_url": "", "post_date": "12/06/2018 01:45:11", "content": "Did you ever find a reference for this? Multiplying my hessian by a factor of 2 improved my CV by a small amount
", "votes": null, "replies": [] }, { "id": 434327, "author_name": "ykaneko1992", "author_url": "", "post_date": "12/06/2018 07:50:03", "content": "Maybe this link is the answer. \nhttps://github.com/Microsoft/LightGBM/issues/1052
\n\nWe used correct hessian, but CV score does'nt improve much... . We may be mistaking something, but We don't yet understand it...
", "votes": null, "replies": [ { "id": 434431, "author_name": "jlomond", "author_url": "", "post_date": "12/06/2018 11:37:30", "content": "Keep in mind parameters like min child weight and the L1/L2 regularization values need to be tuned again after changing the hessian calculation, which could increase your improvement a little
", "votes": null, "replies": [] }, { "id": 434452, "author_name": "ykaneko1992", "author_url": "", "post_date": "12/06/2018 12:29:03", "content": "Thanks a lot! We tried parameter tuning again , and CV score improved. Thanks for the helpful advice!
", "votes": null, "replies": [] } ] }, { "id": 437473, "author_name": "niclasdoce", "author_url": "", "post_date": "12/12/2018 01:36:18", "content": "I am missing something here, my model seems it is not learning when using my custom objective function
", "votes": null, "replies": [ { "id": 437627, "author_name": "andreusancho", "author_url": "", "post_date": "12/12/2018 08:14:22", "content": "Happens the same to me. It performs worse when using the custom objective; I'm missing something as well...
", "votes": null, "replies": [] }, { "id": 437689, "author_name": "mithrillion", "author_url": "", "post_date": "12/12/2018 10:10:30", "content": "There are many tricky issues when implementing custom objective. Here is a list of roadblocks that took away a few hours of my life:\n1. 'F(ortran)' order vs 'C' order regarding in input and output of objective functions\n2. loss function scale (mean vs sum and how they affect learning rate)\n3. Hessian scale and min_sum_hessian_in_leaf\n4. Per-batch N normalisation factor vs static N based on the training set (should only really matter when doing minibatch or unstratified folds)
i manually calculated gradient and hessian. the loss is decreasing when hessian is set to np.ones but when i implement the second derivative equation the loss is at 0 value always
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "419855": "I was trying to implement a customer objective function for the competition. As far as I understand, both LightGBM and XGBoost requires the diagonal part of the Hessian matrix, which is \n