I have been doing feature engineering and while it leads to an improved CV AUC score, my overall score on public leaderboard is not improving. I am thinking of letting my model optimize the custom metric instead of AUC score but have no idea how to achieve it.
\nAny guidance or resources would be highly appreciated
\nThank You.
I have been doing feature engineering and while it leads to an improved CV AUC score, my overall score on public leaderboard is not improving. I am thinking of letting my model optimize the custom metric instead of AUC score but have no idea how to achieve it.
\nAny guidance or resources would be highly appreciated
\nThank You.
The main problem in making a custom loss function as per the metrics for lgbm is to computing the hessian. after some struggle I as able to compete it by it wold take half a second to compute each value and the hessian is a matrics. so its like calculating the loss and for it u need to calculate a average gini, than the slope, for it to pass a line through the points of than its slope, pls the residuals std. all being too computationally expensive to calculate the hessian. again its my opinion, if there is a better way than can be tried. some of ways to try can include combining pytorch or tensorflow to calculate the hessian, or to replace a the metrics with an equivalent metics whose computation is easier and faster. but I than again I feel training with auc is fine and stability needs to be incorporated in a different way.
", "rawMarkdown": "The main problem in making a custom loss function as per the metrics for lgbm is to computing the hessian. after some struggle I as able to compete it by it wold take half a second to compute each value and the hessian is a matrics. so its like calculating the loss and for it u need to calculate a average gini, than the slope, for it to pass a line through the points of than its slope, pls the residuals std. all being too computationally expensive to calculate the hessian. again its my opinion, if there is a better way than can be tried. some of ways to try can include combining pytorch or tensorflow to calculate the hessian, or to replace a the metrics with an equivalent metics whose computation is easier and faster. but I than again I feel training with auc is fine and stability needs to be incorporated in a different way.", "votes": null }, { "id": "2757254", "postDate": "04/17/2024 12:12:13", "content": "Can you share your implementation ?
", "rawMarkdown": "Can you share your implementation ?", "votes": null }, { "id": "2757474", "postDate": "04/17/2024 14:27:15", "content": "https://www.kaggle.com/code/shreyas9181/home-credit-risk-lightgbm-custom
\nIn this notebook I was trying to define a custom loss function for lgbm, the idea was to just to make it working and then figure out a way to make it efficient.
\nIn there there will be a function defined as custom loss, mostly to compute the loss and hessian. My plan was to use other libraries. loss is still straight forward, but for hessian I was trying to merge any library that can calculate it, like tensorflow or pytorch or scipy, My hope was on pytorch or tensorflow to calculate the hessian of the custom loss function, but I guess I got bit of success with scipy library. but it was too slow, that I decided to give up the attempt.
\nThen the other idea was to hypertune the lgbm model on the custom stability metrics. initially I
\nThis notebook was one of the version, not sure if this version was the one without any bugs, but it should give a rough idea of the attempt, thought it will help a lot, but found not much difference than, so now experimenting other ideas for stability, and leaving the training part as it is.
", "rawMarkdown": "https://www.kaggle.com/code/shreyas9181/home-credit-risk-lightgbm-custom\n\nIn this notebook I was trying to define a custom loss function for lgbm, the idea was to just to make it working and then figure out a way to make it efficient.\n\nIn there there will be a function defined as custom loss, mostly to compute the loss and hessian. My plan was to use other libraries. loss is still straight forward, but for hessian I was trying to merge any library that can calculate it, like tensorflow or pytorch or scipy, My hope was on pytorch or tensorflow to calculate the hessian of the custom loss function, but I guess I got bit of success with scipy library. but it was too slow, that I decided to give up the attempt.\n\nThen the other idea was to hypertune the lgbm model on the custom stability metrics. initially I \n\n\nThis notebook was one of the version, not sure if this version was the one without any bugs, but it should give a rough idea of the attempt, thought it will help a lot, but found not much difference than, so now experimenting other ideas for stability, and leaving the training part as it is.", "votes": null } ], "comments": [ { "id": 2754137, "author_name": "shreyas9181", "author_url": "", "post_date": "04/15/2024 20:37:45", "content": "The main problem in making a custom loss function as per the metrics for lgbm is to computing the hessian. after some struggle I as able to compete it by it wold take half a second to compute each value and the hessian is a matrics. so its like calculating the loss and for it u need to calculate a average gini, than the slope, for it to pass a line through the points of than its slope, pls the residuals std. all being too computationally expensive to calculate the hessian. again its my opinion, if there is a better way than can be tried. some of ways to try can include combining pytorch or tensorflow to calculate the hessian, or to replace a the metrics with an equivalent metics whose computation is easier and faster. but I than again I feel training with auc is fine and stability needs to be incorporated in a different way.
", "votes": null, "replies": [ { "id": 2757254, "author_name": "serjhenrique", "author_url": "", "post_date": "04/17/2024 12:12:13", "content": "Can you share your implementation ?
", "votes": null, "replies": [ { "id": 2757474, "author_name": "shreyas9181", "author_url": "", "post_date": "04/17/2024 14:27:15", "content": "https://www.kaggle.com/code/shreyas9181/home-credit-risk-lightgbm-custom
\nIn this notebook I was trying to define a custom loss function for lgbm, the idea was to just to make it working and then figure out a way to make it efficient.
\nIn there there will be a function defined as custom loss, mostly to compute the loss and hessian. My plan was to use other libraries. loss is still straight forward, but for hessian I was trying to merge any library that can calculate it, like tensorflow or pytorch or scipy, My hope was on pytorch or tensorflow to calculate the hessian of the custom loss function, but I guess I got bit of success with scipy library. but it was too slow, that I decided to give up the attempt.
\nThen the other idea was to hypertune the lgbm model on the custom stability metrics. initially I
\nThis notebook was one of the version, not sure if this version was the one without any bugs, but it should give a rough idea of the attempt, thought it will help a lot, but found not much difference than, so now experimenting other ideas for stability, and leaving the training part as it is.
", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2754110": "I have been doing feature engineering and while it leads to an improved CV AUC score, my overall score on public leaderboard is not improving. I am thinking of letting my model optimize the custom metric instead of AUC score but have no idea how to achieve it. \nAny guidance or resources would be highly appreciated\nThank You.", "2754137": "The main problem in making a custom loss function as per the metrics for lgbm is to computing the hessian. after some struggle I as able to compete it by it wold take half a second to compute each value and the hessian is a matrics. so its like calculating the loss and for it u need to calculate a average gini, than the slope, for it to pass a line through the points of than its slope, pls the residuals std. all being too computationally expensive to calculate the hessian. again its my opinion, if there is a better way than can be tried. some of ways to try can include combining pytorch or tensorflow to calculate the hessian, or to replace a the metrics with an equivalent metics whose computation is easier and faster. but I than again I feel training with auc is fine and stability needs to be incorporated in a different way.", "2757254": "Can you share your implementation ?", "2757474": "https://www.kaggle.com/code/shreyas9181/home-credit-risk-lightgbm-custom\n\nIn this notebook I was trying to define a custom loss function for lgbm, the idea was to just to make it working and then figure out a way to make it efficient.\n\nIn there there will be a function defined as custom loss, mostly to compute the loss and hessian. My plan was to use other libraries. loss is still straight forward, but for hessian I was trying to merge any library that can calculate it, like tensorflow or pytorch or scipy, My hope was on pytorch or tensorflow to calculate the hessian of the custom loss function, but I guess I got bit of success with scipy library. but it was too slow, that I decided to give up the attempt.\n\nThen the other idea was to hypertune the lgbm model on the custom stability metrics. initially I \n\n\nThis notebook was one of the version, not sure if this version was the one without any bugs, but it should give a rough idea of the attempt, thought it will help a lot, but found not much difference than, so now experimenting other ideas for stability, and leaving the training part as it is." }, "source": "meta" }