Hi kagglers! In many notebooks here, I saw that If we want to make custom objective for optimizing kappa we can use something like this. Can somebody explain what are \"a\" and \"b\" constants and how they was calculated? Is it something to scale the target distribution or what ?
\na = 2.998
\nb = 1.092
\ndef MakeObj(y_true, y_pred):
\n labels = y_true + a
\n preds = y_pred + a
\n preds = preds.clip(0, np.inf)
\n f = 1/2np.sum((preds-labels)2)\n g = 1/2np.sum((preds-a)2+b)
\n df = preds - labels
\n dg = preds - a
\n grad = (df/g - fdg/g2)len(labels)
\n hess = np.ones(len(labels))
\n return grad, hess
P.S. for this example thanks to @ravi20076
", "messages": [ { "id": "2996578", "postDate": "09/23/2024 16:43:41", "content": "Hi kagglers! In many notebooks here, I saw that If we want to make custom objective for optimizing kappa we can use something like this. Can somebody explain what are \"a\" and \"b\" constants and how they was calculated? Is it something to scale the target distribution or what ?
\na = 2.998
\nb = 1.092
\ndef MakeObj(y_true, y_pred):
\n labels = y_true + a
\n preds = y_pred + a
\n preds = preds.clip(0, np.inf)
\n f = 1/2np.sum((preds-labels)2)\n g = 1/2np.sum((preds-a)2+b)
\n df = preds - labels
\n dg = preds - a
\n grad = (df/g - fdg/g2)len(labels)
\n hess = np.ones(len(labels))
\n return grad, hess
P.S. for this example thanks to @ravi20076
", "rawMarkdown": "Hi kagglers! In many notebooks here, I saw that If we want to make custom objective for optimizing kappa we can use something like this. Can somebody explain what are \"a\" and \"b\" constants and how they was calculated? Is it something to scale the target distribution or what ? \n\na = 2.998\nb = 1.092\ndef MakeObj(y_true, y_pred):\n labels = y_true + a\n preds = y_pred + a\n preds = preds.clip(0, np.inf)\n f = 1/2*np.sum((preds-labels)**2)\n g = 1/2*np.sum((preds-a)**2+b)\n df = preds - labels\n dg = preds - a\n grad = (df/g - f*dg/g**2)*len(labels)\n hess = np.ones(len(labels))\n return grad, hess\n\nP.S. for this example thanks to @ravi20076", "votes": null }, { "id": "2996703", "postDate": "09/23/2024 20:38:21", "content": "@alexshumilov I simply took them from another competition artefacts. They were used there to good effect
", "rawMarkdown": "alexshumilov I simply took them from another competition artefacts. They were used there to good effect", "votes": null }, { "id": "2996758", "postDate": "09/23/2024 21:45:36", "content": "According to comment here in Essay Scoring competition, a is mean of train target and b is std of train target. And the following blog here explains why
The derivation is a straightforward calculation. In standard notations,
\n\\[
\nO_{ij}=\\sum_k\\delta_{iy_k}\\delta_{j\\hat{y_k}}
\n\\]
\n\\[
\nE_{ij}=p_i\\sum_k\\delta_{j\\hat{y_k}}
\n\\]
\nwhere \\(p_i\\) is the proportion of label \\(i\\) in the training set so that \\(\\sum_ip_i=1\\). Then
\n\\[
\n\\mathfrak{f}=\\sum_{i,j}(i-j)^2O_{ij}=\\sum_k\\sum_{i,j}(i-j)^2\\delta_{iy_k}\\delta_{j\\hat{y_k}}=\\sum_k(y_k-\\hat{y_k})^2
\n\\]
\n\\[
\n\\mathfrak{g}=\\sum_{i,j}(i-j)^2E_{ij}=\\sum_k\\sum_ip_i(i-\\hat{y_k})^2=\\sum_k\\mathbb{E}_y[(\\hat{y_k}-y)^2]=\\sum_k\\left((\\hat{y_k}-a)^2+b\\right)
\n\\]
where \\(a\\) and \\(b\\) are respectively the mean and variance of \\(y_{train}\\). This follows from the identity \\(\\mathbb{E}_x[(t-x)^2]=(t-\\mathbb{E}[x])^2+\\mbox{Var}[x]\\). Then \\(1-\\kappa=\\mathfrak{f}/\\mathfrak{g}\\) which is a rational function of \\(\\hat{y_k}\\). Interestingly, it extends QWK to real-valued \\(\\hat{y_k}\\), and we can start taking the gradient and hessian with respect to \\(\\hat{y_k}\\).
\nThe gradient \\(g\\) should just be \\(\\frac{\\mathfrak{f}'}{\\mathfrak{g}}-\\frac{\\mathfrak{f}\\mathfrak{g}'}{\\mathfrak{g}^2}\\). I am not sure why the scaling factor len(labels) is needed; it would affect the learning rate. The hessian \\(h\\) can also be analytically computed, but there is no reason to believe that it would be bounded below by some positive number. In practice, we only need \\(const. + g\\cdot y_{pred}+\\frac{1}{2}h\\cdot y_{pred}^2\\) to be a majorant of the objective function (see the explanation from the inventor of xgboost). Perhaps scaling the gradient by len(labels) and setting the hessian to \\(1\\) can make this happen.
@alexshumilov I simply took them from another competition artefacts. They were used there to good effect
", "votes": null, "replies": [] }, { "id": 2996758, "author_name": "cdeotte", "author_url": "", "post_date": "09/23/2024 21:45:36", "content": "According to comment here in Essay Scoring competition, a is mean of train target and b is std of train target. And the following blog here explains why
The derivation is a straightforward calculation. In standard notations,
\n\\[
\nO_{ij}=\\sum_k\\delta_{iy_k}\\delta_{j\\hat{y_k}}
\n\\]
\n\\[
\nE_{ij}=p_i\\sum_k\\delta_{j\\hat{y_k}}
\n\\]
\nwhere \\(p_i\\) is the proportion of label \\(i\\) in the training set so that \\(\\sum_ip_i=1\\). Then
\n\\[
\n\\mathfrak{f}=\\sum_{i,j}(i-j)^2O_{ij}=\\sum_k\\sum_{i,j}(i-j)^2\\delta_{iy_k}\\delta_{j\\hat{y_k}}=\\sum_k(y_k-\\hat{y_k})^2
\n\\]
\n\\[
\n\\mathfrak{g}=\\sum_{i,j}(i-j)^2E_{ij}=\\sum_k\\sum_ip_i(i-\\hat{y_k})^2=\\sum_k\\mathbb{E}_y[(\\hat{y_k}-y)^2]=\\sum_k\\left((\\hat{y_k}-a)^2+b\\right)
\n\\]
where \\(a\\) and \\(b\\) are respectively the mean and variance of \\(y_{train}\\). This follows from the identity \\(\\mathbb{E}_x[(t-x)^2]=(t-\\mathbb{E}[x])^2+\\mbox{Var}[x]\\). Then \\(1-\\kappa=\\mathfrak{f}/\\mathfrak{g}\\) which is a rational function of \\(\\hat{y_k}\\). Interestingly, it extends QWK to real-valued \\(\\hat{y_k}\\), and we can start taking the gradient and hessian with respect to \\(\\hat{y_k}\\).
\nThe gradient \\(g\\) should just be \\(\\frac{\\mathfrak{f}'}{\\mathfrak{g}}-\\frac{\\mathfrak{f}\\mathfrak{g}'}{\\mathfrak{g}^2}\\). I am not sure why the scaling factor len(labels) is needed; it would affect the learning rate. The hessian \\(h\\) can also be analytically computed, but there is no reason to believe that it would be bounded below by some positive number. In practice, we only need \\(const. + g\\cdot y_{pred}+\\frac{1}{2}h\\cdot y_{pred}^2\\) to be a majorant of the objective function (see the explanation from the inventor of xgboost). Perhaps scaling the gradient by len(labels) and setting the hessian to \\(1\\) can make this happen.