In this topic and notebook I proposed a tentative loss function based on ranking to penalize both the Gradient and the Hessian of the positive cases that are further away from the top positions.
\nHowever, as @davidirudel commented here I was not correctly multiplying the weights to the hessian.
\nIf the loss for one case is L and p(x) the sigmoid function. I want to multiply the weight alpha >1.0 which is proportional to the rank position for each true positive.
\nThen:
\n$$L = -\\alpha y \\ln p - (1-y) \\ln(1-p)$$
\n$$\\frac{\\partial p}{\\partial x} = p(1-p)$$
\n$$\\text{grad} = \\frac{\\partial L}{\\partial x} = \\frac{\\partial L}{\\partial p} \\frac{\\partial p}{\\partial x} = p(1 + \\alpha y - y) - \\alpha y$$
\n$$\\text{hess} = \\frac{\\partial^2 L}{\\partial x^2} = (1 + \\alpha y - y)p(1-p) $$
\nI think, the tentative version 2 of the Weighted LogLoss Function based on rank could be:
\ndef weighted_logloss(preds, dtrain):\n global MULT_NO4PERC, MAX_WEIGHTS\n eps = 1e-16\n labels = dtrain.get_label()\n preds = 1.0 / (1.0 + np.exp(-preds))\n\n # top 4 perc\n labels_mat = np.transpose(np.array([np.arange(len(labels)), labels, preds]))\n pos_ord = labels_mat[:, 2].argsort()[::-1]\n labels_mat = labels_mat[pos_ord]\n weights_4perc = np.where(labels_mat[:,1]==0, 20, 1)\n top4 = np.cumsum(weights_4perc) <= int(0.04 * np.sum(weights_4perc))\n top4 = top4[labels_mat[:, 0].argsort()]\n\n weights = 1+np.exp(-MULT_NO4PERC*np.linspace(MAX_WEIGHTS-1,0,len(top4)))[labels_mat[:, 0].argsort()]\n # Set to one weights of positive labels in top 4perc\n weights[top4 & (labels==1.0)] = 1.0 \n # Set to one weights of negative labels\n weights[(labels==0.0)] = 1.0 \n\n grad = preds * (1 + weights*labels - labels) - (weights * labels)\n hess = np.maximum(preds * (1.0 - preds) * (1 + weights*labels - labels) , eps)\n\n return grad, hess\nIn this topic and notebook I proposed a tentative loss function based on ranking to penalize both the Gradient and the Hessian of the positive cases that are further away from the top positions.
\nHowever, as @davidirudel commented here I was not correctly multiplying the weights to the hessian.
\nIf the loss for one case is L and p(x) the sigmoid function. I want to multiply the weight alpha >1.0 which is proportional to the rank position for each true positive.
\nThen:
\n$$L = -\\alpha y \\ln p - (1-y) \\ln(1-p)$$
\n$$\\frac{\\partial p}{\\partial x} = p(1-p)$$
\n$$\\text{grad} = \\frac{\\partial L}{\\partial x} = \\frac{\\partial L}{\\partial p} \\frac{\\partial p}{\\partial x} = p(1 + \\alpha y - y) - \\alpha y$$
\n$$\\text{hess} = \\frac{\\partial^2 L}{\\partial x^2} = (1 + \\alpha y - y)p(1-p) $$
\nI think, the tentative version 2 of the Weighted LogLoss Function based on rank could be:
\ndef weighted_logloss(preds, dtrain):\n global MULT_NO4PERC, MAX_WEIGHTS\n eps = 1e-16\n labels = dtrain.get_label()\n preds = 1.0 / (1.0 + np.exp(-preds))\n\n # top 4 perc\n labels_mat = np.transpose(np.array([np.arange(len(labels)), labels, preds]))\n pos_ord = labels_mat[:, 2].argsort()[::-1]\n labels_mat = labels_mat[pos_ord]\n weights_4perc = np.where(labels_mat[:,1]==0, 20, 1)\n top4 = np.cumsum(weights_4perc) <= int(0.04 * np.sum(weights_4perc))\n top4 = top4[labels_mat[:, 0].argsort()]\n\n weights = 1+np.exp(-MULT_NO4PERC*np.linspace(MAX_WEIGHTS-1,0,len(top4)))[labels_mat[:, 0].argsort()]\n # Set to one weights of positive labels in top 4perc\n weights[top4 & (labels==1.0)] = 1.0 \n # Set to one weights of negative labels\n weights[(labels==0.0)] = 1.0 \n\n grad = preds * (1 + weights*labels - labels) - (weights * labels)\n hess = np.maximum(preds * (1.0 - preds) * (1 + weights*labels - labels) , eps)\n\n return grad, hess\nI think hessian is wrong.
\nCan you please confirm?
$$\\text{grad} = p(1 + \\alpha y - y) - \\alpha y$$
\n$$\\text{hess} = \\frac{\\partial grad}{\\partial x} = \\frac{\\partial grad}{\\partial p} \\frac{\\partial p}{\\partial x} = (1 + \\alpha y - y) p (1 - p) $$
", "rawMarkdown": "$$\\text{grad} = p(1 + \\alpha y - y) - \\alpha y$$\n\n$$\\text{hess} = \\frac{\\partial grad}{\\partial x} = \\frac{\\partial grad}{\\partial p} \\frac{\\partial p}{\\partial x} = (1 + \\alpha y - y) p (1 - p) $$", "votes": null }, { "id": "1902986", "postDate": "08/17/2022 03:28:03", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!", "votes": null }, { "id": "1903256", "postDate": "08/17/2022 08:32:26", "content": "Why do you use maximum() operation for hessian in the code?
I think hessian is wrong.
\nCan you please confirm?
$$\\text{grad} = p(1 + \\alpha y - y) - \\alpha y$$
\n$$\\text{hess} = \\frac{\\partial grad}{\\partial x} = \\frac{\\partial grad}{\\partial p} \\frac{\\partial p}{\\partial x} = (1 + \\alpha y - y) p (1 - p) $$
", "votes": null, "replies": [] }, { "id": 1902986, "author_name": "leewook", "author_url": "", "post_date": "08/17/2022 03:28:03", "content": "Thanks for sharing!
", "votes": null, "replies": [] } ] }, { "id": 1903256, "author_name": "rasoulmojtahedzadeh", "author_url": "", "post_date": "08/17/2022 08:32:26", "content": "Why do you use maximum() operation for hessian in the code?