{ "id": 39951, "title": "weighing difficult pixel by focal loss", "url": "/competitions/carvana-image-masking-challenge/discussion/39951", "author_name": "", "post_date": "2017-09-24T17:34:55.369968600Z", "votes": 16, "comment_count": 6, "views": 0, "content": "

This is a method to weigh easy and difficult samples. It is useful when there is a imbalance of such training samples.

\n\n

\"enter

\n\n

\"enter

\n\n

reference:

\n\n
    \n

  1. https://arxiv.org/abs/1708.02002: \n\"Focal Loss for Dense Object Detection\" - Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, Piotr Dollár

    2. \n

  2. https://arxiv.org/pdf/1709.03741.pdf

    3. \n
", "messages": [ { "id": "223998", "postDate": "09/24/2017 17:34:55", "content": "

This is a method to weigh easy and difficult samples. It is useful when there is a imbalance of such training samples.

\n\n

\"enter

\n\n

\"enter

\n\n

reference:

\n\n
    \n

  1. https://arxiv.org/abs/1708.02002: \n\"Focal Loss for Dense Object Detection\" - Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, Piotr Dollár

    2. \n

  2. https://arxiv.org/pdf/1709.03741.pdf

    3. \n
", "rawMarkdown": "This is a method to weigh easy and difficult samples. It is useful when there is a imbalance of such training samples.\n\n ![enter image description here][1]\n\n ![enter image description here][2]\n\n\n\nreference:\n\n1. https://arxiv.org/abs/1708.02002: \n\"Focal Loss for Dense Object Detection\" - Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, Piotr Dollár\n\n2. https://arxiv.org/pdf/1709.03741.pdf\n\n\n [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/223998/7373/focal%20loss1.png\n [2]: https://kaggle2.blob.core.windows.net/forum-message-attachments/223998/7374/focal%20loss.png", "votes": null }, { "id": "224002", "postDate": "09/24/2017 18:08:35", "content": "

Thanks for a nice idea!

", "rawMarkdown": "Thanks for a nice idea!", "votes": null }, { "id": "224026", "postDate": "09/24/2017 20:00:53", "content": "

Don't even think about it! You are already the first!

", "rawMarkdown": "Don't even think about it! You are already the first!", "votes": null }, { "id": "224557", "postDate": "09/26/2017 18:53:19", "content": "
def focus_loss(y_true,y_pred):\ngamma = 2.\ny_true = K.flatten(y_true)\ny_pred = K.flatten(y_pred)\nloss = y_true*K.log(y_pred+K.epsilon())*(1-y_pred+K.epsilon())**gamma + \\\n       (1-y_true)*K.log(1-y_pred+K.epsilon())*(y_pred+K.epsilon())**gamma\nreturn -K.mean(loss)\n
\n\n

Does this look right?

", "rawMarkdown": "def focus_loss(y_true,y_pred):\n gamma = 2.\n y_true = K.flatten(y_true)\n y_pred = K.flatten(y_pred)\n loss = y_true*K.log(y_pred+K.epsilon())*(1-y_pred+K.epsilon())**gamma + \\\n (1-y_true)*K.log(1-y_pred+K.epsilon())*(y_pred+K.epsilon())**gamma\n return -K.mean(loss)\n\nDoes this look right?", "votes": null }, { "id": "224639", "postDate": "09/27/2017 05:04:24", "content": "

looks correct.

\n\n
def compute_focal_weights(logits, labels,gamma=2):\nlabels = labels.detach()\nlogits = logits.detach()\nprobs  = F.sigmoid(logits)\n\nbatch_size,H,W = labels.size()\n\nw_pos = torch.pow((1-probs), gamma)\nw_neg = torch.pow((probs), gamma)\nweights = (labels==1).float()*w_pos + (labels==0).float()*w_neg\n\n#debug\nif 0:\n    for b in range(batch_size):\n        weight = weights[b].data.cpu().numpy()*255\n        im_show('weight', weight, resize=1)\n        cv2.waitKey(0)\n\nreturn weights\n
\n\n

def criterion(logits, labels):

\n\n
weights  = compute_focal_weights(logits, labels) \nloss = WeightedBCELoss2d()(logits, labels, weights)\nreturn loss\n
", "rawMarkdown": "looks correct. \n\n def compute_focal_weights(logits, labels,gamma=2):\n labels = labels.detach()\n logits = logits.detach()\n probs = F.sigmoid(logits)\n\n batch_size,H,W = labels.size()\n\n w_pos = torch.pow((1-probs), gamma)\n w_neg = torch.pow((probs), gamma)\n weights = (labels==1).float()*w_pos + (labels==0).float()*w_neg\n\n #debug\n if 0:\n for b in range(batch_size):\n weight = weights[b].data.cpu().numpy()*255\n im_show('weight', weight, resize=1)\n cv2.waitKey(0)\n\n return weights\n\n\n \ndef criterion(logits, labels): \n \n weights = compute_focal_weights(logits, labels) \n loss = WeightedBCELoss2d()(logits, labels, weights)\n return loss", "votes": null }, { "id": "224640", "postDate": "09/27/2017 05:04:48", "content": "

this is focal loss quite magic when you use for finetunning. it speeds up and reduce loss

", "rawMarkdown": "this is focal loss quite magic when you use for finetunning. it speeds up and reduce loss", "votes": null }, { "id": "228564", "postDate": "10/07/2017 04:27:30", "content": "

there is a bug in my code. The variables should not be deteched. WeightedBCELoss2d() cannot be used if the weight terms don't propagate gradients.

\n\n

after some checking, the weighing terms (1-p)^gamma and p^gamma are back propagated as well. you can refer to:

\n\n

https://github.com/zimenglan-sysu-512/paper-note/blob/master/focal_loss.pdf\nhttps://github.com/unsky/focal-loss

", "rawMarkdown": "there is a bug in my code. The variables should not be deteched. WeightedBCELoss2d() cannot be used if the weight terms don't propagate gradients.\n\nafter some checking, the weighing terms (1-p)^gamma and p^gamma are back propagated as well. you can refer to:\n\nhttps://github.com/zimenglan-sysu-512/paper-note/blob/master/focal_loss.pdf\nhttps://github.com/unsky/focal-loss", "votes": null } ], "comments": [ { "id": 224002, "author_name": "asanakoev", "author_url": "", "post_date": "09/24/2017 18:08:35", "content": "

Thanks for a nice idea!

", "votes": null, "replies": [ { "id": 224026, "author_name": "ceperaang", "author_url": "", "post_date": "09/24/2017 20:00:53", "content": "

Don't even think about it! You are already the first!

", "votes": null, "replies": [] }, { "id": 224640, "author_name": "hengck23", "author_url": "", "post_date": "09/27/2017 05:04:48", "content": "

this is focal loss quite magic when you use for finetunning. it speeds up and reduce loss

", "votes": null, "replies": [] } ] }, { "id": 224557, "author_name": "gpradk", "author_url": "", "post_date": "09/26/2017 18:53:19", "content": "
def focus_loss(y_true,y_pred):\ngamma = 2.\ny_true = K.flatten(y_true)\ny_pred = K.flatten(y_pred)\nloss = y_true*K.log(y_pred+K.epsilon())*(1-y_pred+K.epsilon())**gamma + \\\n       (1-y_true)*K.log(1-y_pred+K.epsilon())*(y_pred+K.epsilon())**gamma\nreturn -K.mean(loss)\n
\n\n

Does this look right?

", "votes": null, "replies": [ { "id": 224639, "author_name": "hengck23", "author_url": "", "post_date": "09/27/2017 05:04:24", "content": "

looks correct.

\n\n
def compute_focal_weights(logits, labels,gamma=2):\nlabels = labels.detach()\nlogits = logits.detach()\nprobs  = F.sigmoid(logits)\n\nbatch_size,H,W = labels.size()\n\nw_pos = torch.pow((1-probs), gamma)\nw_neg = torch.pow((probs), gamma)\nweights = (labels==1).float()*w_pos + (labels==0).float()*w_neg\n\n#debug\nif 0:\n    for b in range(batch_size):\n        weight = weights[b].data.cpu().numpy()*255\n        im_show('weight', weight, resize=1)\n        cv2.waitKey(0)\n\nreturn weights\n
\n\n

def criterion(logits, labels):

\n\n
weights  = compute_focal_weights(logits, labels) \nloss = WeightedBCELoss2d()(logits, labels, weights)\nreturn loss\n
", "votes": null, "replies": [] }, { "id": 228564, "author_name": "hengck23", "author_url": "", "post_date": "10/07/2017 04:27:30", "content": "

there is a bug in my code. The variables should not be deteched. WeightedBCELoss2d() cannot be used if the weight terms don't propagate gradients.

\n\n

after some checking, the weighing terms (1-p)^gamma and p^gamma are back propagated as well. you can refer to:

\n\n

https://github.com/zimenglan-sysu-512/paper-note/blob/master/focal_loss.pdf\nhttps://github.com/unsky/focal-loss

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "223998": "This is a method to weigh easy and difficult samples. It is useful when there is a imbalance of such training samples.\n\n ![enter image description here][1]\n\n ![enter image description here][2]\n\n\n\nreference:\n\n1. https://arxiv.org/abs/1708.02002: \n\"Focal Loss for Dense Object Detection\" - Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, Piotr Dollár\n\n2. https://arxiv.org/pdf/1709.03741.pdf\n\n\n [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/223998/7373/focal%20loss1.png\n [2]: https://kaggle2.blob.core.windows.net/forum-message-attachments/223998/7374/focal%20loss.png", "224002": "Thanks for a nice idea!", "224026": "Don't even think about it! You are already the first!", "224557": "def focus_loss(y_true,y_pred):\n gamma = 2.\n y_true = K.flatten(y_true)\n y_pred = K.flatten(y_pred)\n loss = y_true*K.log(y_pred+K.epsilon())*(1-y_pred+K.epsilon())**gamma + \\\n (1-y_true)*K.log(1-y_pred+K.epsilon())*(y_pred+K.epsilon())**gamma\n return -K.mean(loss)\n\nDoes this look right?", "224639": "looks correct. \n\n def compute_focal_weights(logits, labels,gamma=2):\n labels = labels.detach()\n logits = logits.detach()\n probs = F.sigmoid(logits)\n\n batch_size,H,W = labels.size()\n\n w_pos = torch.pow((1-probs), gamma)\n w_neg = torch.pow((probs), gamma)\n weights = (labels==1).float()*w_pos + (labels==0).float()*w_neg\n\n #debug\n if 0:\n for b in range(batch_size):\n weight = weights[b].data.cpu().numpy()*255\n im_show('weight', weight, resize=1)\n cv2.waitKey(0)\n\n return weights\n\n\n \ndef criterion(logits, labels): \n \n weights = compute_focal_weights(logits, labels) \n loss = WeightedBCELoss2d()(logits, labels, weights)\n return loss", "224640": "this is focal loss quite magic when you use for finetunning. it speeds up and reduce loss", "228564": "there is a bug in my code. The variables should not be deteched. WeightedBCELoss2d() cannot be used if the weight terms don't propagate gradients.\n\nafter some checking, the weighing terms (1-p)^gamma and p^gamma are back propagated as well. you can refer to:\n\nhttps://github.com/zimenglan-sysu-512/paper-note/blob/master/focal_loss.pdf\nhttps://github.com/unsky/focal-loss" }, "source": "meta" }