{
  "id": 341854,
  "title": "Competition metric in PyTorch",
  "url": "/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/341854",
  "author_name": "",
  "post_date": "2022-08-04T13:58:40.368446500Z",
  "votes": 25,
  "comment_count": 4,
  "views": 0,
  "content": "<p>edit added <code>return loss / weights.sum()</code> <a href=\"https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/340392#1885329\" target=\"_blank\">https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/340392#1885329</a><br>\nedit (29/08/2022) Regarding loss normalization. It should in fact be done on instance leven, not batch level. Addition and division are not commutative. Sorry for not rethinking it while updating with normalization. Now I get really correlated LB score.</p>\n<p>Code to compute comptetition metric in PyTorch according to this host's post with weights<br>\n<a href=\"https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/340392\" target=\"_blank\">https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/340392</a></p>\n<pre><code>loss_fn = nn.BCEWithLogitsLoss(reduction='none')\n\ncompetition_weights = {\n    '-' : torch.tensor([7, 1, 1, 1, 1, 1, 1, 1], dtype=torch.float, device=device),\n    '+' : torch.tensor([14, 2, 2, 2, 2, 2, 2, 2], dtype=torch.float, device=device),\n}\n\n# def competiton_loss(y_hat, y):\n#     loss = loss_fn(y_hat, y)\n#     weights = y * competition_weights['+'] + (1 - y) * competition_weights['-']\n#     loss = (loss * weights).sum(axis=1).mean()\n\n#     return loss / weights.sum()\n\n\ndef competiton_loss(y_hat, y):\n    loss = loss_fn(y_hat, y)\n    weights = y * competition_weights['+'] + (1 - y) * competition_weights['-']\n    loss = (loss * weights).sum(axis=1)\n    loss = loss / weights.sum(axis=1)\n    loss = loss.mean()\n\n    return loss \n</code></pre>",
  "messages": [
    {
      "id": "1884562",
      "postDate": "08/04/2022 13:58:40",
      "content": "<p>edit added <code>return loss / weights.sum()</code> <a href=\"https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/340392#1885329\" target=\"_blank\">https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/340392#1885329</a><br>\nedit (29/08/2022) Regarding loss normalization. It should in fact be done on instance leven, not batch level. Addition and division are not commutative. Sorry for not rethinking it while updating with normalization. Now I get really correlated LB score.</p>\n<p>Code to compute comptetition metric in PyTorch according to this host's post with weights<br>\n<a href=\"https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/340392\" target=\"_blank\">https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/340392</a></p>\n<pre><code>loss_fn = nn.BCEWithLogitsLoss(reduction='none')\n\ncompetition_weights = {\n    '-' : torch.tensor([7, 1, 1, 1, 1, 1, 1, 1], dtype=torch.float, device=device),\n    '+' : torch.tensor([14, 2, 2, 2, 2, 2, 2, 2], dtype=torch.float, device=device),\n}\n\n# def competiton_loss(y_hat, y):\n#     loss = loss_fn(y_hat, y)\n#     weights = y * competition_weights['+'] + (1 - y) * competition_weights['-']\n#     loss = (loss * weights).sum(axis=1).mean()\n\n#     return loss / weights.sum()\n\n\ndef competiton_loss(y_hat, y):\n    loss = loss_fn(y_hat, y)\n    weights = y * competition_weights['+'] + (1 - y) * competition_weights['-']\n    loss = (loss * weights).sum(axis=1)\n    loss = loss / weights.sum(axis=1)\n    loss = loss.mean()\n\n    return loss \n</code></pre>",
      "rawMarkdown": "edit added `return loss / weights.sum()` https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/340392#1885329\nedit (29/08/2022) Regarding loss normalization. It should in fact be done on instance leven, not batch level. Addition and division are not commutative. Sorry for not rethinking it while updating with normalization. Now I get really correlated LB score.\n\n\nCode to compute comptetition metric in PyTorch according to this host's post with weights\nhttps://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/340392\n\n```\nloss_fn = nn.BCEWithLogitsLoss(reduction='none')\n\ncompetition_weights = {\n    '-' : torch.tensor([7, 1, 1, 1, 1, 1, 1, 1], dtype=torch.float, device=device),\n    '+' : torch.tensor([14, 2, 2, 2, 2, 2, 2, 2], dtype=torch.float, device=device),\n}\n\n# def competiton_loss(y_hat, y):\n#     loss = loss_fn(y_hat, y)\n#     weights = y * competition_weights['+'] + (1 - y) * competition_weights['-']\n#     loss = (loss * weights).sum(axis=1).mean()\n    \n#     return loss / weights.sum()\n\n\ndef competiton_loss(y_hat, y):\n    loss = loss_fn(y_hat, y)\n    weights = y * competition_weights['+'] + (1 - y) * competition_weights['-']\n    loss = (loss * weights).sum(axis=1)\n    loss = loss / weights.sum(axis=1)\n    loss = loss.mean()\n    \n    return loss \n```",
      "votes": null
    },
    {
      "id": "1885303",
      "postDate": "08/05/2022 04:47:15",
      "content": "<p>You can normalize this loss by using <code>return loss / weights.sum()</code></p>",
      "rawMarkdown": "You can normalize this loss by using `return loss / weights.sum()`",
      "votes": null
    },
    {
      "id": "1890222",
      "postDate": "08/08/2022 15:58:59",
      "content": "<p>correct me if I am wrong but according to this: </p>\n<pre><code>Vertebrae negative: 1\nVertebrae positive: 2\nPatient negative: 7\nPatient positive: 14\n</code></pre>\n<p>if we have labels <code>0, 0, 0, 0, 0, 1, 0, 1</code> (<code>C1-C7</code>, <code>overall patient</code>) i thought the weights will be <code>1, 1, 1, 1, 1, 2, 1, 14</code>? </p>",
      "rawMarkdown": "correct me if I am wrong but according to this: \n```\nVertebrae negative: 1\nVertebrae positive: 2\nPatient negative: 7\nPatient positive: 14\n```\nif we have labels `0, 0, 0, 0, 0, 1, 0, 1 ` (`C1-C7`, `overall patient`) i thought the weights will be `1, 1, 1, 1, 1, 2, 1, 14`?",
      "votes": null
    },
    {
      "id": "1890247",
      "postDate": "08/08/2022 16:16:20",
      "content": "<p>That's correct, though I believe the overall prediction comes after the vertebrae-level predictions in the actual submission file.</p>",
      "rawMarkdown": "That's correct, though I believe the overall prediction comes after the vertebrae-level predictions in the actual submission file.",
      "votes": null
    },
    {
      "id": "1890251",
      "postDate": "08/08/2022 16:18:52",
      "content": "<p>Thank you for the replay. I edited my comment<br>\n!</p>",
      "rawMarkdown": "Thank you for the replay. I edited my comment\n!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1885303,
      "author_name": "harshitsheoran",
      "author_url": "",
      "post_date": "08/05/2022 04:47:15",
      "content": "<p>You can normalize this loss by using <code>return loss / weights.sum()</code></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1890222,
      "author_name": "drhabib",
      "author_url": "",
      "post_date": "08/08/2022 15:58:59",
      "content": "<p>correct me if I am wrong but according to this: </p>\n<pre><code>Vertebrae negative: 1\nVertebrae positive: 2\nPatient negative: 7\nPatient positive: 14\n</code></pre>\n<p>if we have labels <code>0, 0, 0, 0, 0, 1, 0, 1</code> (<code>C1-C7</code>, <code>overall patient</code>) i thought the weights will be <code>1, 1, 1, 1, 1, 2, 1, 14</code>? </p>",
      "votes": null,
      "replies": [
        {
          "id": 1890247,
          "author_name": "sohier",
          "author_url": "",
          "post_date": "08/08/2022 16:16:20",
          "content": "<p>That's correct, though I believe the overall prediction comes after the vertebrae-level predictions in the actual submission file.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1890251,
          "author_name": "drhabib",
          "author_url": "",
          "post_date": "08/08/2022 16:18:52",
          "content": "<p>Thank you for the replay. I edited my comment<br>\n!</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1884562": "edit added `return loss / weights.sum()` https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/340392#1885329\nedit (29/08/2022) Regarding loss normalization. It should in fact be done on instance leven, not batch level. Addition and division are not commutative. Sorry for not rethinking it while updating with normalization. Now I get really correlated LB score.\n\n\nCode to compute comptetition metric in PyTorch according to this host's post with weights\nhttps://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/340392\n\n```\nloss_fn = nn.BCEWithLogitsLoss(reduction='none')\n\ncompetition_weights = {\n    '-' : torch.tensor([7, 1, 1, 1, 1, 1, 1, 1], dtype=torch.float, device=device),\n    '+' : torch.tensor([14, 2, 2, 2, 2, 2, 2, 2], dtype=torch.float, device=device),\n}\n\n# def competiton_loss(y_hat, y):\n#     loss = loss_fn(y_hat, y)\n#     weights = y * competition_weights['+'] + (1 - y) * competition_weights['-']\n#     loss = (loss * weights).sum(axis=1).mean()\n    \n#     return loss / weights.sum()\n\n\ndef competiton_loss(y_hat, y):\n    loss = loss_fn(y_hat, y)\n    weights = y * competition_weights['+'] + (1 - y) * competition_weights['-']\n    loss = (loss * weights).sum(axis=1)\n    loss = loss / weights.sum(axis=1)\n    loss = loss.mean()\n    \n    return loss \n```",
    "1885303": "You can normalize this loss by using `return loss / weights.sum()`",
    "1890222": "correct me if I am wrong but according to this: \n```\nVertebrae negative: 1\nVertebrae positive: 2\nPatient negative: 7\nPatient positive: 14\n```\nif we have labels `0, 0, 0, 0, 0, 1, 0, 1 ` (`C1-C7`, `overall patient`) i thought the weights will be `1, 1, 1, 1, 1, 2, 1, 14`?",
    "1890247": "That's correct, though I believe the overall prediction comes after the vertebrae-level predictions in the actual submission file.",
    "1890251": "Thank you for the replay. I edited my comment\n!"
  },
  "source": "meta"
}