{
  "id": 218299,
  "title": "How to calculate custom AUC with distributed training(DDP)",
  "url": "/competitions/ranzcr-clip-catheter-line-classification/discussion/218299",
  "author_name": "cswwp",
  "post_date": "2021-02-10T04:14:54.752000",
  "votes": 7,
  "comment_count": 7,
  "views": 0,
  "content": "<p>For speed training, i prepare to train in my local with distributed training, but i am struggle to calculate custom AUC with distributed training, anyone how to implement this compete AUC with distributed training? </p>",
  "messages": [
    {
      "id": 1194152,
      "postDate": "2021-02-10T04:14:54.753Z",
      "content": "<p>For speed training, i prepare to train in my local with distributed training, but i am struggle to calculate custom AUC with distributed training, anyone how to implement this compete AUC with distributed training? </p>",
      "rawMarkdown": "For speed training, i prepare to train in my local with distributed training, but i am struggle to calculate custom AUC with distributed training, anyone how to implement this compete AUC with distributed training? ",
      "votes": 7
    },
    {
      "id": 1197166,
      "postDate": "2021-02-12T01:30:01.957Z",
      "content": "<p>As <a href=\"https://www.kaggle.com/underwearfitting\" target=\"_blank\">@underwearfitting</a> suggested, you may gather predictions and targets from all devices and then calculate AUC as usual. I share a simple example on vanilla pytorch.</p>\n<pre><code>import torch\nimport torch.distributed as dist\n\ndef gather_tensor(tensor):\n    world_size = dist.get_world_size()\n    if world_size == 1:\n        return tensor\n    tensor_list = [torch.zeros_like(tensor) for _ in range(world_size)]\n    dist.all_gather(tensor_list, tensor)\n    return torch.cat(tensor_list)\n</code></pre>\n<p>Call <code>predictions = gather_tensor(predictions)</code> and <code>targets = gather_tensor(targets)</code> right before calculating AUC.</p>",
      "rawMarkdown": "As @underwearfitting suggested, you may gather predictions and targets from all devices and then calculate AUC as usual. I share a simple example on vanilla pytorch.\n\n```python\nimport torch\nimport torch.distributed as dist\n\ndef gather_tensor(tensor):\n    world_size = dist.get_world_size()\n    if world_size == 1:\n        return tensor\n    tensor_list = [torch.zeros_like(tensor) for _ in range(world_size)]\n    dist.all_gather(tensor_list, tensor)\n    return torch.cat(tensor_list)\n```\n\nCall `predictions = gather_tensor(predictions)` and `targets = gather_tensor(targets)` right before calculating AUC.",
      "votes": 8,
      "replies": [
        {
          "id": 1201187,
          "postDate": "2021-02-15T08:08:05.633Z",
          "content": "<p>Thank you, i will try</p>",
          "rawMarkdown": "Thank you, i will try"
        },
        {
          "id": 1211522,
          "postDate": "2021-02-20T09:57:34.770Z",
          "rawMarkdown": "",
          "isDeleted": true
        }
      ]
    },
    {
      "id": 1194569,
      "postDate": "2021-02-10T08:45:03.053Z",
      "content": "<p>Hi, you may gather each batch then calculate.</p>",
      "rawMarkdown": "Hi, you may gather each batch then calculate.",
      "votes": 4,
      "replies": [
        {
          "id": 1201189,
          "postDate": "2021-02-15T08:08:56.227Z",
          "content": "<p>ok, thank you</p>",
          "rawMarkdown": "ok, thank you"
        }
      ]
    },
    {
      "id": 1194745,
      "postDate": "2021-02-10T10:42:46.550Z",
      "rawMarkdown": "",
      "votes": 2,
      "isDeleted": true,
      "replies": [
        {
          "id": 1201192,
          "postDate": "2021-02-15T08:10:31.920Z",
          "content": "<p>But this compete's custom auc is different with pytorch lightning's. \"To calculate the final score, AUC is calculated for each of the 11 labels, then averaged. The score is then the average of the individual AUCs of each predicted column.\"</p>",
          "rawMarkdown": "But this compete's custom auc is different with pytorch lightning's. \"To calculate the final score, AUC is calculated for each of the 11 labels, then averaged. The score is then the average of the individual AUCs of each predicted column.\""
        }
      ]
    }
  ],
  "comments": [
    {
      "id": 1197166,
      "author_name": "RabotniKuma",
      "author_url": "",
      "post_date": "2021-02-12T01:30:01.957000",
      "content": "<p>As <a href=\"https://www.kaggle.com/underwearfitting\" target=\"_blank\">@underwearfitting</a> suggested, you may gather predictions and targets from all devices and then calculate AUC as usual. I share a simple example on vanilla pytorch.</p>\n<pre><code>import torch\nimport torch.distributed as dist\n\ndef gather_tensor(tensor):\n    world_size = dist.get_world_size()\n    if world_size == 1:\n        return tensor\n    tensor_list = [torch.zeros_like(tensor) for _ in range(world_size)]\n    dist.all_gather(tensor_list, tensor)\n    return torch.cat(tensor_list)\n</code></pre>\n<p>Call <code>predictions = gather_tensor(predictions)</code> and <code>targets = gather_tensor(targets)</code> right before calculating AUC.</p>",
      "votes": 8,
      "replies": [
        {
          "id": 1201187,
          "author_name": "cswwp",
          "author_url": "",
          "post_date": "2021-02-15T08:08:05.633000",
          "content": "<p>Thank you, i will try</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1211522,
          "author_name": "",
          "author_url": "",
          "post_date": "2021-02-20T09:57:34.770000",
          "content": "",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1194569,
      "author_name": "sin",
      "author_url": "",
      "post_date": "2021-02-10T08:45:03.053000",
      "content": "<p>Hi, you may gather each batch then calculate.</p>",
      "votes": 4,
      "replies": [
        {
          "id": 1201189,
          "author_name": "cswwp",
          "author_url": "",
          "post_date": "2021-02-15T08:08:56.227000",
          "content": "<p>ok, thank you</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1194745,
      "author_name": "",
      "author_url": "",
      "post_date": "2021-02-10T10:42:46.550000",
      "content": "",
      "votes": 2,
      "replies": [
        {
          "id": 1201192,
          "author_name": "cswwp",
          "author_url": "",
          "post_date": "2021-02-15T08:10:31.920000",
          "content": "<p>But this compete's custom auc is different with pytorch lightning's. \"To calculate the final score, AUC is calculated for each of the 11 labels, then averaged. The score is then the average of the individual AUCs of each predicted column.\"</p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1194152": "For speed training, i prepare to train in my local with distributed training, but i am struggle to calculate custom AUC with distributed training, anyone how to implement this compete AUC with distributed training? ",
    "1197166": "As @underwearfitting suggested, you may gather predictions and targets from all devices and then calculate AUC as usual. I share a simple example on vanilla pytorch.\n\n```python\nimport torch\nimport torch.distributed as dist\n\ndef gather_tensor(tensor):\n    world_size = dist.get_world_size()\n    if world_size == 1:\n        return tensor\n    tensor_list = [torch.zeros_like(tensor) for _ in range(world_size)]\n    dist.all_gather(tensor_list, tensor)\n    return torch.cat(tensor_list)\n```\n\nCall `predictions = gather_tensor(predictions)` and `targets = gather_tensor(targets)` right before calculating AUC.",
    "1194569": "Hi, you may gather each batch then calculate.",
    "1194745": ""
  }
}