{
  "id": 263587,
  "title": "Very slow dataloader/low gpu utilization - any tips?",
  "url": "/competitions/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/263587",
  "author_name": "",
  "post_date": "2021-08-09T18:32:45.286832600Z",
  "votes": 2,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Hello everyone,<br>\nI've noticed that I have very low gpu utilization, which stems from the fact that my dataloader is very slow.<br>\nWhat I'm currently doing is randomly choose 50 images from each scan type (FLAIR, T1w, T1Gd, T2), with size 200  and stacking them so my input size is (200, 200, 200). When there are not enough images, I add blanks.<br>\nThis is making it so that I can notice my GPU utilization is in small bursts, instead of being continuous.</p>\n<p>Do you have any advice on solving this issue? </p>",
  "messages": [
    {
      "id": "1462227",
      "postDate": "08/09/2021 18:32:45",
      "content": "<p>Hello everyone,<br>\nI've noticed that I have very low gpu utilization, which stems from the fact that my dataloader is very slow.<br>\nWhat I'm currently doing is randomly choose 50 images from each scan type (FLAIR, T1w, T1Gd, T2), with size 200  and stacking them so my input size is (200, 200, 200). When there are not enough images, I add blanks.<br>\nThis is making it so that I can notice my GPU utilization is in small bursts, instead of being continuous.</p>\n<p>Do you have any advice on solving this issue? </p>",
      "rawMarkdown": "Hello everyone,\nI've noticed that I have very low gpu utilization, which stems from the fact that my dataloader is very slow.\nWhat I'm currently doing is randomly choose 50 images from each scan type (FLAIR, T1w, T1Gd, T2), with size 200  and stacking them so my input size is (200, 200, 200). When there are not enough images, I add blanks.\nThis is making it so that I can notice my GPU utilization is in small bursts, instead of being continuous.\n\nDo you have any advice on solving this issue?",
      "votes": null
    },
    {
      "id": "1462332",
      "postDate": "08/09/2021 19:41:17",
      "content": "<p>Reading from disk is probably the culprit. You should try caching your data in memory, if you have enough ram available</p>",
      "rawMarkdown": "Reading from disk is probably the culprit. You should try caching your data in memory, if you have enough ram available",
      "votes": null
    },
    {
      "id": "1474367",
      "postDate": "08/16/2021 04:29:32",
      "content": "<p><code>DataLoader(dataset, batch_size=1, shuffle=False, sampler=None,\n           batch_sampler=None, num_workers=0, collate_fn=None,\n           pin_memory=False, drop_last=False, timeout=0,\n           worker_init_fn=None, *, prefetch_factor=2,\n           persistent_workers=False)</code></p>\n<p>If your CPUs are still free, try increase num_workers to parallelize the loading process. </p>",
      "rawMarkdown": "`DataLoader(dataset, batch_size=1, shuffle=False, sampler=None,\n           batch_sampler=None, num_workers=0, collate_fn=None,\n           pin_memory=False, drop_last=False, timeout=0,\n           worker_init_fn=None, *, prefetch_factor=2,\n           persistent_workers=False)`\n\nIf your CPUs are still free, try increase num_workers to parallelize the loading process.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1462332,
      "author_name": "carloalbertobarbano",
      "author_url": "",
      "post_date": "08/09/2021 19:41:17",
      "content": "<p>Reading from disk is probably the culprit. You should try caching your data in memory, if you have enough ram available</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1474367,
      "author_name": "tungvs",
      "author_url": "",
      "post_date": "08/16/2021 04:29:32",
      "content": "<p><code>DataLoader(dataset, batch_size=1, shuffle=False, sampler=None,\n           batch_sampler=None, num_workers=0, collate_fn=None,\n           pin_memory=False, drop_last=False, timeout=0,\n           worker_init_fn=None, *, prefetch_factor=2,\n           persistent_workers=False)</code></p>\n<p>If your CPUs are still free, try increase num_workers to parallelize the loading process. </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1462227": "Hello everyone,\nI've noticed that I have very low gpu utilization, which stems from the fact that my dataloader is very slow.\nWhat I'm currently doing is randomly choose 50 images from each scan type (FLAIR, T1w, T1Gd, T2), with size 200  and stacking them so my input size is (200, 200, 200). When there are not enough images, I add blanks.\nThis is making it so that I can notice my GPU utilization is in small bursts, instead of being continuous.\n\nDo you have any advice on solving this issue?",
    "1462332": "Reading from disk is probably the culprit. You should try caching your data in memory, if you have enough ram available",
    "1474367": "`DataLoader(dataset, batch_size=1, shuffle=False, sampler=None,\n           batch_sampler=None, num_workers=0, collate_fn=None,\n           pin_memory=False, drop_last=False, timeout=0,\n           worker_init_fn=None, *, prefetch_factor=2,\n           persistent_workers=False)`\n\nIf your CPUs are still free, try increase num_workers to parallelize the loading process."
  },
  "source": "meta"
}