{
  "id": 507727,
  "title": "Does anyone have any advice on how to avoid memory issues?",
  "url": "/competitions/rsna-2024-lumbar-spine-degenerative-classification/discussion/507727",
  "author_name": "",
  "post_date": "2024-05-27T02:20:27.515906300Z",
  "votes": 1,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I was trying to first divide the dataset into training/validation subsets but my computer can't handle that much memory. I was thinking that maybe there's an option to work with \"batches\" but I'm not sure about this. Does anyone have some resources or have a methodology regarding this issue?</p>",
  "messages": [
    {
      "id": "2838370",
      "postDate": "05/27/2024 02:20:27",
      "content": "<p>I was trying to first divide the dataset into training/validation subsets but my computer can't handle that much memory. I was thinking that maybe there's an option to work with \"batches\" but I'm not sure about this. Does anyone have some resources or have a methodology regarding this issue?</p>",
      "rawMarkdown": "I was trying to first divide the dataset into training/validation subsets but my computer can't handle that much memory. I was thinking that maybe there's an option to work with \"batches\" but I'm not sure about this. Does anyone have some resources or have a methodology regarding this issue?",
      "votes": null
    },
    {
      "id": "2840009",
      "postDate": "05/27/2024 22:18:28",
      "content": "<p>Since the competition is computer vision, deep learning models are probably most appropriate. The „mainstream“ libraries (pytorch and tensorflow) both provide batch-loading of data. In pytorch it’s simply called Dataloader class. I’d recommend to take a look at that :)</p>",
      "rawMarkdown": "Since the competition is computer vision, deep learning models are probably most appropriate. The „mainstream“ libraries (pytorch and tensorflow) both provide batch-loading of data. In pytorch it’s simply called Dataloader class. I’d recommend to take a look at that :)",
      "votes": null
    },
    {
      "id": "2846168",
      "postDate": "05/31/2024 01:44:56",
      "content": "<p>What works for me is too only open sections of the dataset at a time (something made easy as you can load a handful of scans at a time) and train on that. Would be great to hear other solutions</p>",
      "rawMarkdown": "What works for me is too only open sections of the dataset at a time (something made easy as you can load a handful of scans at a time) and train on that. Would be great to hear other solutions",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2840009,
      "author_name": "johnbergmann",
      "author_url": "",
      "post_date": "05/27/2024 22:18:28",
      "content": "<p>Since the competition is computer vision, deep learning models are probably most appropriate. The „mainstream“ libraries (pytorch and tensorflow) both provide batch-loading of data. In pytorch it’s simply called Dataloader class. I’d recommend to take a look at that :)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2846168,
      "author_name": "qmot66",
      "author_url": "",
      "post_date": "05/31/2024 01:44:56",
      "content": "<p>What works for me is too only open sections of the dataset at a time (something made easy as you can load a handful of scans at a time) and train on that. Would be great to hear other solutions</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2838370": "I was trying to first divide the dataset into training/validation subsets but my computer can't handle that much memory. I was thinking that maybe there's an option to work with \"batches\" but I'm not sure about this. Does anyone have some resources or have a methodology regarding this issue?",
    "2840009": "Since the competition is computer vision, deep learning models are probably most appropriate. The „mainstream“ libraries (pytorch and tensorflow) both provide batch-loading of data. In pytorch it’s simply called Dataloader class. I’d recommend to take a look at that :)",
    "2846168": "What works for me is too only open sections of the dataset at a time (something made easy as you can load a handful of scans at a time) and train on that. Would be great to hear other solutions"
  },
  "source": "meta"
}