{
  "id": 89805,
  "title": "Loading images for training is too long",
  "url": "/competitions/inaturalist-2019-fgvc6/discussion/89805",
  "author_name": "",
  "post_date": "2019-04-17T18:47:14.511149700Z",
  "votes": 2,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I am trying to train the dataset into a convolutional network using mini batch SGD, I am using opencv to load the minibatches but just loading the images of a single epoch takes forever, any tip on how to load them? I am using my local machine. Does the kaggle kernel load the images fast?</p>",
  "messages": [
    {
      "id": "518760",
      "postDate": "04/17/2019 18:47:14",
      "content": "<p>I am trying to train the dataset into a convolutional network using mini batch SGD, I am using opencv to load the minibatches but just loading the images of a single epoch takes forever, any tip on how to load them? I am using my local machine. Does the kaggle kernel load the images fast?</p>",
      "rawMarkdown": "I am trying to train the dataset into a convolutional network using mini batch SGD, I am using opencv to load the minibatches but just loading the images of a single epoch takes forever, any tip on how to load them? I am using my local machine. Does the kaggle kernel load the images fast?",
      "votes": null
    },
    {
      "id": "519421",
      "postDate": "04/18/2019 23:31:02",
      "content": "<p>In the past I have used frameworks like PyTorch that parallelize the loading of images across multiple threads. The bottleneck for me is usually not loading images, but actually the computation on the GPU. If you are still finding things slow, one option is to resize all the images on disk so they are much smaller.</p>",
      "rawMarkdown": "In the past I have used frameworks like PyTorch that parallelize the loading of images across multiple threads. The bottleneck for me is usually not loading images, but actually the computation on the GPU. If you are still finding things slow, one option is to resize all the images on disk so they are much smaller.",
      "votes": null
    },
    {
      "id": "528248",
      "postDate": "05/07/2019 10:47:45",
      "content": "<p>Mmm okay, I will try that, Thanks!. Sorry for the delay... <a href=\"/macaodha\">@macaodha</a> </p>",
      "rawMarkdown": "Mmm okay, I will try that, Thanks!. Sorry for the delay... @macaodha",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 519421,
      "author_name": "macaodha",
      "author_url": "",
      "post_date": "04/18/2019 23:31:02",
      "content": "<p>In the past I have used frameworks like PyTorch that parallelize the loading of images across multiple threads. The bottleneck for me is usually not loading images, but actually the computation on the GPU. If you are still finding things slow, one option is to resize all the images on disk so they are much smaller.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 528248,
      "author_name": "polmonroig",
      "author_url": "",
      "post_date": "05/07/2019 10:47:45",
      "content": "<p>Mmm okay, I will try that, Thanks!. Sorry for the delay... <a href=\"/macaodha\">@macaodha</a> </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "518760": "I am trying to train the dataset into a convolutional network using mini batch SGD, I am using opencv to load the minibatches but just loading the images of a single epoch takes forever, any tip on how to load them? I am using my local machine. Does the kaggle kernel load the images fast?",
    "519421": "In the past I have used frameworks like PyTorch that parallelize the loading of images across multiple threads. The bottleneck for me is usually not loading images, but actually the computation on the GPU. If you are still finding things slow, one option is to resize all the images on disk so they are much smaller.",
    "528248": "Mmm okay, I will try that, Thanks!. Sorry for the delay... @macaodha"
  },
  "source": "meta"
}