{
  "id": 106573,
  "title": "How to extract and use this large dataset ",
  "url": "/competitions/open-images-2019-object-detection/discussion/106573",
  "author_name": "",
  "post_date": "2019-08-30T02:19:50.220433400Z",
  "votes": 1,
  "comment_count": 2,
  "views": 0,
  "content": "<p>how can we train using such large dataset? I have 600 GB storage, but not in one partition. How to deal with model training in that ?</p>",
  "messages": [
    {
      "id": "612787",
      "postDate": "08/30/2019 02:19:50",
      "content": "<p>how can we train using such large dataset? I have 600 GB storage, but not in one partition. How to deal with model training in that ?</p>",
      "rawMarkdown": "how can we train using such large dataset? I have 600 GB storage, but not in one partition. How to deal with model training in that ?",
      "votes": null
    },
    {
      "id": "612928",
      "postDate": "08/30/2019 05:13:19",
      "content": "<p>you can use kaggles kernels, during training you can use datagenerator to load the images in batches. you can find a templeta of a data generator here: <a href=\"https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly\">https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly</a></p>",
      "rawMarkdown": "you can use kaggles kernels, during training you can use datagenerator to load the images in batches. you can find a templeta of a data generator here: https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly",
      "votes": null
    },
    {
      "id": "613510",
      "postDate": "08/30/2019 15:13:15",
      "content": "<p>Try to train on resized images first:  <a href=\"https://www.kaggle.com/c/open-images-2019-object-detection/discussion/96795#latest-566555\">https://www.kaggle.com/c/open-images-2019-object-detection/discussion/96795#latest-566555</a></p>",
      "rawMarkdown": "Try to train on resized images first:  https://www.kaggle.com/c/open-images-2019-object-detection/discussion/96795#latest-566555",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 612928,
      "author_name": "imperadorhades",
      "author_url": "",
      "post_date": "08/30/2019 05:13:19",
      "content": "<p>you can use kaggles kernels, during training you can use datagenerator to load the images in batches. you can find a templeta of a data generator here: <a href=\"https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly\">https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 613510,
      "author_name": "zorock",
      "author_url": "",
      "post_date": "08/30/2019 15:13:15",
      "content": "<p>Try to train on resized images first:  <a href=\"https://www.kaggle.com/c/open-images-2019-object-detection/discussion/96795#latest-566555\">https://www.kaggle.com/c/open-images-2019-object-detection/discussion/96795#latest-566555</a></p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "612787": "how can we train using such large dataset? I have 600 GB storage, but not in one partition. How to deal with model training in that ?",
    "612928": "you can use kaggles kernels, during training you can use datagenerator to load the images in batches. you can find a templeta of a data generator here: https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly",
    "613510": "Try to train on resized images first:  https://www.kaggle.com/c/open-images-2019-object-detection/discussion/96795#latest-566555"
  },
  "source": "meta"
}