{
  "id": 39509,
  "title": "so... how to put this data into memory?",
  "url": "/competitions/cdiscount-image-classification-challenge/discussion/39509",
  "author_name": "",
  "post_date": "2017-09-15T09:50:55.458481Z",
  "votes": null,
  "comment_count": 4,
  "views": 0,
  "content": "<p>hello everyone,</p>\n\n<p>if I understand correctly data is over 50GB, does it mean it's good idea to use only part of the data during training phase, or maybe to downscale data during training? or do we need huge RAM for this competition?</p>",
  "messages": [
    {
      "id": "221443",
      "postDate": "09/15/2017 09:50:55",
      "content": "<p>hello everyone,</p>\n\n<p>if I understand correctly data is over 50GB, does it mean it's good idea to use only part of the data during training phase, or maybe to downscale data during training? or do we need huge RAM for this competition?</p>",
      "rawMarkdown": "hello everyone,\n\nif I understand correctly data is over 50GB, does it mean it's good idea to use only part of the data during training phase, or maybe to downscale data during training? or do we need huge RAM for this competition?",
      "votes": null
    },
    {
      "id": "221486",
      "postDate": "09/15/2017 12:51:03",
      "content": "<p>No need for loading the whole dataset into memory at once. Just iterate through (<a href=\"https://www.kaggle.com/blazeka/multi-gpu-tensorflow-convnet-0-55\">example</a>), generate files (<a href=\"https://www.kaggle.com/nimararora/script-to-generate-raw-images-in-subdirs\">example</a>) or check out this <a href=\"https://www.kaggle.com/vfdev5/random-item-access\">random-access hack</a>.</p>",
      "rawMarkdown": "No need for loading the whole dataset into memory at once. Just iterate through ([example](https://www.kaggle.com/blazeka/multi-gpu-tensorflow-convnet-0-55)), generate files ([example](https://www.kaggle.com/nimararora/script-to-generate-raw-images-in-subdirs)) or check out this [random-access hack](https://www.kaggle.com/vfdev5/random-item-access).",
      "votes": null
    },
    {
      "id": "221497",
      "postDate": "09/15/2017 13:23:32",
      "content": "<p>thanks for tips :)</p>",
      "rawMarkdown": "thanks for tips :)",
      "votes": null
    },
    {
      "id": "224388",
      "postDate": "09/26/2017 04:58:38",
      "content": "<p>You are probably trying a full batch learner, you can do this if you have enough RAM.  I think people will use online learner or mini-batch learner to get the training.. I hope this helps</p>",
      "rawMarkdown": "You are probably trying a full batch learner, you can do this if you have enough RAM.  I think people will use online learner or mini-batch learner to get the training.. I hope this helps",
      "votes": null
    },
    {
      "id": "224417",
      "postDate": "09/26/2017 08:59:58",
      "content": "<p>I understand the idea but I always put my whole date into X and Y then used XGBoost or Neural Network or other algorithm, and in this competition I can't have whole X in the memory, so I need to learn is it possible to make for instance CNN network and use random-access data in it - so data will be loaded/unloaded during training</p>",
      "rawMarkdown": "I understand the idea but I always put my whole date into X and Y then used XGBoost or Neural Network or other algorithm, and in this competition I can't have whole X in the memory, so I need to learn is it possible to make for instance CNN network and use random-access data in it - so data will be loaded/unloaded during training",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 221486,
      "author_name": "blazeka",
      "author_url": "",
      "post_date": "09/15/2017 12:51:03",
      "content": "<p>No need for loading the whole dataset into memory at once. Just iterate through (<a href=\"https://www.kaggle.com/blazeka/multi-gpu-tensorflow-convnet-0-55\">example</a>), generate files (<a href=\"https://www.kaggle.com/nimararora/script-to-generate-raw-images-in-subdirs\">example</a>) or check out this <a href=\"https://www.kaggle.com/vfdev5/random-item-access\">random-access hack</a>.</p>",
      "votes": null,
      "replies": [
        {
          "id": 221497,
          "author_name": "jacekpoplawski",
          "author_url": "",
          "post_date": "09/15/2017 13:23:32",
          "content": "<p>thanks for tips :)</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 224388,
      "author_name": "abualabed",
      "author_url": "",
      "post_date": "09/26/2017 04:58:38",
      "content": "<p>You are probably trying a full batch learner, you can do this if you have enough RAM.  I think people will use online learner or mini-batch learner to get the training.. I hope this helps</p>",
      "votes": null,
      "replies": [
        {
          "id": 224417,
          "author_name": "jacekpoplawski",
          "author_url": "",
          "post_date": "09/26/2017 08:59:58",
          "content": "<p>I understand the idea but I always put my whole date into X and Y then used XGBoost or Neural Network or other algorithm, and in this competition I can't have whole X in the memory, so I need to learn is it possible to make for instance CNN network and use random-access data in it - so data will be loaded/unloaded during training</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "221443": "hello everyone,\n\nif I understand correctly data is over 50GB, does it mean it's good idea to use only part of the data during training phase, or maybe to downscale data during training? or do we need huge RAM for this competition?",
    "221486": "No need for loading the whole dataset into memory at once. Just iterate through ([example](https://www.kaggle.com/blazeka/multi-gpu-tensorflow-convnet-0-55)), generate files ([example](https://www.kaggle.com/nimararora/script-to-generate-raw-images-in-subdirs)) or check out this [random-access hack](https://www.kaggle.com/vfdev5/random-item-access).",
    "221497": "thanks for tips :)",
    "224388": "You are probably trying a full batch learner, you can do this if you have enough RAM.  I think people will use online learner or mini-batch learner to get the training.. I hope this helps",
    "224417": "I understand the idea but I always put my whole date into X and Y then used XGBoost or Neural Network or other algorithm, and in this competition I can't have whole X in the memory, so I need to learn is it possible to make for instance CNN network and use random-access data in it - so data will be loaded/unloaded during training"
  },
  "source": "meta"
}