{
  "id": 172585,
  "title": "Too large file size",
  "url": "/competitions/landmark-recognition-2020/discussion/172585",
  "author_name": "",
  "post_date": "2020-08-05T16:30:30.417479900Z",
  "votes": 5,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I'll be using kaggle kernel to train the model since I don't have any physical hardware. But the data size of 98GB is huge for kaggle kernels given it's 16GB RAM.\nJust wanted to know if there are any way arounds for dealing with such large training data in the kernel itself.</p>",
  "messages": [
    {
      "id": "959512",
      "postDate": "08/05/2020 16:30:30",
      "content": "<p>I'll be using kaggle kernel to train the model since I don't have any physical hardware. But the data size of 98GB is huge for kaggle kernels given it's 16GB RAM.\nJust wanted to know if there are any way arounds for dealing with such large training data in the kernel itself.</p>",
      "rawMarkdown": "I'll be using kaggle kernel to train the model since I don't have any physical hardware. But the data size of 98GB is huge for kaggle kernels given it's 16GB RAM.\nJust wanted to know if there are any way arounds for dealing with such large training data in the kernel itself.",
      "votes": null
    },
    {
      "id": "959647",
      "postDate": "08/05/2020 19:20:46",
      "content": "<p>You don't have to load all data in the ram. You can load the data while the model is training. This can be done with\n1. Keras/Tensorflow -&gt; flowfromdirectory\n2. pyTorch -&gt; DataLoader\nYou just need to give them the path of the image and the label.</p>",
      "rawMarkdown": "You don't have to load all data in the ram. You can load the data while the model is training. This can be done with\n1. Keras/Tensorflow -&gt; flowfromdirectory\n2. pyTorch -&gt; DataLoader\nYou just need to give them the path of the image and the label.",
      "votes": null
    },
    {
      "id": "963133",
      "postDate": "08/08/2020 17:21:32",
      "content": "<p><a href=\"https://www.kaggle.com/pawan28a95\" target=\"_blank\">@pawan28a95</a> <br>\nFlow from directory is slow (but easy to use), however tensorfliw has alternative for that, take a look on their docs, but I think it is big complicated…</p>",
      "rawMarkdown": "pawan28a95 \nFlow from directory is slow (but easy to use), however tensorfliw has alternative for that, take a look on their docs, but I think it is big complicated...",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 963133,
      "author_name": "michalbrezk",
      "author_url": "",
      "post_date": "08/08/2020 17:21:32",
      "content": "<p><a href=\"https://www.kaggle.com/pawan28a95\" target=\"_blank\">@pawan28a95</a> <br>\nFlow from directory is slow (but easy to use), however tensorfliw has alternative for that, take a look on their docs, but I think it is big complicated…</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 959647,
      "author_name": "derinformatiker",
      "author_url": "",
      "post_date": "08/05/2020 19:20:46",
      "content": "<p>You don't have to load all data in the ram. You can load the data while the model is training. This can be done with\n1. Keras/Tensorflow -&gt; flowfromdirectory\n2. pyTorch -&gt; DataLoader\nYou just need to give them the path of the image and the label.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "959512": "I'll be using kaggle kernel to train the model since I don't have any physical hardware. But the data size of 98GB is huge for kaggle kernels given it's 16GB RAM.\nJust wanted to know if there are any way arounds for dealing with such large training data in the kernel itself.",
    "959647": "You don't have to load all data in the ram. You can load the data while the model is training. This can be done with\n1. Keras/Tensorflow -&gt; flowfromdirectory\n2. pyTorch -&gt; DataLoader\nYou just need to give them the path of the image and the label.",
    "963133": "pawan28a95 \nFlow from directory is slow (but easy to use), however tensorfliw has alternative for that, take a look on their docs, but I think it is big complicated..."
  },
  "source": "meta"
}