{
  "id": 295978,
  "title": "I'm sorry, I'm not sure.",
  "url": "/competitions/sartorius-cell-instance-segmentation/discussion/295978",
  "author_name": "",
  "post_date": "2021-12-19T06:21:12.205648700Z",
  "votes": null,
  "comment_count": 1,
  "views": 0,
  "content": "<p>Please tell me how to deal with the case where the data is as heavy as 200gb.<br>\n/kaggle/?<br>\nI'm sorry, I'm not sure.</p>",
  "messages": [
    {
      "id": "1622763",
      "postDate": "12/19/2021 06:21:12",
      "content": "<p>Please tell me how to deal with the case where the data is as heavy as 200gb.<br>\n/kaggle/?<br>\nI'm sorry, I'm not sure.</p>",
      "rawMarkdown": "Please tell me how to deal with the case where the data is as heavy as 200gb.\n/kaggle/?\nI'm sorry, I'm not sure.",
      "votes": null
    },
    {
      "id": "1623069",
      "postDate": "12/19/2021 13:39:59",
      "content": "<p><em>Generator functions</em> might be the key here.<br>\nWe import data in batches much smaller than the original dataset and repeat the process in steps while also shuffling the data till we train enough. This is the link to the page from where I learnt about them, you may have a look and see if it helps. Keep us updated :)<br>\nLink: <a href=\"https://www.jessicayung.com/using-generators-in-python-to-train-machine-learning-models/\" target=\"_blank\">Generator Models in Python</a></p>",
      "rawMarkdown": "*Generator functions* might be the key here.\nWe import data in batches much smaller than the original dataset and repeat the process in steps while also shuffling the data till we train enough. This is the link to the page from where I learnt about them, you may have a look and see if it helps. Keep us updated :)\nLink: [Generator Models in Python](https://www.jessicayung.com/using-generators-in-python-to-train-machine-learning-models/)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1623069,
      "author_name": "emperorarthurix",
      "author_url": "",
      "post_date": "12/19/2021 13:39:59",
      "content": "<p><em>Generator functions</em> might be the key here.<br>\nWe import data in batches much smaller than the original dataset and repeat the process in steps while also shuffling the data till we train enough. This is the link to the page from where I learnt about them, you may have a look and see if it helps. Keep us updated :)<br>\nLink: <a href=\"https://www.jessicayung.com/using-generators-in-python-to-train-machine-learning-models/\" target=\"_blank\">Generator Models in Python</a></p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1622763": "Please tell me how to deal with the case where the data is as heavy as 200gb.\n/kaggle/?\nI'm sorry, I'm not sure.",
    "1623069": "*Generator functions* might be the key here.\nWe import data in batches much smaller than the original dataset and repeat the process in steps while also shuffling the data till we train enough. This is the link to the page from where I learnt about them, you may have a look and see if it helps. Keep us updated :)\nLink: [Generator Models in Python](https://www.jessicayung.com/using-generators-in-python-to-train-machine-learning-models/)"
  },
  "source": "meta"
}