{
  "id": 236494,
  "title": "np.save() HDD space full!",
  "url": "/competitions/bms-molecular-translation/discussion/236494",
  "author_name": "",
  "post_date": "2021-05-04T15:26:25.931289900Z",
  "votes": 3,
  "comment_count": 1,
  "views": 0,
  "content": "<p><a href=\"https://drive.google.com/file/d/1r4b2ue_Rx3opa4C-caec-2Bg_yLJpywA/view?usp=sharing\" target=\"_blank\">https://drive.google.com/file/d/1r4b2ue_Rx3opa4C-caec-2Bg_yLJpywA/view?usp=sharing</a><br>\nBasically, i wanted to save my extracted features in an array. I was doing it using np.save(). but after some iteration, it showed an error, which said the disk(HDD) space is full.<br>\nso how to overcome this?<br>\nThe above link redirects you to the error image</p>",
  "messages": [
    {
      "id": "1293138",
      "postDate": "05/04/2021 15:26:25",
      "content": "<p><a href=\"https://drive.google.com/file/d/1r4b2ue_Rx3opa4C-caec-2Bg_yLJpywA/view?usp=sharing\" target=\"_blank\">https://drive.google.com/file/d/1r4b2ue_Rx3opa4C-caec-2Bg_yLJpywA/view?usp=sharing</a><br>\nBasically, i wanted to save my extracted features in an array. I was doing it using np.save(). but after some iteration, it showed an error, which said the disk(HDD) space is full.<br>\nso how to overcome this?<br>\nThe above link redirects you to the error image</p>",
      "rawMarkdown": "https://drive.google.com/file/d/1r4b2ue_Rx3opa4C-caec-2Bg_yLJpywA/view?usp=sharing\n\nBasically, i wanted to save my extracted features in an array. I was doing it using np.save(). but after some iteration, it showed an error, which said the disk(HDD) space is full.\n\nso how to overcome this?\n\nThe above link redirects you to the error image",
      "votes": null
    },
    {
      "id": "1293198",
      "postDate": "05/04/2021 16:29:17",
      "content": "<p>Suppose you have a small image model and image size, then you still have something like this:<br>\n2,424,186 images * 36 image-patch-features * 512 feature dimension * 4 bytes = 178 GigaByte<br>\nAnd numpy saves data in text form, so you need probably 2,5x this space. <br>\nAnd with a bigger image size and -model it will increase even more. So a efficient database, even one which can compress floating point numbers still wouldn't help.<br>\nI used the train images as example, but 1.6 million test images don't change the facts.</p>\n<p>TLDR: Don't save your neural network output to disk, it uses a LOT of memory</p>",
      "rawMarkdown": "Suppose you have a small image model and image size, then you still have something like this:\n2,424,186 images * 36 image-patch-features * 512 feature dimension * 4 bytes = 178 GigaByte\nAnd numpy saves data in text form, so you need probably 2,5x this space. \nAnd with a bigger image size and -model it will increase even more. So a efficient database, even one which can compress floating point numbers still wouldn't help.\nI used the train images as example, but 1.6 million test images don't change the facts.\n\nTLDR: Don't save your neural network output to disk, it uses a LOT of memory",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1293198,
      "author_name": "cepheidq",
      "author_url": "",
      "post_date": "05/04/2021 16:29:17",
      "content": "<p>Suppose you have a small image model and image size, then you still have something like this:<br>\n2,424,186 images * 36 image-patch-features * 512 feature dimension * 4 bytes = 178 GigaByte<br>\nAnd numpy saves data in text form, so you need probably 2,5x this space. <br>\nAnd with a bigger image size and -model it will increase even more. So a efficient database, even one which can compress floating point numbers still wouldn't help.<br>\nI used the train images as example, but 1.6 million test images don't change the facts.</p>\n<p>TLDR: Don't save your neural network output to disk, it uses a LOT of memory</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1293138": "https://drive.google.com/file/d/1r4b2ue_Rx3opa4C-caec-2Bg_yLJpywA/view?usp=sharing\n\nBasically, i wanted to save my extracted features in an array. I was doing it using np.save(). but after some iteration, it showed an error, which said the disk(HDD) space is full.\n\nso how to overcome this?\n\nThe above link redirects you to the error image",
    "1293198": "Suppose you have a small image model and image size, then you still have something like this:\n2,424,186 images * 36 image-patch-features * 512 feature dimension * 4 bytes = 178 GigaByte\nAnd numpy saves data in text form, so you need probably 2,5x this space. \nAnd with a bigger image size and -model it will increase even more. So a efficient database, even one which can compress floating point numbers still wouldn't help.\nI used the train images as example, but 1.6 million test images don't change the facts.\n\nTLDR: Don't save your neural network output to disk, it uses a LOT of memory"
  },
  "source": "meta"
}