{
  "id": 31312,
  "title": "Use numpy memmap to resolve your 'out of memory' errors when using numpy arrays",
  "url": "/competitions/intel-mobileodt-cervical-cancer-screening/discussion/31312",
  "author_name": "Rodney Thomas",
  "post_date": "2017-04-08T01:25:59.424000",
  "votes": 4,
  "comment_count": 0,
  "views": 0,
  "content": "<p>My machine only has 16GB of memory and my SSD has 256GB of storage. When I store images in a numpy array, most of the time I get the 'out of memory' error and I have to resize the images to something smaller so it can fit in memory, but at the cost of 'loss of data'. Below is a snippet that I use to resolve my memory errors and reduce the chance of resizing too:</p>\n\n<p>row = len(allFiles)</p>\n\n<p>col = len(f)</p>\n\n<p>f_image='./images.npy'</p>\n\n<p>f_targets='./targets.npy'</p>\n\n<p>if os.path.isfile(f_image):</p>\n\n<pre><code>X = np.memmap(f_image, dtype='int', mode='r', shape=(row, col))\n\ny = np.load(f_targets)\n</code></pre>\n\n<p>else:</p>\n\n<pre><code>X = np.memmap(f_image, dtype='int', mode='w+', shape=(row, col))\n\ny = []\n</code></pre>",
  "messages": [
    {
      "id": 173690,
      "postDate": "2017-04-08T01:25:59.427Z",
      "content": "<p>My machine only has 16GB of memory and my SSD has 256GB of storage. When I store images in a numpy array, most of the time I get the 'out of memory' error and I have to resize the images to something smaller so it can fit in memory, but at the cost of 'loss of data'. Below is a snippet that I use to resolve my memory errors and reduce the chance of resizing too:</p>\n\n<p>row = len(allFiles)</p>\n\n<p>col = len(f)</p>\n\n<p>f_image='./images.npy'</p>\n\n<p>f_targets='./targets.npy'</p>\n\n<p>if os.path.isfile(f_image):</p>\n\n<pre><code>X = np.memmap(f_image, dtype='int', mode='r', shape=(row, col))\n\ny = np.load(f_targets)\n</code></pre>\n\n<p>else:</p>\n\n<pre><code>X = np.memmap(f_image, dtype='int', mode='w+', shape=(row, col))\n\ny = []\n</code></pre>",
      "rawMarkdown": "My machine only has 16GB of memory and my SSD has 256GB of storage. When I store images in a numpy array, most of the time I get the 'out of memory' error and I have to resize the images to something smaller so it can fit in memory, but at the cost of 'loss of data'. Below is a snippet that I use to resolve my memory errors and reduce the chance of resizing too:\n\nrow = len(allFiles)\n\ncol = len(f)\n\nf_image='./images.npy'\n\nf_targets='./targets.npy'\n\nif os.path.isfile(f_image):\n\n    X = np.memmap(f_image, dtype='int', mode='r', shape=(row, col))\n\n    y = np.load(f_targets)\n\nelse:\n\n    X = np.memmap(f_image, dtype='int', mode='w+', shape=(row, col))\n\n    y = []\n",
      "votes": 4
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "173690": "My machine only has 16GB of memory and my SSD has 256GB of storage. When I store images in a numpy array, most of the time I get the 'out of memory' error and I have to resize the images to something smaller so it can fit in memory, but at the cost of 'loss of data'. Below is a snippet that I use to resolve my memory errors and reduce the chance of resizing too:\n\nrow = len(allFiles)\n\ncol = len(f)\n\nf_image='./images.npy'\n\nf_targets='./targets.npy'\n\nif os.path.isfile(f_image):\n\n    X = np.memmap(f_image, dtype='int', mode='r', shape=(row, col))\n\n    y = np.load(f_targets)\n\nelse:\n\n    X = np.memmap(f_image, dtype='int', mode='w+', shape=(row, col))\n\n    y = []\n"
  }
}