{
  "id": 398740,
  "title": "Reading data by memory references",
  "url": "/competitions/vesuvius-challenge-ink-detection/discussion/398740",
  "author_name": "",
  "post_date": "2023-03-31T13:50:41.156905500Z",
  "votes": 3,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Hello everyone</p>\n<p>I have been researching for optimize the memory when the data is read, and I found something.</p>\n<p>There is a library called <em>tifffile</em> that can be used to optimize the file reading to seconds by memory references.</p>\n<p>Also, in this library, a class called <em>tifffile.TiffSequence</em> can be used to optimize the multiple data reading using memory references. This class can take a filelist and generate a numpy array without reading the images, all done by references. This can be done using the <em>asarray(out='memmap')</em> method to the image sequence object.</p>\n<p>Here is an example to define the file reader</p>\n<pre><code>path = '/kaggle/input/vesuvius-challenge/train/1/surface_volume/'\nall_files = np.sort(glob.glob(path + '*'))\n\ntiffsquence = tifffile.TiffSequence(\n    files=all_files[:3]\n)\nimg_pointer = tiffsquence.asarray(out='memmap')\n</code></pre>\n<p>This class is used to read the reference of the first 3 tiff files from the train image 1.</p>\n<p>More info can be found in my notebook:<br>\n<a href=\"https://www.kaggle.com/code/riferji/optimized-data-reading-on-example-submission\" target=\"_blank\">https://www.kaggle.com/code/riferji/optimized-data-reading-on-example-submission</a></p>",
  "messages": [
    {
      "id": "2204347",
      "postDate": "03/31/2023 13:50:41",
      "content": "<p>Hello everyone</p>\n<p>I have been researching for optimize the memory when the data is read, and I found something.</p>\n<p>There is a library called <em>tifffile</em> that can be used to optimize the file reading to seconds by memory references.</p>\n<p>Also, in this library, a class called <em>tifffile.TiffSequence</em> can be used to optimize the multiple data reading using memory references. This class can take a filelist and generate a numpy array without reading the images, all done by references. This can be done using the <em>asarray(out='memmap')</em> method to the image sequence object.</p>\n<p>Here is an example to define the file reader</p>\n<pre><code>path = '/kaggle/input/vesuvius-challenge/train/1/surface_volume/'\nall_files = np.sort(glob.glob(path + '*'))\n\ntiffsquence = tifffile.TiffSequence(\n    files=all_files[:3]\n)\nimg_pointer = tiffsquence.asarray(out='memmap')\n</code></pre>\n<p>This class is used to read the reference of the first 3 tiff files from the train image 1.</p>\n<p>More info can be found in my notebook:<br>\n<a href=\"https://www.kaggle.com/code/riferji/optimized-data-reading-on-example-submission\" target=\"_blank\">https://www.kaggle.com/code/riferji/optimized-data-reading-on-example-submission</a></p>",
      "rawMarkdown": "Hello everyone\n\nI have been researching for optimize the memory when the data is read, and I found something.\n\nThere is a library called *tifffile* that can be used to optimize the file reading to seconds by memory references.\n\nAlso, in this library, a class called *tifffile.TiffSequence* can be used to optimize the multiple data reading using memory references. This class can take a filelist and generate a numpy array without reading the images, all done by references. This can be done using the *asarray(out='memmap')* method to the image sequence object.\n\nHere is an example to define the file reader\n```\npath = '/kaggle/input/vesuvius-challenge/train/1/surface_volume/'\nall_files = np.sort(glob.glob(path + '*'))\n\ntiffsquence = tifffile.TiffSequence(\n    files=all_files[:3]\n)\nimg_pointer = tiffsquence.asarray(out='memmap')\n```\n\nThis class is used to read the reference of the first 3 tiff files from the train image 1.\n\nMore info can be found in my notebook:\nhttps://www.kaggle.com/code/riferji/optimized-data-reading-on-example-submission",
      "votes": null
    },
    {
      "id": "2204365",
      "postDate": "03/31/2023 14:03:37",
      "content": "<p>You should be careful when memory mapping files. It's opening a pointer to the data on disk rather than to a copy of the data in RAM. Modifications to the memory-mapped array in your code could lead to unintended changes to on-disk data.</p>\n<pre><code> tifffile\n numpy  np\n\n()\nimg = np.ones((, ), np.uint8) * \ntifffile.imwrite(, img)\n\n()\n\n()\nimg = tifffile.memmap()\nimg[...] = np.zeros((, ), np.uint8)\n\n</code></pre>",
      "rawMarkdown": "You should be careful when memory mapping files. It's opening a pointer to the data on disk rather than to a copy of the data in RAM. Modifications to the memory-mapped array in your code could lead to unintended changes to on-disk data.\n\n```python\nimport tifffile\nimport numpy as np\n\nprint('writing white image...')\nimg = np.ones((100, 100), np.uint8) * 255\ntifffile.imwrite('test.tif', img)\n\ninput('check \"test.tif\" and press any key...')\n\nprint('assigning black to array')\nimg = tifffile.memmap('test.tif')\nimg[...] = np.zeros((100, 100), np.uint8)\n# test.tif will turn black on script exit\n```",
      "votes": null
    },
    {
      "id": "2251074",
      "postDate": "05/09/2023 03:52:24",
      "content": "<p>Have you ever encountered a <code>Notebook Threw Exception</code> when you submit, even though it works fine in the notebook? After some testing I seem to be using memory references.</p>",
      "rawMarkdown": "Have you ever encountered a `Notebook Threw Exception` when you submit, even though it works fine in the notebook? After some testing I seem to be using memory references.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2204365,
      "author_name": "csparker",
      "author_url": "",
      "post_date": "03/31/2023 14:03:37",
      "content": "<p>You should be careful when memory mapping files. It's opening a pointer to the data on disk rather than to a copy of the data in RAM. Modifications to the memory-mapped array in your code could lead to unintended changes to on-disk data.</p>\n<pre><code> tifffile\n numpy  np\n\n()\nimg = np.ones((, ), np.uint8) * \ntifffile.imwrite(, img)\n\n()\n\n()\nimg = tifffile.memmap()\nimg[...] = np.zeros((, ), np.uint8)\n\n</code></pre>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2251074,
      "author_name": "jimmyisme1",
      "author_url": "",
      "post_date": "05/09/2023 03:52:24",
      "content": "<p>Have you ever encountered a <code>Notebook Threw Exception</code> when you submit, even though it works fine in the notebook? After some testing I seem to be using memory references.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2204347": "Hello everyone\n\nI have been researching for optimize the memory when the data is read, and I found something.\n\nThere is a library called *tifffile* that can be used to optimize the file reading to seconds by memory references.\n\nAlso, in this library, a class called *tifffile.TiffSequence* can be used to optimize the multiple data reading using memory references. This class can take a filelist and generate a numpy array without reading the images, all done by references. This can be done using the *asarray(out='memmap')* method to the image sequence object.\n\nHere is an example to define the file reader\n```\npath = '/kaggle/input/vesuvius-challenge/train/1/surface_volume/'\nall_files = np.sort(glob.glob(path + '*'))\n\ntiffsquence = tifffile.TiffSequence(\n    files=all_files[:3]\n)\nimg_pointer = tiffsquence.asarray(out='memmap')\n```\n\nThis class is used to read the reference of the first 3 tiff files from the train image 1.\n\nMore info can be found in my notebook:\nhttps://www.kaggle.com/code/riferji/optimized-data-reading-on-example-submission",
    "2204365": "You should be careful when memory mapping files. It's opening a pointer to the data on disk rather than to a copy of the data in RAM. Modifications to the memory-mapped array in your code could lead to unintended changes to on-disk data.\n\n```python\nimport tifffile\nimport numpy as np\n\nprint('writing white image...')\nimg = np.ones((100, 100), np.uint8) * 255\ntifffile.imwrite('test.tif', img)\n\ninput('check \"test.tif\" and press any key...')\n\nprint('assigning black to array')\nimg = tifffile.memmap('test.tif')\nimg[...] = np.zeros((100, 100), np.uint8)\n# test.tif will turn black on script exit\n```",
    "2251074": "Have you ever encountered a `Notebook Threw Exception` when you submit, even though it works fine in the notebook? After some testing I seem to be using memory references."
  },
  "source": "meta"
}