{
  "id": 237053,
  "title": "What is the best way to handle RAM with when we copy the tiff image to a numpy array?",
  "url": "/competitions/hubmap-kidney-segmentation/discussion/237053",
  "author_name": "",
  "post_date": "2021-05-06T21:33:37.548403700Z",
  "votes": null,
  "comment_count": 1,
  "views": 0,
  "content": "<p>Hi! I am reading the tiff file to a numpy array with this code [1]. After I read the image and rescale it by 0.25, so I got 4x less image. I was able to process the public leaderboard with this code [2], but it stuck with the private one.  I tried this method [3] as well, but if the image shape is not (x, y, 3) then I lost dimensions (1, 30720, 47340), and also I can't use the resize function.</p>\n<p>I can't use <a href=\"https://www.kaggle.com/leighplt/pytorch-fcn-resnet50\" target=\"_blank\">this method</a> with <code>Window.from_slices</code>, because I am using a different approach.</p>\n<p>Does anyone have a suggestion on how could I handle this?</p>\n<p>Thanks for the help!</p>\n<p>[1]</p>\n<pre><code>scale = 0.25\nimage = read_tiff(image_file)\nimage_small = cv2.resize(image, dsize=None, fx=scale, fy=scale, interpolation=cv2.INTER_LINEAR)\n</code></pre>\n<p>[2]</p>\n<pre><code>def read_tiff(image_file):\n    image = tiff.imread(image_file)\n    if len(image.shape) == 5: \n        image = np.transpose(image.squeeze(), (1,2,0))\n    elif image.shape[0] == 3:\n        image = image.transpose(1, 2, 0)\n    return image\n</code></pre>\n<p>[3]</p>\n<pre><code>def read_tiff(image_file):\n    image = rasterio.open(image_file, transform = identity).read()\n    if len(image.shape) == 5: \n        image = np.transpose(image.squeeze(), (1,2,0))\n    elif image.shape[0] == 3:\n        image = image.transpose(1, 2, 0)\n    return image\n</code></pre>",
  "messages": [
    {
      "id": "1295997",
      "postDate": "05/06/2021 21:33:37",
      "content": "<p>Hi! I am reading the tiff file to a numpy array with this code [1]. After I read the image and rescale it by 0.25, so I got 4x less image. I was able to process the public leaderboard with this code [2], but it stuck with the private one.  I tried this method [3] as well, but if the image shape is not (x, y, 3) then I lost dimensions (1, 30720, 47340), and also I can't use the resize function.</p>\n<p>I can't use <a href=\"https://www.kaggle.com/leighplt/pytorch-fcn-resnet50\" target=\"_blank\">this method</a> with <code>Window.from_slices</code>, because I am using a different approach.</p>\n<p>Does anyone have a suggestion on how could I handle this?</p>\n<p>Thanks for the help!</p>\n<p>[1]</p>\n<pre><code>scale = 0.25\nimage = read_tiff(image_file)\nimage_small = cv2.resize(image, dsize=None, fx=scale, fy=scale, interpolation=cv2.INTER_LINEAR)\n</code></pre>\n<p>[2]</p>\n<pre><code>def read_tiff(image_file):\n    image = tiff.imread(image_file)\n    if len(image.shape) == 5: \n        image = np.transpose(image.squeeze(), (1,2,0))\n    elif image.shape[0] == 3:\n        image = image.transpose(1, 2, 0)\n    return image\n</code></pre>\n<p>[3]</p>\n<pre><code>def read_tiff(image_file):\n    image = rasterio.open(image_file, transform = identity).read()\n    if len(image.shape) == 5: \n        image = np.transpose(image.squeeze(), (1,2,0))\n    elif image.shape[0] == 3:\n        image = image.transpose(1, 2, 0)\n    return image\n</code></pre>",
      "rawMarkdown": "Hi! I am reading the tiff file to a numpy array with this code [1]. After I read the image and rescale it by 0.25, so I got 4x less image. I was able to process the public leaderboard with this code [2], but it stuck with the private one.  I tried this method [3] as well, but if the image shape is not (x, y, 3) then I lost dimensions (1, 30720, 47340), and also I can't use the resize function.\n\nI can't use [this method](https://www.kaggle.com/leighplt/pytorch-fcn-resnet50) with `Window.from_slices`, because I am using a different approach.\n\nDoes anyone have a suggestion on how could I handle this?\n\nThanks for the help!\n\n[1]\n```python\nscale = 0.25\nimage = read_tiff(image_file)\nimage_small = cv2.resize(image, dsize=None, fx=scale, fy=scale, interpolation=cv2.INTER_LINEAR)\n```\n\n\n[2]\n```python\ndef read_tiff(image_file):\n    image = tiff.imread(image_file)\n    if len(image.shape) == 5: \n        image = np.transpose(image.squeeze(), (1,2,0))\n    elif image.shape[0] == 3:\n        image = image.transpose(1, 2, 0)\n    return image\n   \n```\n\n[3]\n```python\ndef read_tiff(image_file):\n    image = rasterio.open(image_file, transform = identity).read()\n    if len(image.shape) == 5: \n        image = np.transpose(image.squeeze(), (1,2,0))\n    elif image.shape[0] == 3:\n        image = image.transpose(1, 2, 0)\n    return image\n```",
      "votes": null
    },
    {
      "id": "1296350",
      "postDate": "05/07/2021 07:58:44",
      "content": "<p>Have you seen this notebook? <br>\n<a href=\"https://www.kaggle.com/mistag/inference-hubmap-u-net-mobilenetv2-256x256l\" target=\"_blank\">to save memory the images are mapped to disk using numpy.memmap()</a></p>\n<p>also if the image does not have 3 channels you may need to stack a single channel to get 3. like what is done here<br>\n<code>with rasterio.open(filename, transform = identity) as dataset:</code><br>\n           <code>channels = [1,2,3] if dataset.count == 3 else [1,1,1]</code></p>",
      "rawMarkdown": "Have you seen this notebook? \n[to save memory the images are mapped to disk using numpy.memmap()](https://www.kaggle.com/mistag/inference-hubmap-u-net-mobilenetv2-256x256l)\n\nalso if the image does not have 3 channels you may need to stack a single channel to get 3. like what is done here\n`with rasterio.open(filename, transform = identity) as dataset:`\n           ` channels = [1,2,3] if dataset.count == 3 else [1,1,1]`",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1296350,
      "author_name": "something4kag",
      "author_url": "",
      "post_date": "05/07/2021 07:58:44",
      "content": "<p>Have you seen this notebook? <br>\n<a href=\"https://www.kaggle.com/mistag/inference-hubmap-u-net-mobilenetv2-256x256l\" target=\"_blank\">to save memory the images are mapped to disk using numpy.memmap()</a></p>\n<p>also if the image does not have 3 channels you may need to stack a single channel to get 3. like what is done here<br>\n<code>with rasterio.open(filename, transform = identity) as dataset:</code><br>\n           <code>channels = [1,2,3] if dataset.count == 3 else [1,1,1]</code></p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1295997": "Hi! I am reading the tiff file to a numpy array with this code [1]. After I read the image and rescale it by 0.25, so I got 4x less image. I was able to process the public leaderboard with this code [2], but it stuck with the private one.  I tried this method [3] as well, but if the image shape is not (x, y, 3) then I lost dimensions (1, 30720, 47340), and also I can't use the resize function.\n\nI can't use [this method](https://www.kaggle.com/leighplt/pytorch-fcn-resnet50) with `Window.from_slices`, because I am using a different approach.\n\nDoes anyone have a suggestion on how could I handle this?\n\nThanks for the help!\n\n[1]\n```python\nscale = 0.25\nimage = read_tiff(image_file)\nimage_small = cv2.resize(image, dsize=None, fx=scale, fy=scale, interpolation=cv2.INTER_LINEAR)\n```\n\n\n[2]\n```python\ndef read_tiff(image_file):\n    image = tiff.imread(image_file)\n    if len(image.shape) == 5: \n        image = np.transpose(image.squeeze(), (1,2,0))\n    elif image.shape[0] == 3:\n        image = image.transpose(1, 2, 0)\n    return image\n   \n```\n\n[3]\n```python\ndef read_tiff(image_file):\n    image = rasterio.open(image_file, transform = identity).read()\n    if len(image.shape) == 5: \n        image = np.transpose(image.squeeze(), (1,2,0))\n    elif image.shape[0] == 3:\n        image = image.transpose(1, 2, 0)\n    return image\n```",
    "1296350": "Have you seen this notebook? \n[to save memory the images are mapped to disk using numpy.memmap()](https://www.kaggle.com/mistag/inference-hubmap-u-net-mobilenetv2-256x256l)\n\nalso if the image does not have 3 channels you may need to stack a single channel to get 3. like what is done here\n`with rasterio.open(filename, transform = identity) as dataset:`\n           ` channels = [1,2,3] if dataset.count == 3 else [1,1,1]`"
  },
  "source": "meta"
}