{
  "id": 198390,
  "title": "Run-Length Encoding for Full-Size Images",
  "url": "/competitions/hubmap-kidney-segmentation/discussion/198390",
  "author_name": "Marco Polo",
  "post_date": "2020-11-21T01:23:03.182000",
  "votes": 1,
  "comment_count": 0,
  "views": 0,
  "content": "<p>The memory bottleneck is numpy array operations - reshape/flatten and broadcasting, at a time when we have already filled almost all the memory with an array.</p>\n<p>I have 2 ideas how to get around this problem:</p>\n<ol>\n<li><p>First reconstruct a full-size image from image tiles, then perform coordinate-wise RLE encoding.<br>\nMinuses - too long (30+ mins for training data).<br>\nPluses - it works).</p></li>\n<li><p>Reconstruct the full-size image directly to a flat array, then convert the 2d coordinates to 1d and then perform standard RLE encoding - with reshaping and broadcasting.<br>\nMinuses - (not implemented).<br>\nPluses - works fast.</p>\n<p>Please refer to the following notebook -  <a href=\"https://www.kaggle.com/polomarco/run-length-encoding-for-full-size-images\" target=\"_blank\">https://www.kaggle.com/polomarco/run-length-encoding-for-full-size-images</a></p></li>\n</ol>",
  "messages": [
    {
      "id": 1085563,
      "postDate": "2020-11-21T01:23:03.183Z",
      "content": "<p>The memory bottleneck is numpy array operations - reshape/flatten and broadcasting, at a time when we have already filled almost all the memory with an array.</p>\n<p>I have 2 ideas how to get around this problem:</p>\n<ol>\n<li><p>First reconstruct a full-size image from image tiles, then perform coordinate-wise RLE encoding.<br>\nMinuses - too long (30+ mins for training data).<br>\nPluses - it works).</p></li>\n<li><p>Reconstruct the full-size image directly to a flat array, then convert the 2d coordinates to 1d and then perform standard RLE encoding - with reshaping and broadcasting.<br>\nMinuses - (not implemented).<br>\nPluses - works fast.</p>\n<p>Please refer to the following notebook -  <a href=\"https://www.kaggle.com/polomarco/run-length-encoding-for-full-size-images\" target=\"_blank\">https://www.kaggle.com/polomarco/run-length-encoding-for-full-size-images</a></p></li>\n</ol>",
      "rawMarkdown": "The memory bottleneck is numpy array operations - reshape/flatten and broadcasting, at a time when we have already filled almost all the memory with an array.\n\nI have 2 ideas how to get around this problem:\n\n1. First reconstruct a full-size image from image tiles, then perform coordinate-wise RLE encoding.\nMinuses - too long (30+ mins for training data).\nPluses - it works).\n\n2. Reconstruct the full-size image directly to a flat array, then convert the 2d coordinates to 1d and then perform standard RLE encoding - with reshaping and broadcasting.\nMinuses - (not implemented).\nPluses - works fast.\n\n Please refer to the following notebook -  https://www.kaggle.com/polomarco/run-length-encoding-for-full-size-images",
      "votes": 1
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "1085563": "The memory bottleneck is numpy array operations - reshape/flatten and broadcasting, at a time when we have already filled almost all the memory with an array.\n\nI have 2 ideas how to get around this problem:\n\n1. First reconstruct a full-size image from image tiles, then perform coordinate-wise RLE encoding.\nMinuses - too long (30+ mins for training data).\nPluses - it works).\n\n2. Reconstruct the full-size image directly to a flat array, then convert the 2d coordinates to 1d and then perform standard RLE encoding - with reshaping and broadcasting.\nMinuses - (not implemented).\nPluses - works fast.\n\n Please refer to the following notebook -  https://www.kaggle.com/polomarco/run-length-encoding-for-full-size-images"
  }
}