In this great EDA notebook created by @ihelon, I found Inversion's classical RLE encoding function. The function has definitely stood the test of time and works great here, but I had a bit of difficulty understanding it, so I refactored it to be a bit easier for beginners. Here's the code (with some personal notes):
\n# modified from: https://www.kaggle.com/inversion/run-length-decoding-quick-start\ndef rle_decode_refactored(mask_rle, shape, color=1):\n '''\n mask_rle: run-length as string formated (start length)\n shape: (height,width) of array to return \n Returns numpy array, 1 - mask, 0 - background\n '''\n # Split the string by space, then convert it into a integer array\n s = np.array(mask_rle.split(), dtype=int)\n\n # Every even value is the start, every odd value is the \"run\" length\n starts = s[0::2] - 1\n lengths = s[1::2]\n ends = starts + lengths\n\n # The image image is actually flattened since RLE is a 1D \"run\"\n h, w, d = shape\n img = np.zeros((h * w, d), dtype=np.float32)\n\n # The color here is actually just any integer you want!\n for lo, hi in zip(starts, ends):\n img[lo : hi] = color\n # Don't forget to change the image back to the original shape\n return img.reshape(shape)\nIn this great EDA notebook created by @ihelon, I found Inversion's classical RLE encoding function. The function has definitely stood the test of time and works great here, but I had a bit of difficulty understanding it, so I refactored it to be a bit easier for beginners. Here's the code (with some personal notes):
\n# modified from: https://www.kaggle.com/inversion/run-length-decoding-quick-start\ndef rle_decode_refactored(mask_rle, shape, color=1):\n '''\n mask_rle: run-length as string formated (start length)\n shape: (height,width) of array to return \n Returns numpy array, 1 - mask, 0 - background\n '''\n # Split the string by space, then convert it into a integer array\n s = np.array(mask_rle.split(), dtype=int)\n\n # Every even value is the start, every odd value is the \"run\" length\n starts = s[0::2] - 1\n lengths = s[1::2]\n ends = starts + lengths\n\n # The image image is actually flattened since RLE is a 1D \"run\"\n h, w, d = shape\n img = np.zeros((h * w, d), dtype=np.float32)\n\n # The color here is actually just any integer you want!\n for lo, hi in zip(starts, ends):\n img[lo : hi] = color\n # Don't forget to change the image back to the original shape\n return img.reshape(shape)\nIncluding tf version here as well for future generations:
\ndef rle_decode_tf(mask_rle, shape):\n shape = tf.convert_to_tensor(shape, tf.int64)\n size = tf.math.reduce_prod(shape)\n # Split string\n s = tf.strings.split(mask_rle)\n s = tf.strings.to_number(s, tf.int64)\n # Get starts and lengths\n starts = s[::2] - 1\n lens = s[1::2]\n # Make ones to be scattered\n total_ones = tf.reduce_sum(lens)\n ones = tf.ones([total_ones], tf.uint8)\n # Make scattering indices\n r = tf.range(total_ones)\n lens_cum = tf.math.cumsum(lens)\n s = tf.searchsorted(lens_cum, r, 'right')\n idx = r + tf.gather(starts - tf.pad(lens_cum[:-1], [(1, 0)]), s)\n # Scatter ones into flattened mask\n mask_flat = tf.scatter_nd(tf.expand_dims(idx, 1), ones, [size])\n # Reshape into mask\n return tf.reshape(mask_flat, shape)\nThis came from here:
\nhttps://stackoverflow.com/questions/58693261/create-a-rle-run-lenth-encoding-mask-with-tensorflow-datasets
found this about a year ago for something else I needed..
", "rawMarkdown": "Including tf version here as well for future generations: \n\n```\n\ndef rle_decode_tf(mask_rle, shape):\n shape = tf.convert_to_tensor(shape, tf.int64)\n size = tf.math.reduce_prod(shape)\n # Split string\n s = tf.strings.split(mask_rle)\n s = tf.strings.to_number(s, tf.int64)\n # Get starts and lengths\n starts = s[::2] - 1\n lens = s[1::2]\n # Make ones to be scattered\n total_ones = tf.reduce_sum(lens)\n ones = tf.ones([total_ones], tf.uint8)\n # Make scattering indices\n r = tf.range(total_ones)\n lens_cum = tf.math.cumsum(lens)\n s = tf.searchsorted(lens_cum, r, 'right')\n idx = r + tf.gather(starts - tf.pad(lens_cum[:-1], [(1, 0)]), s)\n # Scatter ones into flattened mask\n mask_flat = tf.scatter_nd(tf.expand_dims(idx, 1), ones, [size])\n # Reshape into mask\n return tf.reshape(mask_flat, shape)\n\n\n```\n\nThis came from here:\nhttps://stackoverflow.com/questions/58693261/create-a-rle-run-lenth-encoding-mask-with-tensorflow-datasets\n\nfound this about a year ago for something else I needed..", "votes": null }, { "id": "1545792", "postDate": "10/15/2021 15:13:33", "content": "great find! I think this makes it much easier to directly train on the string targets (rather than the expanded targets). also curious if this would receive a boost on gpus.
", "rawMarkdown": "great find! I think this makes it much easier to directly train on the string targets (rather than the expanded targets). also curious if this would receive a boost on gpus.", "votes": null }, { "id": "1546716", "postDate": "10/16/2021 13:02:51", "content": "This might be helpful
\nhttps://www.kaggle.com/c/sartorius-cell-instance-segmentation/discussion/url
Very nice. Here's another version that gets masks for all the images at once. Takes about 6 seconds.
\ndef make_masks(annos, shape=(520, 704, 1), color=1):\n mask = np.zeros((shape[0]*shape[1], shape[2]))\n for rle_str in annos:\n runs = np.fromstring(rle_str, sep=\" \", dtype=np.int32).reshape(-1, 2)\n runs[:,0] -= 1\n runs[:,1] = runs.sum(axis=1)\n for r in runs:\n mask[r[0]:r[1]] = color\n return mask.reshape(shape)\n\nmasks = df_train.groupby('id')['annotation'].apply(make_masks)\nThis numba version is more complex and runs in under a second.
\n@njit\ndef color_mask(runs, mask, color=1):\n runs[::2]-=1\n runs[1::2]+=runs[::2]\n for i in np.arange(len(runs)//2):\n m = i*2\n mask[runs[m]:runs[m+1]] = color\n\ndef make_masks_split(annos, shape=(520, 704, 1)):\n mask = np.zeros((np.prod(shape[:2]), 1), dtype=np.int32)\n for rle_str in annos:\n runs = np.fromstring(rle_str, sep=\" \", \n dtype=np.int32)\n color_mask(runs, mask)\n return mask.reshape(shape)\n\ntrain_array = df_train[['id', 'annotation']].to_numpy()\nuniques = np.unique(train_array[:,0], return_inverse=True)\nmasks = []\nfor i in np.arange(len(uniques[0])):\n mask = make_masks_split(train_array[uniques[1]==i, 1])\n masks.append(mask)\nI see you are decrementing both the starts and the lengths by one here:
\nruns = np.fromstring(rle_str, sep=\" \", dtype=np.int32).reshape(-1, 2)-1\nin the original code by inversion, only the starts is decremented i think.
", "rawMarkdown": "I see you are decrementing both the starts and the lengths by one here:\n```\nruns = np.fromstring(rle_str, sep=\" \", dtype=np.int32).reshape(-1, 2)-1\n```\n\nin the original code by inversion, only the starts is decremented i think.", "votes": null }, { "id": "1549047", "postDate": "10/18/2021 18:43:51", "content": "Absolutely right! Thanks for catching it - I'll change the code.
", "rawMarkdown": "Absolutely right! Thanks for catching it - I'll change the code.", "votes": null }, { "id": "1588322", "postDate": "11/19/2021 11:07:34", "content": "Dose it help improve performance?
", "rawMarkdown": "Dose it help improve performance?", "votes": null } ], "comments": [ { "id": 1545663, "author_name": "", "author_url": "", "post_date": "10/15/2021 12:49:13", "content": "Including tf version here as well for future generations:
\ndef rle_decode_tf(mask_rle, shape):\n shape = tf.convert_to_tensor(shape, tf.int64)\n size = tf.math.reduce_prod(shape)\n # Split string\n s = tf.strings.split(mask_rle)\n s = tf.strings.to_number(s, tf.int64)\n # Get starts and lengths\n starts = s[::2] - 1\n lens = s[1::2]\n # Make ones to be scattered\n total_ones = tf.reduce_sum(lens)\n ones = tf.ones([total_ones], tf.uint8)\n # Make scattering indices\n r = tf.range(total_ones)\n lens_cum = tf.math.cumsum(lens)\n s = tf.searchsorted(lens_cum, r, 'right')\n idx = r + tf.gather(starts - tf.pad(lens_cum[:-1], [(1, 0)]), s)\n # Scatter ones into flattened mask\n mask_flat = tf.scatter_nd(tf.expand_dims(idx, 1), ones, [size])\n # Reshape into mask\n return tf.reshape(mask_flat, shape)\nThis came from here:
\nhttps://stackoverflow.com/questions/58693261/create-a-rle-run-lenth-encoding-mask-with-tensorflow-datasets
found this about a year ago for something else I needed..
", "votes": null, "replies": [ { "id": 1545792, "author_name": "xhlulu", "author_url": "", "post_date": "10/15/2021 15:13:33", "content": "great find! I think this makes it much easier to directly train on the string targets (rather than the expanded targets). also curious if this would receive a boost on gpus.
", "votes": null, "replies": [] } ] }, { "id": 1546716, "author_name": "bhargav6031", "author_url": "", "post_date": "10/16/2021 13:02:51", "content": "This might be helpful
\nhttps://www.kaggle.com/c/sartorius-cell-instance-segmentation/discussion/url
Very nice. Here's another version that gets masks for all the images at once. Takes about 6 seconds.
\ndef make_masks(annos, shape=(520, 704, 1), color=1):\n mask = np.zeros((shape[0]*shape[1], shape[2]))\n for rle_str in annos:\n runs = np.fromstring(rle_str, sep=\" \", dtype=np.int32).reshape(-1, 2)\n runs[:,0] -= 1\n runs[:,1] = runs.sum(axis=1)\n for r in runs:\n mask[r[0]:r[1]] = color\n return mask.reshape(shape)\n\nmasks = df_train.groupby('id')['annotation'].apply(make_masks)\nThis numba version is more complex and runs in under a second.
\n@njit\ndef color_mask(runs, mask, color=1):\n runs[::2]-=1\n runs[1::2]+=runs[::2]\n for i in np.arange(len(runs)//2):\n m = i*2\n mask[runs[m]:runs[m+1]] = color\n\ndef make_masks_split(annos, shape=(520, 704, 1)):\n mask = np.zeros((np.prod(shape[:2]), 1), dtype=np.int32)\n for rle_str in annos:\n runs = np.fromstring(rle_str, sep=\" \", \n dtype=np.int32)\n color_mask(runs, mask)\n return mask.reshape(shape)\n\ntrain_array = df_train[['id', 'annotation']].to_numpy()\nuniques = np.unique(train_array[:,0], return_inverse=True)\nmasks = []\nfor i in np.arange(len(uniques[0])):\n mask = make_masks_split(train_array[uniques[1]==i, 1])\n masks.append(mask)\nI see you are decrementing both the starts and the lengths by one here:
\nruns = np.fromstring(rle_str, sep=\" \", dtype=np.int32).reshape(-1, 2)-1\nin the original code by inversion, only the starts is decremented i think.
", "votes": null, "replies": [] }, { "id": 1549047, "author_name": "jpmiller", "author_url": "", "post_date": "10/18/2021 18:43:51", "content": "Absolutely right! Thanks for catching it - I'll change the code.
", "votes": null, "replies": [] } ] }, { "id": 1588322, "author_name": "liuzwin98", "author_url": "", "post_date": "11/19/2021 11:07:34", "content": "Dose it help improve performance?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1545202": "In this great [EDA notebook](https://www.kaggle.com/ihelon/cell-segmentation-run-length-decoding) created by @ihelon, I found Inversion's classical [RLE encoding function](https://www.kaggle.com/ihelon/cell-segmentation-run-length-decoding). The function has definitely stood the test of time and works great here, but I had a bit of difficulty understanding it, so I refactored it to be a bit easier for beginners. Here's the code (with some personal notes):\n\n```\n# modified from: https://www.kaggle.com/inversion/run-length-decoding-quick-start\ndef rle_decode_refactored(mask_rle, shape, color=1):\n '''\n mask_rle: run-length as string formated (start length)\n shape: (height,width) of array to return \n Returns numpy array, 1 - mask, 0 - background\n '''\n # Split the string by space, then convert it into a integer array\n s = np.array(mask_rle.split(), dtype=int)\n \n # Every even value is the start, every odd value is the \"run\" length\n starts = s[0::2] - 1\n lengths = s[1::2]\n ends = starts + lengths\n \n # The image image is actually flattened since RLE is a 1D \"run\"\n h, w, d = shape\n img = np.zeros((h * w, d), dtype=np.float32)\n \n # The color here is actually just any integer you want!\n for lo, hi in zip(starts, ends):\n img[lo : hi] = color\n # Don't forget to change the image back to the original shape\n return img.reshape(shape)\n```", "1545663": "Including tf version here as well for future generations: \n\n```\n\ndef rle_decode_tf(mask_rle, shape):\n shape = tf.convert_to_tensor(shape, tf.int64)\n size = tf.math.reduce_prod(shape)\n # Split string\n s = tf.strings.split(mask_rle)\n s = tf.strings.to_number(s, tf.int64)\n # Get starts and lengths\n starts = s[::2] - 1\n lens = s[1::2]\n # Make ones to be scattered\n total_ones = tf.reduce_sum(lens)\n ones = tf.ones([total_ones], tf.uint8)\n # Make scattering indices\n r = tf.range(total_ones)\n lens_cum = tf.math.cumsum(lens)\n s = tf.searchsorted(lens_cum, r, 'right')\n idx = r + tf.gather(starts - tf.pad(lens_cum[:-1], [(1, 0)]), s)\n # Scatter ones into flattened mask\n mask_flat = tf.scatter_nd(tf.expand_dims(idx, 1), ones, [size])\n # Reshape into mask\n return tf.reshape(mask_flat, shape)\n\n\n```\n\nThis came from here:\nhttps://stackoverflow.com/questions/58693261/create-a-rle-run-lenth-encoding-mask-with-tensorflow-datasets\n\nfound this about a year ago for something else I needed..", "1545792": "great find! I think this makes it much easier to directly train on the string targets (rather than the expanded targets). also curious if this would receive a boost on gpus.", "1546716": "This might be helpful\n[https://www.kaggle.com/c/sartorius-cell-instance-segmentation/discussion/url](url)", "1547200": "Very nice. Here's another version that gets masks for all the images at once. Takes about 6 seconds.\n```\ndef make_masks(annos, shape=(520, 704, 1), color=1):\n mask = np.zeros((shape[0]*shape[1], shape[2]))\n for rle_str in annos:\n runs = np.fromstring(rle_str, sep=\" \", dtype=np.int32).reshape(-1, 2)\n runs[:,0] -= 1\n runs[:,1] = runs.sum(axis=1)\n for r in runs:\n mask[r[0]:r[1]] = color\n return mask.reshape(shape)\n\nmasks = df_train.groupby('id')['annotation'].apply(make_masks)\n\n```\n\nThis numba version is more complex and runs in under a second.\n```\n@njit\ndef color_mask(runs, mask, color=1):\n runs[::2]-=1\n runs[1::2]+=runs[::2]\n for i in np.arange(len(runs)//2):\n m = i*2\n mask[runs[m]:runs[m+1]] = color\n \ndef make_masks_split(annos, shape=(520, 704, 1)):\n mask = np.zeros((np.prod(shape[:2]), 1), dtype=np.int32)\n for rle_str in annos:\n runs = np.fromstring(rle_str, sep=\" \", \n dtype=np.int32)\n color_mask(runs, mask)\n return mask.reshape(shape)\n\ntrain_array = df_train[['id', 'annotation']].to_numpy()\nuniques = np.unique(train_array[:,0], return_inverse=True)\nmasks = []\nfor i in np.arange(len(uniques[0])):\n mask = make_masks_split(train_array[uniques[1]==i, 1])\n masks.append(mask)\n```", "1548980": "I see you are decrementing both the starts and the lengths by one here:\n```\nruns = np.fromstring(rle_str, sep=\" \", dtype=np.int32).reshape(-1, 2)-1\n```\n\nin the original code by inversion, only the starts is decremented i think.", "1549047": "Absolutely right! Thanks for catching it - I'll change the code.", "1588322": "Dose it help improve performance?" }, "source": "meta" }