{ "id": 395067, "title": "Run Length Encoding Explained", "url": "/competitions/vesuvius-challenge-ink-detection/discussion/395067", "author_name": "", "post_date": "2023-03-15T18:45:46.448039300Z", "votes": 40, "comment_count": 7, "views": 0, "content": "

Edit: I wanted to clear out that there are different ways of performing RLE:

\n\n

The different ordering results in completely different RLE encodings. However, regardless the ordering method, this post will give you a very clear understanding of how RLE works.

\n

Definition

\n

Wikipedia: Run-length encoding (RLE) is a form of lossless data compression in which runs of data (sequences in which the same data value occurs in many consecutive data elements) are stored as a single data value and count, rather than as the original run. This is most efficient on data that contains many such runs, for example, simple graphic images such as icons.

\n

Informal definition: It's essentially a mechanism we use to encode masks in Computer Vision. In semantic segmentation we have an image and its mask (image of the same shape but with 0s and 1s) and we encode the mask using RLE instead of creating a new mask image. This saves us a lot of disk space and its efficient.

\n

How RLE works

\n
    \n
  1. The top left corner is the first pixel. Pixels then are \"numbered\" incrementing from top to bottom and then from left to right.
    2. \n
  2. You start \"walking\" in the way numbers increment.
    3. \n
  3. When you encounter a \"colored\" pixel you annotate that pixel's number and start counting.
    4. \n
  4. When you stop encountering \"colored\" pixels the count stops.
    5. \n
  5. Repeat the process until you finish \"walking\" the image.
    6. \n
\n

\n

Example

\n

The following example has the letter \"E\" it's RLE encoding is: \"11 7 20 1 23 1 26 1 29 1 32 1 35 1 38 1 44 1\"

\n

Breakdown:

\n
    \n
  1. We start \"walking\" the image, the first colored pixel is at position 11, then we encounter 7 colored pixels and the count stops.
    2. \n
  2. Then we encounter a colored pixel at position 20 but the count stops at 1 since the next pixel is white.
    3. \n
  3. Repeat this process until we end the \"walk\" at position 54.
    4. \n
", "messages": [ { "id": "2183552", "postDate": "03/15/2023 18:45:46", "content": "

Edit: I wanted to clear out that there are different ways of performing RLE:

\n\n

The different ordering results in completely different RLE encodings. However, regardless the ordering method, this post will give you a very clear understanding of how RLE works.

\n

Definition

\n

Wikipedia: Run-length encoding (RLE) is a form of lossless data compression in which runs of data (sequences in which the same data value occurs in many consecutive data elements) are stored as a single data value and count, rather than as the original run. This is most efficient on data that contains many such runs, for example, simple graphic images such as icons.

\n

Informal definition: It's essentially a mechanism we use to encode masks in Computer Vision. In semantic segmentation we have an image and its mask (image of the same shape but with 0s and 1s) and we encode the mask using RLE instead of creating a new mask image. This saves us a lot of disk space and its efficient.

\n

How RLE works

\n
    \n
  1. The top left corner is the first pixel. Pixels then are \"numbered\" incrementing from top to bottom and then from left to right.
    2. \n
  2. You start \"walking\" in the way numbers increment.
    3. \n
  3. When you encounter a \"colored\" pixel you annotate that pixel's number and start counting.
    4. \n
  4. When you stop encountering \"colored\" pixels the count stops.
    5. \n
  5. Repeat the process until you finish \"walking\" the image.
    6. \n
\n

\n

Example

\n

The following example has the letter \"E\" it's RLE encoding is: \"11 7 20 1 23 1 26 1 29 1 32 1 35 1 38 1 44 1\"

\n

Breakdown:

\n
    \n
  1. We start \"walking\" the image, the first colored pixel is at position 11, then we encounter 7 colored pixels and the count stops.
    2. \n
  2. Then we encounter a colored pixel at position 20 but the count stops at 1 since the next pixel is white.
    3. \n
  3. Repeat this process until we end the \"walk\" at position 54.
    4. \n
", "rawMarkdown": "**Edit:** I wanted to clear out that there are different ways of performing RLE:\n- The method depicted on this topic's illustration corresponds to the \"F\" order, which enumerates pixels first from top to bottom then from left to right. \n- This competition uses the \"C\" order (which I think is more common), it enumerates pixels first from left to right then from top to bottom. \n\nThe different ordering results in completely different RLE encodings. However, regardless the ordering method, this post will give you a very clear understanding of how RLE works.\n\n### Definition\n\n**Wikipedia:** Run-length encoding (RLE) is a form of lossless data compression in which runs of data (sequences in which the same data value occurs in many consecutive data elements) are stored as a single data value and count, rather than as the original run. This is most efficient on data that contains many such runs, for example, simple graphic images such as icons.\n\n**Informal definition:** It's essentially a mechanism we use to encode masks in Computer Vision. In semantic segmentation we have an image and its mask (image of the same shape but with 0s and 1s) and we encode the mask using RLE instead of creating a new mask image. This saves us a lot of disk space and its efficient.\n\n### How RLE works\n\n1. The top left corner is the first pixel. Pixels then are \"numbered\" incrementing from top to bottom and then from left to right.\n2. You start \"walking\" in the way numbers increment. \n3. When you encounter a \"colored\" pixel you annotate that pixel's number and start counting.\n4. When you stop encountering \"colored\" pixels the count stops. \n5. Repeat the process until you finish \"walking\" the image.\n\n\n
\n\n### Example\n\nThe following example has the letter \"E\" it's RLE encoding is: \"11 7 20 1 23 1 26 1 29 1 32 1 35 1 38 1 44 1\"\n\nBreakdown:\n1. We start \"walking\" the image, the first colored pixel is at position 11, then we encounter 7 colored pixels and the count stops.\n2. Then we encounter a colored pixel at position 20 but the count stops at 1 since the next pixel is white.\n3. Repeat this process until we end the \"walk\" at position 54.", "votes": null }, { "id": "2183578", "postDate": "03/15/2023 19:10:29", "content": "

Nice, thanks for sharing!

\n

We've been using this snippet a bunch, e.g. in our notebook and example submission. :)

", "rawMarkdown": "Nice, thanks for sharing!\n\nWe've been using [this snippet](https://gist.github.com/janpaul123/ca3477c1db6de4346affca37e0e3d5b0) a bunch, e.g. in our [notebook](https://www.kaggle.com/jpposma/vesuvius-challenge-ink-detection-tutorial/edit) and [example submission](https://www.kaggle.com/code/danielhavir/vesuvius-challenge-example-submission?scriptVersionId=122259384&cellId=24). :)", "votes": null }, { "id": "2188495", "postDate": "03/19/2023 17:26:00", "content": "

it's very helpful

", "rawMarkdown": "it's very helpful", "votes": null }, { "id": "2280864", "postDate": "05/30/2023 12:33:47", "content": "

@jpposma , @wcukierski , @ryanholbrook

\n

Hello! May I confirm one thing? Like this topic, I believe that the scoring is also being done in C order this time.

\n
This competition uses the \"C\" order (which I think is more common), it enumerates pixels first from left to right then from top to bottom.\n
\n

The following is an example using your encoding code from here

\n
test = np.array([\n    [0,1,1,0],\n    [1,0,0,0],\n    [0,0,1,0]\n])\n\ndef rle (img):\n    flat_img = img.flatten()\n    flat_img = np.where(flat_img > 0.5, 1, 0).astype(np.uint8)\n\n    starts = np.array((flat_img[:-1] == 0) & (flat_img[1:] == 1))\n    ends = np.array((flat_img[:-1] == 1) & (flat_img[1:] == 0))\n    starts_ix = np.where(starts)[0] + 2\n    ends_ix = np.where(ends)[0] + 2\n    lengths = ends_ix - starts_ix\n\n    return starts_ix, lengths\n\n#inklabels = np.array(Image.open('inklabels.png'), dtype=np.uint8)\nstarts_ix, lengths = rle(test)\ninklabels_rle = \" \".join(map(str, sum(zip(starts_ix, lengths), ())))\n#print(\"Id,Predicted\\n1,\" + inklabels_rle, file=open('inklabels_rle.csv', 'w'))\nprint(inklabels_rle)\n
\n

This answer is : 2 2 5 1 11 1

\n

However, the explanation on the evaluation page of this competition here states the following:

\n
The pixels are numbered from top to bottom, then left to right: 1 is pixel (1,1), 2 is pixel (2,1), etc.\n
\n

This is an explanation for Type F, and I believe it is incorrect for this competition.
\nTo avoid confusion, could you please confirm and make the necessary corrections?
\n(I believe the ongoing competition \"Google Research - Identify Contrails to Reduce Global Warming\" follows the F order.)

\n

If I made a mistake, I apologize. Please let me know accordingly.
\nThank you in advance.

", "rawMarkdown": "jpposma , @wcukierski , @ryanholbrook \n\nHello! May I confirm one thing? Like this topic, I believe that the scoring is also being done in C order this time.\n\n~~~\nThis competition uses the \"C\" order (which I think is more common), it enumerates pixels first from left to right then from top to bottom.\n~~~\n\nThe following is an example using your encoding code [from here](https://gist.github.com/janpaul123/ca3477c1db6de4346affca37e0e3d5b0)\n\n~~~\ntest = np.array([\n [0,1,1,0],\n [1,0,0,0],\n [0,0,1,0]\n])\n\ndef rle (img):\n flat_img = img.flatten()\n flat_img = np.where(flat_img > 0.5, 1, 0).astype(np.uint8)\n\n starts = np.array((flat_img[:-1] == 0) & (flat_img[1:] == 1))\n ends = np.array((flat_img[:-1] == 1) & (flat_img[1:] == 0))\n starts_ix = np.where(starts)[0] + 2\n ends_ix = np.where(ends)[0] + 2\n lengths = ends_ix - starts_ix\n\n return starts_ix, lengths\n\n#inklabels = np.array(Image.open('inklabels.png'), dtype=np.uint8)\nstarts_ix, lengths = rle(test)\ninklabels_rle = \" \".join(map(str, sum(zip(starts_ix, lengths), ())))\n#print(\"Id,Predicted\\n1,\" + inklabels_rle, file=open('inklabels_rle.csv', 'w'))\nprint(inklabels_rle)\n\n~~~\nThis answer is : 2 2 5 1 11 1\n\nHowever, the explanation on the evaluation page of this competition [here](https://www.kaggle.com/competitions/vesuvius-challenge-ink-detection/overview/evaluation) states the following:\n~~~\nThe pixels are numbered from top to bottom, then left to right: 1 is pixel (1,1), 2 is pixel (2,1), etc.\n~~~\n\nThis is an explanation for Type F, and I believe it is incorrect for this competition.\nTo avoid confusion, could you please confirm and make the necessary corrections?\n(I believe the ongoing competition \"Google Research - Identify Contrails to Reduce Global Warming\" follows the F order.)\n\nIf I made a mistake, I apologize. Please let me know accordingly.\nThank you in advance.", "votes": null }, { "id": "2281302", "postDate": "05/30/2023 18:45:38", "content": "

Hi @chumajin, Thanks for the heads up! Let me confirm and I will correct as necessary.

", "rawMarkdown": "Hi @chumajin, Thanks for the heads up! Let me confirm and I will correct as necessary.", "votes": null }, { "id": "2281401", "postDate": "05/30/2023 20:58:34", "content": "

This competition does indeed use Type C order. I'll update the evaluation page. Thanks again!

", "rawMarkdown": "This competition does indeed use ~~Type F~~ Type C order. I'll update the evaluation page. Thanks again!", "votes": null }, { "id": "2281550", "postDate": "05/31/2023 01:37:20", "content": "

@ryanholbrook

\n

Thank you for checking and making the corrections so promptly! Right now, the evaluation page looks correct!!

\n

However, It seems like your comment below is interpreting Type C and F in reverse.

\n
This competition does indeed use Type F order.\n
\n

This competition is using Type C.

\n

ā€» type C : pixels first from left to right then from top to bottom (this competition. evaluation page was fixed.)
\nā€» type F : pixels first from top to bottom then from left to right.

\n

Thank you for always!!

", "rawMarkdown": "ryanholbrook \n\nThank you for checking and making the corrections so promptly! Right now, the evaluation page looks correct!!\n\nHowever, It seems like your comment below is interpreting Type C and F in reverse.\n\n~~~\nThis competition does indeed use Type F order.\n~~~\n\nThis competition is using Type C.\n\nā€» type C : pixels first from left to right then from top to bottom (this competition. evaluation page was fixed.)\nā€» type F : pixels first from top to bottom then from left to right.\n\nThank you for always!!", "votes": null }, { "id": "2282254", "postDate": "05/31/2023 13:32:06", "content": "

Yes, apologies! šŸ˜…

", "rawMarkdown": "Yes, apologies! šŸ˜…", "votes": null } ], "comments": [ { "id": 2183578, "author_name": "jpposma", "author_url": "", "post_date": "03/15/2023 19:10:29", "content": "

Nice, thanks for sharing!

\n

We've been using this snippet a bunch, e.g. in our notebook and example submission. :)

", "votes": null, "replies": [] }, { "id": 2188495, "author_name": "ahmedelkelany", "author_url": "", "post_date": "03/19/2023 17:26:00", "content": "

it's very helpful

", "votes": null, "replies": [] }, { "id": 2280864, "author_name": "chumajin", "author_url": "", "post_date": "05/30/2023 12:33:47", "content": "

@jpposma , @wcukierski , @ryanholbrook

\n

Hello! May I confirm one thing? Like this topic, I believe that the scoring is also being done in C order this time.

\n
This competition uses the \"C\" order (which I think is more common), it enumerates pixels first from left to right then from top to bottom.\n
\n

The following is an example using your encoding code from here

\n
test = np.array([\n    [0,1,1,0],\n    [1,0,0,0],\n    [0,0,1,0]\n])\n\ndef rle (img):\n    flat_img = img.flatten()\n    flat_img = np.where(flat_img > 0.5, 1, 0).astype(np.uint8)\n\n    starts = np.array((flat_img[:-1] == 0) & (flat_img[1:] == 1))\n    ends = np.array((flat_img[:-1] == 1) & (flat_img[1:] == 0))\n    starts_ix = np.where(starts)[0] + 2\n    ends_ix = np.where(ends)[0] + 2\n    lengths = ends_ix - starts_ix\n\n    return starts_ix, lengths\n\n#inklabels = np.array(Image.open('inklabels.png'), dtype=np.uint8)\nstarts_ix, lengths = rle(test)\ninklabels_rle = \" \".join(map(str, sum(zip(starts_ix, lengths), ())))\n#print(\"Id,Predicted\\n1,\" + inklabels_rle, file=open('inklabels_rle.csv', 'w'))\nprint(inklabels_rle)\n
\n

This answer is : 2 2 5 1 11 1

\n

However, the explanation on the evaluation page of this competition here states the following:

\n
The pixels are numbered from top to bottom, then left to right: 1 is pixel (1,1), 2 is pixel (2,1), etc.\n
\n

This is an explanation for Type F, and I believe it is incorrect for this competition.
\nTo avoid confusion, could you please confirm and make the necessary corrections?
\n(I believe the ongoing competition \"Google Research - Identify Contrails to Reduce Global Warming\" follows the F order.)

\n

If I made a mistake, I apologize. Please let me know accordingly.
\nThank you in advance.

", "votes": null, "replies": [ { "id": 2281302, "author_name": "ryanholbrook", "author_url": "", "post_date": "05/30/2023 18:45:38", "content": "

Hi @chumajin, Thanks for the heads up! Let me confirm and I will correct as necessary.

", "votes": null, "replies": [] }, { "id": 2281401, "author_name": "ryanholbrook", "author_url": "", "post_date": "05/30/2023 20:58:34", "content": "

This competition does indeed use Type C order. I'll update the evaluation page. Thanks again!

", "votes": null, "replies": [ { "id": 2281550, "author_name": "chumajin", "author_url": "", "post_date": "05/31/2023 01:37:20", "content": "

@ryanholbrook

\n

Thank you for checking and making the corrections so promptly! Right now, the evaluation page looks correct!!

\n

However, It seems like your comment below is interpreting Type C and F in reverse.

\n
This competition does indeed use Type F order.\n
\n

This competition is using Type C.

\n

ā€» type C : pixels first from left to right then from top to bottom (this competition. evaluation page was fixed.)
\nā€» type F : pixels first from top to bottom then from left to right.

\n

Thank you for always!!

", "votes": null, "replies": [ { "id": 2282254, "author_name": "ryanholbrook", "author_url": "", "post_date": "05/31/2023 13:32:06", "content": "

Yes, apologies! šŸ˜…

", "votes": null, "replies": [] } ] } ] } ] } ], "raw_markdown_by_id": { "2183552": "**Edit:** I wanted to clear out that there are different ways of performing RLE:\n- The method depicted on this topic's illustration corresponds to the \"F\" order, which enumerates pixels first from top to bottom then from left to right. \n- This competition uses the \"C\" order (which I think is more common), it enumerates pixels first from left to right then from top to bottom. \n\nThe different ordering results in completely different RLE encodings. However, regardless the ordering method, this post will give you a very clear understanding of how RLE works.\n\n### Definition\n\n**Wikipedia:** Run-length encoding (RLE) is a form of lossless data compression in which runs of data (sequences in which the same data value occurs in many consecutive data elements) are stored as a single data value and count, rather than as the original run. This is most efficient on data that contains many such runs, for example, simple graphic images such as icons.\n\n**Informal definition:** It's essentially a mechanism we use to encode masks in Computer Vision. In semantic segmentation we have an image and its mask (image of the same shape but with 0s and 1s) and we encode the mask using RLE instead of creating a new mask image. This saves us a lot of disk space and its efficient.\n\n### How RLE works\n\n1. The top left corner is the first pixel. Pixels then are \"numbered\" incrementing from top to bottom and then from left to right.\n2. You start \"walking\" in the way numbers increment. \n3. When you encounter a \"colored\" pixel you annotate that pixel's number and start counting.\n4. When you stop encountering \"colored\" pixels the count stops. \n5. Repeat the process until you finish \"walking\" the image.\n\n\n
\n\n### Example\n\nThe following example has the letter \"E\" it's RLE encoding is: \"11 7 20 1 23 1 26 1 29 1 32 1 35 1 38 1 44 1\"\n\nBreakdown:\n1. We start \"walking\" the image, the first colored pixel is at position 11, then we encounter 7 colored pixels and the count stops.\n2. Then we encounter a colored pixel at position 20 but the count stops at 1 since the next pixel is white.\n3. Repeat this process until we end the \"walk\" at position 54.", "2183578": "Nice, thanks for sharing!\n\nWe've been using [this snippet](https://gist.github.com/janpaul123/ca3477c1db6de4346affca37e0e3d5b0) a bunch, e.g. in our [notebook](https://www.kaggle.com/jpposma/vesuvius-challenge-ink-detection-tutorial/edit) and [example submission](https://www.kaggle.com/code/danielhavir/vesuvius-challenge-example-submission?scriptVersionId=122259384&cellId=24). :)", "2188495": "it's very helpful", "2280864": "jpposma , @wcukierski , @ryanholbrook \n\nHello! May I confirm one thing? Like this topic, I believe that the scoring is also being done in C order this time.\n\n~~~\nThis competition uses the \"C\" order (which I think is more common), it enumerates pixels first from left to right then from top to bottom.\n~~~\n\nThe following is an example using your encoding code [from here](https://gist.github.com/janpaul123/ca3477c1db6de4346affca37e0e3d5b0)\n\n~~~\ntest = np.array([\n [0,1,1,0],\n [1,0,0,0],\n [0,0,1,0]\n])\n\ndef rle (img):\n flat_img = img.flatten()\n flat_img = np.where(flat_img > 0.5, 1, 0).astype(np.uint8)\n\n starts = np.array((flat_img[:-1] == 0) & (flat_img[1:] == 1))\n ends = np.array((flat_img[:-1] == 1) & (flat_img[1:] == 0))\n starts_ix = np.where(starts)[0] + 2\n ends_ix = np.where(ends)[0] + 2\n lengths = ends_ix - starts_ix\n\n return starts_ix, lengths\n\n#inklabels = np.array(Image.open('inklabels.png'), dtype=np.uint8)\nstarts_ix, lengths = rle(test)\ninklabels_rle = \" \".join(map(str, sum(zip(starts_ix, lengths), ())))\n#print(\"Id,Predicted\\n1,\" + inklabels_rle, file=open('inklabels_rle.csv', 'w'))\nprint(inklabels_rle)\n\n~~~\nThis answer is : 2 2 5 1 11 1\n\nHowever, the explanation on the evaluation page of this competition [here](https://www.kaggle.com/competitions/vesuvius-challenge-ink-detection/overview/evaluation) states the following:\n~~~\nThe pixels are numbered from top to bottom, then left to right: 1 is pixel (1,1), 2 is pixel (2,1), etc.\n~~~\n\nThis is an explanation for Type F, and I believe it is incorrect for this competition.\nTo avoid confusion, could you please confirm and make the necessary corrections?\n(I believe the ongoing competition \"Google Research - Identify Contrails to Reduce Global Warming\" follows the F order.)\n\nIf I made a mistake, I apologize. Please let me know accordingly.\nThank you in advance.", "2281302": "Hi @chumajin, Thanks for the heads up! Let me confirm and I will correct as necessary.", "2281401": "This competition does indeed use ~~Type F~~ Type C order. I'll update the evaluation page. Thanks again!", "2281550": "ryanholbrook \n\nThank you for checking and making the corrections so promptly! Right now, the evaluation page looks correct!!\n\nHowever, It seems like your comment below is interpreting Type C and F in reverse.\n\n~~~\nThis competition does indeed use Type F order.\n~~~\n\nThis competition is using Type C.\n\nā€» type C : pixels first from left to right then from top to bottom (this competition. evaluation page was fixed.)\nā€» type F : pixels first from top to bottom then from left to right.\n\nThank you for always!!", "2282254": "Yes, apologies! šŸ˜…" }, "source": "meta" }