Is there an efficient way to extract from a mask image/array the multi-polygon format needed for submission file? I found the rasterio package that kinda works but it is extremely slow. The data processing tutorial will be more helpful if it included libraries or samples to generate submission file.
\n\nimport rasterio.features
\n\nfor vec in rasterio.features.shapes(MASKArray):\n print(vec)\n\n//prints ({'type': 'Polygon', 'coordinates': [[(1952.0, 0.0), (1952.0, 3.0), (1967.0, 3.0 ), (1967.0, 0.0), (1952.0, 0.0)]]}, 1.0)\nIs there an efficient way to extract from a mask image/array the multi-polygon format needed for submission file? I found the rasterio package that kinda works but it is extremely slow. The data processing tutorial will be more helpful if it included libraries or samples to generate submission file.
\n\nimport rasterio.features
\n\nfor vec in rasterio.features.shapes(MASKArray):\n print(vec)\n\n//prints ({'type': 'Polygon', 'coordinates': [[(1952.0, 0.0), (1952.0, 3.0), (1967.0, 3.0 ), (1967.0, 0.0), (1952.0, 0.0)]]}, 1.0)\nI'm planning to work on this problem, but it might be a while before I get to it because of the holidays. I think a basic approach follows. You can define any mask by its minimum and maximum y value, and then for each pixel layer within that defining its min and max x value. You can perform additional checks for gaps in the mask that are not continuous. In this way you could go layer by layer and define a vertex for every single bounding pixel of the mask. This will almost certainly be much too high resolution for the polygon definition, so from here you could downsample by taking only every 3rd or 5th or 10th vertex, etc. You could also find ways to further decrease the number of vertices by requiring a minimum change in normal vector before defining a new vertex
", "rawMarkdown": "I'm planning to work on this problem, but it might be a while before I get to it because of the holidays. I think a basic approach follows. You can define any mask by its minimum and maximum y value, and then for each pixel layer within that defining its min and max x value. You can perform additional checks for gaps in the mask that are not continuous. In this way you could go layer by layer and define a vertex for every single bounding pixel of the mask. This will almost certainly be much too high resolution for the polygon definition, so from here you could downsample by taking only every 3rd or 5th or 10th vertex, etc. You could also find ways to further decrease the number of vertices by requiring a minimum change in normal vector before defining a new vertex", "votes": null }, { "id": "152422", "postDate": "12/26/2016 03:16:11", "content": "@Alan thanks! That sounds really complicated, I was hoping there was some one line function somewhere that would take bitmap or 2D numpy array and return the vector coordinates for the submission. The masks can contain hundered or thousands of polygons and dealing with the holes will be tricky as well not to add the impact this will have on the submission score.
", "rawMarkdown": "Alan thanks! That sounds really complicated, I was hoping there was some one line function somewhere that would take bitmap or 2D numpy array and return the vector coordinates for the submission. The masks can contain hundered or thousands of polygons and dealing with the holes will be tricky as well not to add the impact this will have on the submission score.", "votes": null }, { "id": "152423", "postDate": "12/26/2016 03:17:06", "content": "David, how slow is it? I'm using that same method and it's plenty fast for me.
", "rawMarkdown": "David, how slow is it? I'm using that same method and it's plenty fast for me.", "votes": null }, { "id": "152424", "postDate": "12/26/2016 03:22:49", "content": "@shawn, How big is your mask, I tried with the original ~3300 x ~3300 image, Maybe that is the problem. It's pretty slow, could be the pc I am testing things out on as well.
", "rawMarkdown": "shawn, How big is your mask, I tried with the original ~3300 x ~3300 image, Maybe that is the problem. It's pretty slow, could be the pc I am testing things out on as well.", "votes": null }, { "id": "152426", "postDate": "12/26/2016 03:35:08", "content": "Yeah, that's probably it. I'm using the smallest A image, only 134x134.
", "rawMarkdown": "Yeah, that's probably it. I'm using the smallest A image, only 134x134.", "votes": null }, { "id": "152428", "postDate": "12/26/2016 04:02:08", "content": "Also make sure you have the latest 1.0 rasterio version. Version 0.3 was still floating around in the conda repo.
", "rawMarkdown": "Also make sure you have the latest 1.0 rasterio version. Version 0.3 was still floating around in the conda repo.", "votes": null }, { "id": "152440", "postDate": "12/26/2016 08:01:20", "content": "Hi!
\n\nWhat's wrong with opencv? It can extract connected components and return tree structures (patches inside holes of other patches containing holes with patches, well you get the point).\nAnd wkt library used for loading, permits constructing objects \"on the fly\" and then serialize them.\nI mean this is I would do. (I am not there yet)
", "rawMarkdown": "Hi! \r\n\r\nWhat's wrong with opencv? It can extract connected components and return tree structures (patches inside holes of other patches containing holes with patches, well you get the point).\r\nAnd wkt library used for loading, permits constructing objects \"on the fly\" and then serialize them.\r\nI mean this is I would do. (I am not there yet)", "votes": null }, { "id": "152477", "postDate": "12/26/2016 15:56:06", "content": "@visoft Thanks - FYI rasterio was slow because of the PC I was using or version, runs much faster on a different instance, I also updated to version 1.0a4 it was 0.25. It would be nice if you indicate the opencv components to achieve similar results. The author of shapely the wkt library you mentioned is also the author of rasterio so, both should play nicely with each other.
", "rawMarkdown": "visoft Thanks - FYI rasterio was slow because of the PC I was using or version, runs much faster on a different instance, I also updated to version 1.0a4 it was 0.25. It would be nice if you indicate the opencv components to achieve similar results. The author of shapely the wkt library you mentioned is also the author of rasterio so, both should play nicely with each other.", "votes": null }, { "id": "152510", "postDate": "12/26/2016 20:19:45", "content": "I tried a few methods for doing this, and I had the same problem with speed. However, I eventually came across openCV's fillPoly function, and that get's the job done in a couple seconds.
Here's a notebook showing how I did it: Turn JSON polygons into a mask using openCV
", "rawMarkdown": "I tried a few methods for doing this, and I had the same problem with speed. However, I eventually came across openCV's `fillPoly` function, and that get's the job done in a couple seconds. \r\n\r\nHere's a notebook showing how I did it: [Turn JSON polygons into a mask using openCV][1]\r\n\r\n\r\n [1]: https://www.kaggle.com/chatcat/dstl-satellite-imagery-feature-detection/notebook90ddd2fb97/", "votes": null }, { "id": "152512", "postDate": "12/26/2016 20:33:39", "content": "@Alan you generated a mask from the JSON polygons. This thread refers to the opposite, deriving the polygon co-ordinates from the image mask. \"Turn random mask into JSON polygons\"
", "rawMarkdown": "Alan you generated a mask from the JSON polygons. This thread refers to the opposite, deriving the polygon co-ordinates from the image mask. \"Turn random mask into JSON polygons\"", "votes": null }, { "id": "152513", "postDate": "12/26/2016 20:41:01", "content": "Oops! I used openCV for that too lol. Again, I find that it works pretty quickly (by my standards).
\n\n im = np.array(mask * 255, dtype = np.uint8)\n image, contours, hierarchy = cv2.findContours(im, cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_SIMPLE)\n@Alan, Thx!!
", "rawMarkdown": "Alan, Thx!!", "votes": null }, { "id": "152522", "postDate": "12/26/2016 22:17:09", "content": "Generating a mask from polygons and polygons from mask is really two sides of the same coin from the method I described. If there are existing methods in packages like openCV to do this I would highly recommend using them as they will be more robust and faster. I just haven't looked and haven't coded this up myself yet because I am home for the holidays.
\n\nOnce I'm back in the swing of things I'll take a closer look, and if it is necessary to write the code by hand I will be sure to share it with everybody.
", "rawMarkdown": "Generating a mask from polygons and polygons from mask is really two sides of the same coin from the method I described. If there are existing methods in packages like openCV to do this I would highly recommend using them as they will be more robust and faster. I just haven't looked and haven't coded this up myself yet because I am home for the holidays. \r\n\r\nOnce I'm back in the swing of things I'll take a closer look, and if it is necessary to write the code by hand I will be sure to share it with everybody.", "votes": null }, { "id": "152558", "postDate": "12/27/2016 06:29:14", "content": "I think opencv's approxpolyDP could be the first choice for me. This could add to the list of hyperparameters :D
", "rawMarkdown": "I think opencv's [approxpolyDP][1] could be the first choice for me. This could add to the list of hyperparameters :D\r\n\r\n\r\n [1]: http://docs.opencv.org/3.1.0/dd/d49/tutorial_py_contour_features.html", "votes": null } ], "comments": [ { "id": 152421, "author_name": "apryor6", "author_url": "", "post_date": "12/26/2016 02:31:47", "content": "I'm planning to work on this problem, but it might be a while before I get to it because of the holidays. I think a basic approach follows. You can define any mask by its minimum and maximum y value, and then for each pixel layer within that defining its min and max x value. You can perform additional checks for gaps in the mask that are not continuous. In this way you could go layer by layer and define a vertex for every single bounding pixel of the mask. This will almost certainly be much too high resolution for the polygon definition, so from here you could downsample by taking only every 3rd or 5th or 10th vertex, etc. You could also find ways to further decrease the number of vertices by requiring a minimum change in normal vector before defining a new vertex
", "votes": null, "replies": [] }, { "id": 152422, "author_name": "godaibo", "author_url": "", "post_date": "12/26/2016 03:16:11", "content": "@Alan thanks! That sounds really complicated, I was hoping there was some one line function somewhere that would take bitmap or 2D numpy array and return the vector coordinates for the submission. The masks can contain hundered or thousands of polygons and dealing with the holes will be tricky as well not to add the impact this will have on the submission score.
", "votes": null, "replies": [] }, { "id": 152423, "author_name": "shawn775", "author_url": "", "post_date": "12/26/2016 03:17:06", "content": "David, how slow is it? I'm using that same method and it's plenty fast for me.
", "votes": null, "replies": [] }, { "id": 152424, "author_name": "godaibo", "author_url": "", "post_date": "12/26/2016 03:22:49", "content": "@shawn, How big is your mask, I tried with the original ~3300 x ~3300 image, Maybe that is the problem. It's pretty slow, could be the pc I am testing things out on as well.
", "votes": null, "replies": [] }, { "id": 152426, "author_name": "shawn775", "author_url": "", "post_date": "12/26/2016 03:35:08", "content": "Yeah, that's probably it. I'm using the smallest A image, only 134x134.
", "votes": null, "replies": [] }, { "id": 152428, "author_name": "shawn775", "author_url": "", "post_date": "12/26/2016 04:02:08", "content": "Also make sure you have the latest 1.0 rasterio version. Version 0.3 was still floating around in the conda repo.
", "votes": null, "replies": [] }, { "id": 152440, "author_name": "visoft", "author_url": "", "post_date": "12/26/2016 08:01:20", "content": "Hi!
\n\nWhat's wrong with opencv? It can extract connected components and return tree structures (patches inside holes of other patches containing holes with patches, well you get the point).\nAnd wkt library used for loading, permits constructing objects \"on the fly\" and then serialize them.\nI mean this is I would do. (I am not there yet)
", "votes": null, "replies": [] }, { "id": 152477, "author_name": "godaibo", "author_url": "", "post_date": "12/26/2016 15:56:06", "content": "@visoft Thanks - FYI rasterio was slow because of the PC I was using or version, runs much faster on a different instance, I also updated to version 1.0a4 it was 0.25. It would be nice if you indicate the opencv components to achieve similar results. The author of shapely the wkt library you mentioned is also the author of rasterio so, both should play nicely with each other.
", "votes": null, "replies": [] }, { "id": 152510, "author_name": "chatcat", "author_url": "", "post_date": "12/26/2016 20:19:45", "content": "I tried a few methods for doing this, and I had the same problem with speed. However, I eventually came across openCV's fillPoly function, and that get's the job done in a couple seconds.
Here's a notebook showing how I did it: Turn JSON polygons into a mask using openCV
", "votes": null, "replies": [] }, { "id": 152512, "author_name": "godaibo", "author_url": "", "post_date": "12/26/2016 20:33:39", "content": "@Alan you generated a mask from the JSON polygons. This thread refers to the opposite, deriving the polygon co-ordinates from the image mask. \"Turn random mask into JSON polygons\"
", "votes": null, "replies": [] }, { "id": 152513, "author_name": "chatcat", "author_url": "", "post_date": "12/26/2016 20:41:01", "content": "Oops! I used openCV for that too lol. Again, I find that it works pretty quickly (by my standards).
\n\n im = np.array(mask * 255, dtype = np.uint8)\n image, contours, hierarchy = cv2.findContours(im, cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_SIMPLE)\n@Alan, Thx!!
", "votes": null, "replies": [] }, { "id": 152522, "author_name": "apryor6", "author_url": "", "post_date": "12/26/2016 22:17:09", "content": "Generating a mask from polygons and polygons from mask is really two sides of the same coin from the method I described. If there are existing methods in packages like openCV to do this I would highly recommend using them as they will be more robust and faster. I just haven't looked and haven't coded this up myself yet because I am home for the holidays.
\n\nOnce I'm back in the swing of things I'll take a closer look, and if it is necessary to write the code by hand I will be sure to share it with everybody.
", "votes": null, "replies": [] }, { "id": 152558, "author_name": "srika91", "author_url": "", "post_date": "12/27/2016 06:29:14", "content": "I think opencv's approxpolyDP could be the first choice for me. This could add to the list of hyperparameters :D
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "152414": "Is there an efficient way to extract from a mask image/array the multi-polygon format needed for submission file? I found the rasterio package that kinda works but it is extremely slow. The data processing tutorial will be more helpful if it included libraries or samples to generate submission file.\r\n\r\n import rasterio.features\r\n \r\n for vec in rasterio.features.shapes(MASKArray):\r\n print(vec)\r\n\r\n //prints ({'type': 'Polygon', 'coordinates': [[(1952.0, 0.0), (1952.0, 3.0), (1967.0, 3.0 ), (1967.0, 0.0), (1952.0, 0.0)]]}, 1.0)", "152421": "I'm planning to work on this problem, but it might be a while before I get to it because of the holidays. I think a basic approach follows. You can define any mask by its minimum and maximum y value, and then for each pixel layer within that defining its min and max x value. You can perform additional checks for gaps in the mask that are not continuous. In this way you could go layer by layer and define a vertex for every single bounding pixel of the mask. This will almost certainly be much too high resolution for the polygon definition, so from here you could downsample by taking only every 3rd or 5th or 10th vertex, etc. You could also find ways to further decrease the number of vertices by requiring a minimum change in normal vector before defining a new vertex", "152422": "Alan thanks! That sounds really complicated, I was hoping there was some one line function somewhere that would take bitmap or 2D numpy array and return the vector coordinates for the submission. The masks can contain hundered or thousands of polygons and dealing with the holes will be tricky as well not to add the impact this will have on the submission score.", "152423": "David, how slow is it? I'm using that same method and it's plenty fast for me.", "152424": "shawn, How big is your mask, I tried with the original ~3300 x ~3300 image, Maybe that is the problem. It's pretty slow, could be the pc I am testing things out on as well.", "152426": "Yeah, that's probably it. I'm using the smallest A image, only 134x134.", "152428": "Also make sure you have the latest 1.0 rasterio version. Version 0.3 was still floating around in the conda repo.", "152440": "Hi! \r\n\r\nWhat's wrong with opencv? It can extract connected components and return tree structures (patches inside holes of other patches containing holes with patches, well you get the point).\r\nAnd wkt library used for loading, permits constructing objects \"on the fly\" and then serialize them.\r\nI mean this is I would do. (I am not there yet)", "152477": "visoft Thanks - FYI rasterio was slow because of the PC I was using or version, runs much faster on a different instance, I also updated to version 1.0a4 it was 0.25. It would be nice if you indicate the opencv components to achieve similar results. The author of shapely the wkt library you mentioned is also the author of rasterio so, both should play nicely with each other.", "152510": "I tried a few methods for doing this, and I had the same problem with speed. However, I eventually came across openCV's `fillPoly` function, and that get's the job done in a couple seconds. \r\n\r\nHere's a notebook showing how I did it: [Turn JSON polygons into a mask using openCV][1]\r\n\r\n\r\n [1]: https://www.kaggle.com/chatcat/dstl-satellite-imagery-feature-detection/notebook90ddd2fb97/", "152512": "Alan you generated a mask from the JSON polygons. This thread refers to the opposite, deriving the polygon co-ordinates from the image mask. \"Turn random mask into JSON polygons\"", "152513": "Oops! I used openCV for that too lol. Again, I find that it works pretty quickly (by my standards).\r\n\r\n\r\n im = np.array(mask * 255, dtype = np.uint8)\r\n image, contours, hierarchy = cv2.findContours(im, cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_SIMPLE)", "152514": "Alan, Thx!!", "152522": "Generating a mask from polygons and polygons from mask is really two sides of the same coin from the method I described. If there are existing methods in packages like openCV to do this I would highly recommend using them as they will be more robust and faster. I just haven't looked and haven't coded this up myself yet because I am home for the holidays. \r\n\r\nOnce I'm back in the swing of things I'll take a closer look, and if it is necessary to write the code by hand I will be sure to share it with everybody.", "152558": "I think opencv's [approxpolyDP][1] could be the first choice for me. This could add to the list of hyperparameters :D\r\n\r\n\r\n [1]: http://docs.opencv.org/3.1.0/dd/d49/tutorial_py_contour_features.html" }, "source": "meta" }