{
  "id": 27351,
  "title": "TopologyException, CSV size and other tricks",
  "url": "/competitions/dstl-satellite-imagery-feature-detection/discussion/27351",
  "author_name": "",
  "post_date": "2017-01-05T17:58:32.390Z",
  "votes": 9,
  "comment_count": 11,
  "views": 556,
  "content": "<ul>\n<li><p>Smooth your polygons first as people said before. Out of curiosity, what are typical CSV sizes people are submitting? 300M, 500M?</p></li>\n<li><p>shapely.is_valid uses GEOS to check validity and Kaggle use JTS (they differ, causing lots of trouble).</p></li>\n<li><p>when submission fails with TopologyException, it gives only the first error. We have no way of getting the list of all invalid polygons. And the competition didn't provide a way to check validity off-line thus making troubleshooting difficult.</p></li>\n<li><p>size of submission file: shapely.wkt.dumps(mp, rounding_precision=10), or 8. Use local validation to check the reduced precision is not affecting your score significantly.  And beware truncating the numbers can generate bad polygons as well. Before the final stages of the competition, I think speed of iteration is better than precision. Precision==8 reduces csv size in half.</p></li>\n<li><p>using polygon.buffer(0) helps sometimes</p></li>\n<li><p>polygon.simplify(e, preserveTopology=False) sometimes generates invalid polygons</p></li>\n<li><p>sometimes other polygon operations generate invalid polygons.</p></li>\n<li><p>the algorithm you use for generating polygons can increase/decrease the probability of invalid geometries. I'm several without being able to submit a valid file for now.</p></li>\n</ul>\n\n<p>And as far as I know there's no easy way to guarantee generated polygons are valid. We need to post-process the polygons checking for validity. </p>\n\n<p>For that purpose I wrote a tool to help me, using the same underlying validity check as Kaggle. At least now I now which ones are invalid.</p>\n\n<p><a href=\"https://github.com/cxz/tpex\">https://github.com/cxz/tpex</a>\n(sorry the library is in Java)</p>",
  "messages": [
    {
      "id": "154302",
      "postDate": "01/05/2017 17:58:32",
      "content": "<ul>\n<li><p>Smooth your polygons first as people said before. Out of curiosity, what are typical CSV sizes people are submitting? 300M, 500M?</p></li>\n<li><p>shapely.is_valid uses GEOS to check validity and Kaggle use JTS (they differ, causing lots of trouble).</p></li>\n<li><p>when submission fails with TopologyException, it gives only the first error. We have no way of getting the list of all invalid polygons. And the competition didn't provide a way to check validity off-line thus making troubleshooting difficult.</p></li>\n<li><p>size of submission file: shapely.wkt.dumps(mp, rounding_precision=10), or 8. Use local validation to check the reduced precision is not affecting your score significantly.  And beware truncating the numbers can generate bad polygons as well. Before the final stages of the competition, I think speed of iteration is better than precision. Precision==8 reduces csv size in half.</p></li>\n<li><p>using polygon.buffer(0) helps sometimes</p></li>\n<li><p>polygon.simplify(e, preserveTopology=False) sometimes generates invalid polygons</p></li>\n<li><p>sometimes other polygon operations generate invalid polygons.</p></li>\n<li><p>the algorithm you use for generating polygons can increase/decrease the probability of invalid geometries. I'm several without being able to submit a valid file for now.</p></li>\n</ul>\n\n<p>And as far as I know there's no easy way to guarantee generated polygons are valid. We need to post-process the polygons checking for validity. </p>\n\n<p>For that purpose I wrote a tool to help me, using the same underlying validity check as Kaggle. At least now I now which ones are invalid.</p>\n\n<p><a href=\"https://github.com/cxz/tpex\">https://github.com/cxz/tpex</a>\n(sorry the library is in Java)</p>",
      "rawMarkdown": "- Smooth your polygons first as people said before. Out of curiosity, what are typical CSV sizes people are submitting? 300M, 500M?\r\n\r\n- shapely.is_valid uses GEOS to check validity and Kaggle use JTS (they differ, causing lots of trouble).\r\n\r\n- when submission fails with TopologyException, it gives only the first error. We have no way of getting the list of all invalid polygons. And the competition didn't provide a way to check validity off-line thus making troubleshooting difficult.\r\n\r\n- size of submission file: shapely.wkt.dumps(mp, rounding_precision=10), or 8. Use local validation to check the reduced precision is not affecting your score significantly.  And beware truncating the numbers can generate bad polygons as well. Before the final stages of the competition, I think speed of iteration is better than precision. Precision==8 reduces csv size in half.\r\n\r\n- using polygon.buffer(0) helps sometimes\r\n\r\n- polygon.simplify(e, preserveTopology=False) sometimes generates invalid polygons\r\n\r\n- sometimes other polygon operations generate invalid polygons.\r\n\r\n- the algorithm you use for generating polygons can increase/decrease the probability of invalid geometries. I'm several without being able to submit a valid file for now.\r\n\r\nAnd as far as I know there's no easy way to guarantee generated polygons are valid. We need to post-process the polygons checking for validity. \r\n\r\nFor that purpose I wrote a tool to help me, using the same underlying validity check as Kaggle. At least now I now which ones are invalid.\r\n\r\nhttps://github.com/cxz/tpex\r\n(sorry the library is in Java)",
      "votes": null
    },
    {
      "id": "154305",
      "postDate": "01/05/2017 18:10:48",
      "content": "<p>cool!!  I finally made a successful submission after many tries. It literally took checking every single polygon  in the multi polygons  and then trying buffer(0) on the invalid ones, checking isValid again then throwing away if it failed the second check. This still fails so the second step is to reload csv use shapely simplify then check isValid again then use buffer(0) 3 or 4 times alternating btw isValid checks. This is JUST to generate a submission. It is ridiculous, I think this submission format was a bad idea, it is plagued with problems.</p>",
      "rawMarkdown": "cool!!  I finally made a successful submission after many tries. It literally took checking every single polygon  in the multi polygons  and then trying buffer(0) on the invalid ones, checking isValid again then throwing away if it failed the second check. This still fails so the second step is to reload csv use shapely simplify then check isValid again then use buffer(0) 3 or 4 times alternating btw isValid checks. This is JUST to generate a submission. It is ridiculous, I think this submission format was a bad idea, it is plagued with problems.",
      "votes": null
    },
    {
      "id": "154370",
      "postDate": "01/05/2017 23:28:25",
      "content": "<p>This is a great tool, thanks @amaia. To be clear, Kaggle used NetTopologySuite (NTS) which is a C# port for JTS. They have mostly the same behavior, so your tool will work great to mimic the evaluation code.</p>\n\n<p>@DavidGbodiOdaibo, I'm sorry it's been a struggle to have a valid submission. Before the beginning of the competition we went through many iterations with the vendor that did the hand labeling to generate the solution without geometry errors, too. Unfortunately these polygons are the standard of the Geospatial world, so it's a strict requirement to generate these as algorithm outputs. </p>",
      "rawMarkdown": "This is a great tool, thanks @amaia. To be clear, Kaggle used NetTopologySuite (NTS) which is a C# port for JTS. They have mostly the same behavior, so your tool will work great to mimic the evaluation code.\n\n@DavidGbodiOdaibo, I'm sorry it's been a struggle to have a valid submission. Before the beginning of the competition we went through many iterations with the vendor that did the hand labeling to generate the solution without geometry errors, too. Unfortunately these polygons are the standard of the Geospatial world, so it's a strict requirement to generate these as algorithm outputs.",
      "votes": null
    },
    {
      "id": "155147",
      "postDate": "01/09/2017 21:45:00",
      "content": "<p>so what is the trick with these 500MB file submissions... i get timeouts for my sill 27MB sub, feeling sad now :(</p>",
      "rawMarkdown": "so what is the trick with these 500MB file submissions... i get timeouts for my sill 27MB sub, feeling sad now :(",
      "votes": null
    },
    {
      "id": "155149",
      "postDate": "01/09/2017 21:56:17",
      "content": "<p>[quote=raddar;155147]</p>\n\n<p>so what is the trick with these 500MB file submissions... i get timeouts for my sill 27MB sub, feeling sad now :(</p>\n\n<p>[/quote]</p>\n\n<p>@raddar,\nThe timeout counts only after the upload finishes, so what matters more is the number/complexity of polygons not file size.  My last submission was 115MB uncompressed and 20MB compressed. </p>",
      "rawMarkdown": "[quote=raddar;155147]\r\n\r\nso what is the trick with these 500MB file submissions... i get timeouts for my sill 27MB sub, feeling sad now :(\r\n\r\n[/quote]\r\n\r\n@raddar,\r\nThe timeout counts only after the upload finishes, so what matters more is the number/complexity of polygons not file size.  My last submission was 115MB uncompressed and 20MB compressed.",
      "votes": null
    },
    {
      "id": "155151",
      "postDate": "01/09/2017 22:02:51",
      "content": "<p>108MB uncompressed :| damn ... will try to remove some of the images from scoring, hope to get a score at least</p>",
      "rawMarkdown": "108MB uncompressed :| damn ... will try to remove some of the images from scoring, hope to get a score at least",
      "votes": null
    },
    {
      "id": "155521",
      "postDate": "01/11/2017 17:32:12",
      "content": "<p>Hi, <br>\nUntil now I have been able to submit two files that uncompressed used 572MB and 1.1GB</p>\n\n<p>Since then I only get timeout or when I try to simplify the polygons I get error messages</p>",
      "rawMarkdown": "Hi,  \r\nUntil now I have been able to submit two files that uncompressed used 572MB and 1.1GB\r\n\r\nSince then I only get timeout or when I try to simplify the polygons I get error messages",
      "votes": null
    },
    {
      "id": "155526",
      "postDate": "01/11/2017 17:51:21",
      "content": "<p>my culprit was having GEOMETRYCOLLECTION in my sub file. When i switched to MULTIPOLYGON it kinda worked out.</p>",
      "rawMarkdown": "my culprit was having GEOMETRYCOLLECTION in my sub file. When i switched to MULTIPOLYGON it kinda worked out.",
      "votes": null
    },
    {
      "id": "155534",
      "postDate": "01/11/2017 18:48:17",
      "content": "<p>Hi amaia, <br>\nI have been able to download and compile your tool. For doing that I had to download the maven version from apache website because the ubuntu package didn't work. <br>\n<a href=\"https://maven.apache.org/download.cgi?Preferred=http%3A%2F%2Fwww-eu.apache.org%2Fdist%2F\">https://maven.apache.org/download.cgi?Preferred=http%3A%2F%2Fwww-eu.apache.org%2Fdist%2F</a> <br>\nI think it's very useful because it has find many invalid polygons in my submission.</p>\n\n<p>How do you proceed once you know the invalid polygons? <br>\nIs there a method for fixing them?</p>\n\n<p>I'm thinking that a possibility could be to create a submission without any simplifications. Then we could try to simplify the submission and in those cases were the polygons are invalid we can retrieve the version without simplifications.</p>\n\n<p>Thanks\nironbar</p>",
      "rawMarkdown": "Hi amaia,   \r\nI have been able to download and compile your tool. For doing that I had to download the maven version from apache website because the ubuntu package didn't work.  \r\nhttps://maven.apache.org/download.cgi?Preferred=http%3A%2F%2Fwww-eu.apache.org%2Fdist%2F   \r\nI think it's very useful because it has find many invalid polygons in my submission.\r\n\r\nHow do you proceed once you know the invalid polygons?  \r\nIs there a method for fixing them?\r\n\r\nI'm thinking that a possibility could be to create a submission without any simplifications. Then we could try to simplify the submission and in those cases were the polygons are invalid we can retrieve the version without simplifications.\r\n\r\nThanks\r\nironbar",
      "votes": null
    },
    {
      "id": "155567",
      "postDate": "01/11/2017 22:23:25",
      "content": "<p>I think I have a script in python that checks and correct invalid polygons, if tomorrow I have time I will check it and upload it.</p>",
      "rawMarkdown": "I think I have a script in python that checks and correct invalid polygons, if tomorrow I have time I will check it and upload it.",
      "votes": null
    },
    {
      "id": "159993",
      "postDate": "02/05/2017 11:55:28",
      "content": "<p>Hi ironbar,</p>\n\n<p>I use ubuntu 14.04 and I don't download from apache website.<br>\nI use <code>sudo apt-get install maven</code> alternatively.</p>",
      "rawMarkdown": "Hi ironbar,\n\nI use ubuntu 14.04 and I don't download from apache website.<br>\nI use `sudo apt-get install maven` alternatively.",
      "votes": null
    },
    {
      "id": "160210",
      "postDate": "02/06/2017 16:05:44",
      "content": "<p>Is it already tomorrow? :)</p>",
      "rawMarkdown": "Is it already tomorrow? :)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 154305,
      "author_name": "godaibo",
      "author_url": "",
      "post_date": "01/05/2017 18:10:48",
      "content": "<p>cool!!  I finally made a successful submission after many tries. It literally took checking every single polygon  in the multi polygons  and then trying buffer(0) on the invalid ones, checking isValid again then throwing away if it failed the second check. This still fails so the second step is to reload csv use shapely simplify then check isValid again then use buffer(0) 3 or 4 times alternating btw isValid checks. This is JUST to generate a submission. It is ridiculous, I think this submission format was a bad idea, it is plagued with problems.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 154370,
      "author_name": "wendykan",
      "author_url": "",
      "post_date": "01/05/2017 23:28:25",
      "content": "<p>This is a great tool, thanks @amaia. To be clear, Kaggle used NetTopologySuite (NTS) which is a C# port for JTS. They have mostly the same behavior, so your tool will work great to mimic the evaluation code.</p>\n\n<p>@DavidGbodiOdaibo, I'm sorry it's been a struggle to have a valid submission. Before the beginning of the competition we went through many iterations with the vendor that did the hand labeling to generate the solution without geometry errors, too. Unfortunately these polygons are the standard of the Geospatial world, so it's a strict requirement to generate these as algorithm outputs. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 155147,
      "author_name": "raddar",
      "author_url": "",
      "post_date": "01/09/2017 21:45:00",
      "content": "<p>so what is the trick with these 500MB file submissions... i get timeouts for my sill 27MB sub, feeling sad now :(</p>",
      "votes": null,
      "replies": [
        {
          "id": 155149,
          "author_name": "aamaia",
          "author_url": "",
          "post_date": "01/09/2017 21:56:17",
          "content": "<p>[quote=raddar;155147]</p>\n\n<p>so what is the trick with these 500MB file submissions... i get timeouts for my sill 27MB sub, feeling sad now :(</p>\n\n<p>[/quote]</p>\n\n<p>@raddar,\nThe timeout counts only after the upload finishes, so what matters more is the number/complexity of polygons not file size.  My last submission was 115MB uncompressed and 20MB compressed. </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 155151,
      "author_name": "raddar",
      "author_url": "",
      "post_date": "01/09/2017 22:02:51",
      "content": "<p>108MB uncompressed :| damn ... will try to remove some of the images from scoring, hope to get a score at least</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 155521,
      "author_name": "ironbar",
      "author_url": "",
      "post_date": "01/11/2017 17:32:12",
      "content": "<p>Hi, <br>\nUntil now I have been able to submit two files that uncompressed used 572MB and 1.1GB</p>\n\n<p>Since then I only get timeout or when I try to simplify the polygons I get error messages</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 155526,
      "author_name": "raddar",
      "author_url": "",
      "post_date": "01/11/2017 17:51:21",
      "content": "<p>my culprit was having GEOMETRYCOLLECTION in my sub file. When i switched to MULTIPOLYGON it kinda worked out.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 155534,
      "author_name": "ironbar",
      "author_url": "",
      "post_date": "01/11/2017 18:48:17",
      "content": "<p>Hi amaia, <br>\nI have been able to download and compile your tool. For doing that I had to download the maven version from apache website because the ubuntu package didn't work. <br>\n<a href=\"https://maven.apache.org/download.cgi?Preferred=http%3A%2F%2Fwww-eu.apache.org%2Fdist%2F\">https://maven.apache.org/download.cgi?Preferred=http%3A%2F%2Fwww-eu.apache.org%2Fdist%2F</a> <br>\nI think it's very useful because it has find many invalid polygons in my submission.</p>\n\n<p>How do you proceed once you know the invalid polygons? <br>\nIs there a method for fixing them?</p>\n\n<p>I'm thinking that a possibility could be to create a submission without any simplifications. Then we could try to simplify the submission and in those cases were the polygons are invalid we can retrieve the version without simplifications.</p>\n\n<p>Thanks\nironbar</p>",
      "votes": null,
      "replies": [
        {
          "id": 159993,
          "author_name": "toshik",
          "author_url": "",
          "post_date": "02/05/2017 11:55:28",
          "content": "<p>Hi ironbar,</p>\n\n<p>I use ubuntu 14.04 and I don't download from apache website.<br>\nI use <code>sudo apt-get install maven</code> alternatively.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 155567,
      "author_name": "ironbar",
      "author_url": "",
      "post_date": "01/11/2017 22:23:25",
      "content": "<p>I think I have a script in python that checks and correct invalid polygons, if tomorrow I have time I will check it and upload it.</p>",
      "votes": null,
      "replies": [
        {
          "id": 160210,
          "author_name": "ceperaang",
          "author_url": "",
          "post_date": "02/06/2017 16:05:44",
          "content": "<p>Is it already tomorrow? :)</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "154302": "- Smooth your polygons first as people said before. Out of curiosity, what are typical CSV sizes people are submitting? 300M, 500M?\r\n\r\n- shapely.is_valid uses GEOS to check validity and Kaggle use JTS (they differ, causing lots of trouble).\r\n\r\n- when submission fails with TopologyException, it gives only the first error. We have no way of getting the list of all invalid polygons. And the competition didn't provide a way to check validity off-line thus making troubleshooting difficult.\r\n\r\n- size of submission file: shapely.wkt.dumps(mp, rounding_precision=10), or 8. Use local validation to check the reduced precision is not affecting your score significantly.  And beware truncating the numbers can generate bad polygons as well. Before the final stages of the competition, I think speed of iteration is better than precision. Precision==8 reduces csv size in half.\r\n\r\n- using polygon.buffer(0) helps sometimes\r\n\r\n- polygon.simplify(e, preserveTopology=False) sometimes generates invalid polygons\r\n\r\n- sometimes other polygon operations generate invalid polygons.\r\n\r\n- the algorithm you use for generating polygons can increase/decrease the probability of invalid geometries. I'm several without being able to submit a valid file for now.\r\n\r\nAnd as far as I know there's no easy way to guarantee generated polygons are valid. We need to post-process the polygons checking for validity. \r\n\r\nFor that purpose I wrote a tool to help me, using the same underlying validity check as Kaggle. At least now I now which ones are invalid.\r\n\r\nhttps://github.com/cxz/tpex\r\n(sorry the library is in Java)",
    "154305": "cool!!  I finally made a successful submission after many tries. It literally took checking every single polygon  in the multi polygons  and then trying buffer(0) on the invalid ones, checking isValid again then throwing away if it failed the second check. This still fails so the second step is to reload csv use shapely simplify then check isValid again then use buffer(0) 3 or 4 times alternating btw isValid checks. This is JUST to generate a submission. It is ridiculous, I think this submission format was a bad idea, it is plagued with problems.",
    "154370": "This is a great tool, thanks @amaia. To be clear, Kaggle used NetTopologySuite (NTS) which is a C# port for JTS. They have mostly the same behavior, so your tool will work great to mimic the evaluation code.\n\n@DavidGbodiOdaibo, I'm sorry it's been a struggle to have a valid submission. Before the beginning of the competition we went through many iterations with the vendor that did the hand labeling to generate the solution without geometry errors, too. Unfortunately these polygons are the standard of the Geospatial world, so it's a strict requirement to generate these as algorithm outputs.",
    "155147": "so what is the trick with these 500MB file submissions... i get timeouts for my sill 27MB sub, feeling sad now :(",
    "155149": "[quote=raddar;155147]\r\n\r\nso what is the trick with these 500MB file submissions... i get timeouts for my sill 27MB sub, feeling sad now :(\r\n\r\n[/quote]\r\n\r\n@raddar,\r\nThe timeout counts only after the upload finishes, so what matters more is the number/complexity of polygons not file size.  My last submission was 115MB uncompressed and 20MB compressed.",
    "155151": "108MB uncompressed :| damn ... will try to remove some of the images from scoring, hope to get a score at least",
    "155521": "Hi,  \r\nUntil now I have been able to submit two files that uncompressed used 572MB and 1.1GB\r\n\r\nSince then I only get timeout or when I try to simplify the polygons I get error messages",
    "155526": "my culprit was having GEOMETRYCOLLECTION in my sub file. When i switched to MULTIPOLYGON it kinda worked out.",
    "155534": "Hi amaia,   \r\nI have been able to download and compile your tool. For doing that I had to download the maven version from apache website because the ubuntu package didn't work.  \r\nhttps://maven.apache.org/download.cgi?Preferred=http%3A%2F%2Fwww-eu.apache.org%2Fdist%2F   \r\nI think it's very useful because it has find many invalid polygons in my submission.\r\n\r\nHow do you proceed once you know the invalid polygons?  \r\nIs there a method for fixing them?\r\n\r\nI'm thinking that a possibility could be to create a submission without any simplifications. Then we could try to simplify the submission and in those cases were the polygons are invalid we can retrieve the version without simplifications.\r\n\r\nThanks\r\nironbar",
    "155567": "I think I have a script in python that checks and correct invalid polygons, if tomorrow I have time I will check it and upload it.",
    "159993": "Hi ironbar,\n\nI use ubuntu 14.04 and I don't download from apache website.<br>\nI use `sudo apt-get install maven` alternatively.",
    "160210": "Is it already tomorrow? :)"
  },
  "source": "meta"
}