{
  "id": 37243,
  "title": "If your submission file is too large,use 7z format",
  "url": "/competitions/carvana-image-masking-challenge/discussion/37243",
  "author_name": "",
  "post_date": "2017-07-29T16:35:49.660433400Z",
  "votes": 4,
  "comment_count": 14,
  "views": 0,
  "content": "<p>Before zipping the file is 2.8GB. And only 455Mb using 7z format.</p>",
  "messages": [
    {
      "id": "208391",
      "postDate": "07/29/2017 16:35:49",
      "content": "<p>Before zipping the file is 2.8GB. And only 455Mb using 7z format.</p>",
      "rawMarkdown": "Before zipping the file is 2.8GB. And only 455Mb using 7z format.",
      "votes": null
    },
    {
      "id": "208413",
      "postDate": "07/29/2017 17:55:19",
      "content": "<p>7z is much better than .zip.</p>\n\n<p>In my case, csv file was 820MB. Zip file is 338MB, and 7z is 171MB.</p>",
      "rawMarkdown": "7z is much better than .zip.\n\nIn my case, csv file was 820MB. Zip file is 338MB, and 7z is 171MB.",
      "votes": null
    },
    {
      "id": "208495",
      "postDate": "07/30/2017 00:53:55",
      "content": "<p>I came across an error. \"ERROR: File appears to be empty (Line 1, Column 1)\" Do you know what cause this ?</p>",
      "rawMarkdown": "I came across an error. \"ERROR: File appears to be empty (Line 1, Column 1)\" Do you know what cause this ?",
      "votes": null
    },
    {
      "id": "208504",
      "postDate": "07/30/2017 02:21:39",
      "content": "<p>Not sure. How about checking the file is not encoded in 16-bit unicode and have correct header label?</p>",
      "rawMarkdown": "Not sure. How about checking the file is not encoded in 16-bit unicode and have correct header label?",
      "votes": null
    },
    {
      "id": "209135",
      "postDate": "08/01/2017 10:38:46",
      "content": "<p>@XubinNi\nDid you solve this error? </p>",
      "rawMarkdown": "XubinNi\nDid you solve this error?",
      "votes": null
    },
    {
      "id": "209149",
      "postDate": "08/01/2017 11:35:00",
      "content": "<p>Yes,it can not be opened on my computer so i guess it is also too large to be opened on kaggle's server. I change the code and reduce its size to 800MB, and succeed.</p>",
      "rawMarkdown": "Yes,it can not be opened on my computer so i guess it is also too large to be opened on kaggle's server. I change the code and reduce its size to 800MB, and succeed.",
      "votes": null
    },
    {
      "id": "209150",
      "postDate": "08/01/2017 11:36:16",
      "content": "<p>@KarimAmer</p>",
      "rawMarkdown": "KarimAmer",
      "votes": null
    },
    {
      "id": "209964",
      "postDate": "08/03/2017 21:59:17",
      "content": "<p>How did you reduced the size? I am getting Time outs due to submission size being too large.</p>",
      "rawMarkdown": "How did you reduced the size? I am getting Time outs due to submission size being too large.",
      "votes": null
    },
    {
      "id": "209975",
      "postDate": "08/03/2017 22:57:58",
      "content": "<p>I also had a problem with 3.8GB csv file... I ran following code to remove small dots in prediction mask and the size is reduced 800MB (200MB after 7z) - though I see small reduction in score with train set. </p>\n\n<pre><code>from skimage import measure, morphology\nlimg = measure.label(mask)\nprops = measure.regionprops(limg)\nprops = sorted(props, key=lambda p: -p.area)\n# Erase all except the biggest blob\nmask &amp;= (limg == props[0].label)\n</code></pre>",
      "rawMarkdown": "I also had a problem with 3.8GB csv file... I ran following code to remove small dots in prediction mask and the size is reduced 800MB (200MB after 7z) - though I see small reduction in score with train set. \n\n    from skimage import measure, morphology\n    limg = measure.label(mask)\n    props = measure.regionprops(limg)\n    props = sorted(props, key=lambda p: -p.area)\n    # Erase all except the biggest blob\n    mask &amp;= (limg == props[0].label)",
      "votes": null
    },
    {
      "id": "210074",
      "postDate": "08/04/2017 08:36:36",
      "content": "<p>My file is reasonably large (900/400 Mb csv/zip) yet my submission still timed out. Any ideas?</p>",
      "rawMarkdown": "My file is reasonably large (900/400 Mb csv/zip) yet my submission still timed out. Any ideas?",
      "votes": null
    },
    {
      "id": "210283",
      "postDate": "08/05/2017 00:50:36",
      "content": "<p>my submission file is too large.\nFile size before zipped was 600MB and after zip is 200MB.\nBut my submission couse 'timed out error'.</p>",
      "rawMarkdown": "my submission file is too large.\nFile size before zipped was 600MB and after zip is 200MB.\nBut my submission couse 'timed out error'.",
      "votes": null
    },
    {
      "id": "210284",
      "postDate": "08/05/2017 00:55:16",
      "content": "<p><img src=\"https://drive.google.com/file/d/0B3m8dLXnDxXSNVZFTVB6bWstZjg/view?usp=sharing\" alt=\"I executed your code\" title=\"\">\nI executed your code.\nIs this ok, right?</p>",
      "rawMarkdown": "![I executed your code][1]\nI executed your code.\nIs this ok, right?\n\n  [1]: https://drive.google.com/file/d/0B3m8dLXnDxXSNVZFTVB6bWstZjg/view?usp=sharing",
      "votes": null
    },
    {
      "id": "218038",
      "postDate": "09/01/2017 23:08:11",
      "content": "<p>7z is not working for me.  I have a 1.0 GB submission file  (227 Mb zipped) and it fails with \"Submission timed out. Please reduce size before resubmitting.\".  </p>\n\n<p><strong>EDIT</strong> My predictions were inverted, fixing that solved the issue for me.</p>",
      "rawMarkdown": "7z is not working for me.  I have a 1.0 GB submission file  (227 Mb zipped) and it fails with \"Submission timed out. Please reduce size before resubmitting.\".  \n\n__EDIT__ My predictions were inverted, fixing that solved the issue for me.",
      "votes": null
    },
    {
      "id": "221862",
      "postDate": "09/16/2017 14:53:02",
      "content": "<p>@XubinNi I'm also getting \"ERROR: File appears to be empty (Line 1, Column 1)\" \nWere you able to resolve the issue?\nMy csv was written with Pandas and I am able to read it too.\nI'm not sure what's causing the error </p>",
      "rawMarkdown": "XubinNi I'm also getting \"ERROR: File appears to be empty (Line 1, Column 1)\" \nWere you able to resolve the issue?\nMy csv was written with Pandas and I am able to read it too.\nI'm not sure what's causing the error",
      "votes": null
    },
    {
      "id": "222587",
      "postDate": "09/19/2017 10:08:16",
      "content": "<p>I have the same problem. Unzipped file size is ~2GB, zipped 800MB.  Submission fails with \"ERROR: File appears to be empty (Line 1, Column 1)\" </p>",
      "rawMarkdown": "I have the same problem. Unzipped file size is ~2GB, zipped 800MB.  Submission fails with \"ERROR: File appears to be empty (Line 1, Column 1)\"",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 208413,
      "author_name": "jandjenter",
      "author_url": "",
      "post_date": "07/29/2017 17:55:19",
      "content": "<p>7z is much better than .zip.</p>\n\n<p>In my case, csv file was 820MB. Zip file is 338MB, and 7z is 171MB.</p>",
      "votes": null,
      "replies": [
        {
          "id": 208495,
          "author_name": "dustni",
          "author_url": "",
          "post_date": "07/30/2017 00:53:55",
          "content": "<p>I came across an error. \"ERROR: File appears to be empty (Line 1, Column 1)\" Do you know what cause this ?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 208504,
          "author_name": "jandjenter",
          "author_url": "",
          "post_date": "07/30/2017 02:21:39",
          "content": "<p>Not sure. How about checking the file is not encoded in 16-bit unicode and have correct header label?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 209135,
          "author_name": "karimamer",
          "author_url": "",
          "post_date": "08/01/2017 10:38:46",
          "content": "<p>@XubinNi\nDid you solve this error? </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 209149,
          "author_name": "dustni",
          "author_url": "",
          "post_date": "08/01/2017 11:35:00",
          "content": "<p>Yes,it can not be opened on my computer so i guess it is also too large to be opened on kaggle's server. I change the code and reduce its size to 800MB, and succeed.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 209150,
          "author_name": "dustni",
          "author_url": "",
          "post_date": "08/01/2017 11:36:16",
          "content": "<p>@KarimAmer</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 209964,
          "author_name": "voltaire",
          "author_url": "",
          "post_date": "08/03/2017 21:59:17",
          "content": "<p>How did you reduced the size? I am getting Time outs due to submission size being too large.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 209975,
          "author_name": "jandjenter",
          "author_url": "",
          "post_date": "08/03/2017 22:57:58",
          "content": "<p>I also had a problem with 3.8GB csv file... I ran following code to remove small dots in prediction mask and the size is reduced 800MB (200MB after 7z) - though I see small reduction in score with train set. </p>\n\n<pre><code>from skimage import measure, morphology\nlimg = measure.label(mask)\nprops = measure.regionprops(limg)\nprops = sorted(props, key=lambda p: -p.area)\n# Erase all except the biggest blob\nmask &amp;= (limg == props[0].label)\n</code></pre>",
          "votes": null,
          "replies": []
        },
        {
          "id": 210284,
          "author_name": "hiratarosu",
          "author_url": "",
          "post_date": "08/05/2017 00:55:16",
          "content": "<p><img src=\"https://drive.google.com/file/d/0B3m8dLXnDxXSNVZFTVB6bWstZjg/view?usp=sharing\" alt=\"I executed your code\" title=\"\">\nI executed your code.\nIs this ok, right?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 221862,
          "author_name": "burgalon",
          "author_url": "",
          "post_date": "09/16/2017 14:53:02",
          "content": "<p>@XubinNi I'm also getting \"ERROR: File appears to be empty (Line 1, Column 1)\" \nWere you able to resolve the issue?\nMy csv was written with Pandas and I am able to read it too.\nI'm not sure what's causing the error </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 222587,
          "author_name": "musella",
          "author_url": "",
          "post_date": "09/19/2017 10:08:16",
          "content": "<p>I have the same problem. Unzipped file size is ~2GB, zipped 800MB.  Submission fails with \"ERROR: File appears to be empty (Line 1, Column 1)\" </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 210074,
      "author_name": "blazeka",
      "author_url": "",
      "post_date": "08/04/2017 08:36:36",
      "content": "<p>My file is reasonably large (900/400 Mb csv/zip) yet my submission still timed out. Any ideas?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 210283,
      "author_name": "hiratarosu",
      "author_url": "",
      "post_date": "08/05/2017 00:50:36",
      "content": "<p>my submission file is too large.\nFile size before zipped was 600MB and after zip is 200MB.\nBut my submission couse 'timed out error'.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 218038,
      "author_name": "",
      "author_url": "",
      "post_date": "09/01/2017 23:08:11",
      "content": "<p>7z is not working for me.  I have a 1.0 GB submission file  (227 Mb zipped) and it fails with \"Submission timed out. Please reduce size before resubmitting.\".  </p>\n\n<p><strong>EDIT</strong> My predictions were inverted, fixing that solved the issue for me.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "208391": "Before zipping the file is 2.8GB. And only 455Mb using 7z format.",
    "208413": "7z is much better than .zip.\n\nIn my case, csv file was 820MB. Zip file is 338MB, and 7z is 171MB.",
    "208495": "I came across an error. \"ERROR: File appears to be empty (Line 1, Column 1)\" Do you know what cause this ?",
    "208504": "Not sure. How about checking the file is not encoded in 16-bit unicode and have correct header label?",
    "209135": "XubinNi\nDid you solve this error?",
    "209149": "Yes,it can not be opened on my computer so i guess it is also too large to be opened on kaggle's server. I change the code and reduce its size to 800MB, and succeed.",
    "209150": "KarimAmer",
    "209964": "How did you reduced the size? I am getting Time outs due to submission size being too large.",
    "209975": "I also had a problem with 3.8GB csv file... I ran following code to remove small dots in prediction mask and the size is reduced 800MB (200MB after 7z) - though I see small reduction in score with train set. \n\n    from skimage import measure, morphology\n    limg = measure.label(mask)\n    props = measure.regionprops(limg)\n    props = sorted(props, key=lambda p: -p.area)\n    # Erase all except the biggest blob\n    mask &amp;= (limg == props[0].label)",
    "210074": "My file is reasonably large (900/400 Mb csv/zip) yet my submission still timed out. Any ideas?",
    "210283": "my submission file is too large.\nFile size before zipped was 600MB and after zip is 200MB.\nBut my submission couse 'timed out error'.",
    "210284": "![I executed your code][1]\nI executed your code.\nIs this ok, right?\n\n  [1]: https://drive.google.com/file/d/0B3m8dLXnDxXSNVZFTVB6bWstZjg/view?usp=sharing",
    "218038": "7z is not working for me.  I have a 1.0 GB submission file  (227 Mb zipped) and it fails with \"Submission timed out. Please reduce size before resubmitting.\".  \n\n__EDIT__ My predictions were inverted, fixing that solved the issue for me.",
    "221862": "XubinNi I'm also getting \"ERROR: File appears to be empty (Line 1, Column 1)\" \nWere you able to resolve the issue?\nMy csv was written with Pandas and I am able to read it too.\nI'm not sure what's causing the error",
    "222587": "I have the same problem. Unzipped file size is ~2GB, zipped 800MB.  Submission fails with \"ERROR: File appears to be empty (Line 1, Column 1)\""
  },
  "source": "meta"
}