{
  "id": 223612,
  "title": "Extracting data - taking forever",
  "url": "/competitions/bms-molecular-translation/discussion/223612",
  "author_name": "",
  "post_date": "2021-03-04T16:55:06.680553400Z",
  "votes": 6,
  "comment_count": 16,
  "views": 0,
  "content": "<p>I'm wondering.. is it normal, that in 5 minutes, only 6k (six thousand) images were extracted from the .zip file?</p>\n<p>The zip file is on a SATA SSD and I tried to extract it first onto an NVMe SSD but was so slow. Now I'm trying to extract it onto the same SATA SSD but it is still that slow.</p>\n<p>How much time did it cost you to extract the data?</p>",
  "messages": [
    {
      "id": "1226571",
      "postDate": "03/04/2021 16:55:06",
      "content": "<p>I'm wondering.. is it normal, that in 5 minutes, only 6k (six thousand) images were extracted from the .zip file?</p>\n<p>The zip file is on a SATA SSD and I tried to extract it first onto an NVMe SSD but was so slow. Now I'm trying to extract it onto the same SATA SSD but it is still that slow.</p>\n<p>How much time did it cost you to extract the data?</p>",
      "rawMarkdown": "I'm wondering.. is it normal, that in 5 minutes, only 6k (six thousand) images were extracted from the .zip file?\n\nThe zip file is on a SATA SSD and I tried to extract it first onto an NVMe SSD but was so slow. Now I'm trying to extract it onto the same SATA SSD but it is still that slow.\n\nHow much time did it cost you to extract the data?",
      "votes": null
    },
    {
      "id": "1226608",
      "postDate": "03/04/2021 17:21:01",
      "content": "<p>I'm on Ubuntu. It seems that the archive manager is the weak link.<br>\nI simple <code>unzip bms-molecular-translation.zip</code> from terminal makes the difference.</p>",
      "rawMarkdown": "I'm on Ubuntu. It seems that the archive manager is the weak link.\nI simple `unzip bms-molecular-translation.zip` from terminal makes the difference.",
      "votes": null
    },
    {
      "id": "1226756",
      "postDate": "03/04/2021 20:16:42",
      "content": "<p><code>unzip -qq  bms-molecular-translation.zip</code></p>\n<pre><code>   -q     perform  operations  quietly  (-qq  = even quieter).  Ordinarily\n          unzip prints the names of the files it's extracting or  testing,\n          the extraction methods, any file or zipfile comments that may be\n          stored in the archive, and possibly a summary when finished with\n          each  archive.   The -q[q] options suppress the printing of some\n          or all of these messages.\n</code></pre>\n<p>this should speed up…<br>\n<a href=\"https://stackoverflow.com/questions/19279038/how-to-minimize-the-time-for-unzipping-zipping-the-files-in-unix\" target=\"_blank\">https://stackoverflow.com/questions/19279038/how-to-minimize-the-time-for-unzipping-zipping-the-files-in-unix</a></p>",
      "rawMarkdown": "`unzip -qq  bms-molecular-translation.zip `\n```\n   -q     perform  operations  quietly  (-qq  = even quieter).  Ordinarily\n          unzip prints the names of the files it's extracting or  testing,\n          the extraction methods, any file or zipfile comments that may be\n          stored in the archive, and possibly a summary when finished with\n          each  archive.   The -q[q] options suppress the printing of some\n          or all of these messages.\n```\n\nthis should speed up...\nhttps://stackoverflow.com/questions/19279038/how-to-minimize-the-time-for-unzipping-zipping-the-files-in-unix",
      "votes": null
    },
    {
      "id": "1227094",
      "postDate": "03/05/2021 07:36:08",
      "content": "<p>I try to use c++ based applications to unpack data in python. The packages vtk (for images, and plenty of other data-sources) and python datatable (for tabular data), both add considerable speed-up from what I've seen. And they're both already present in the Kaggle VM!    </p>",
      "rawMarkdown": "I try to use c++ based applications to unpack data in python. The packages vtk (for images, and plenty of other data-sources) and python datatable (for tabular data), both add considerable speed-up from what I've seen. And they're both already present in the Kaggle VM!",
      "votes": null
    },
    {
      "id": "1227555",
      "postDate": "03/05/2021 16:43:41",
      "content": "<p>Now, I am extracting the zip file with 7-Zip on a Windows 10 notebook. About 1319000 files have been extracted in one hour. Another one hour seems to be necessary. 😄</p>",
      "rawMarkdown": "Now, I am extracting the zip file with 7-Zip on a Windows 10 notebook. About 1319000 files have been extracted in one hour. Another one hour seems to be necessary. 😄",
      "votes": null
    },
    {
      "id": "1227598",
      "postDate": "03/05/2021 17:04:21",
      "content": "<p>On Ubuntu on an SSD with the command <code>unzip bms-molecular-translation.zip</code> it extracted for me in less than 5 minutes.<br>\nIt really is important what you are extracting with.</p>",
      "rawMarkdown": "On Ubuntu on an SSD with the command `unzip bms-molecular-translation.zip` it extracted for me in less than 5 minutes.\nIt really is important what you are extracting with.",
      "votes": null
    },
    {
      "id": "1228018",
      "postDate": "03/06/2021 03:13:58",
      "content": "<p>Thank you for your reply. I need more to learn. 😄</p>",
      "rawMarkdown": "Thank you for your reply. I need more to learn. 😄",
      "votes": null
    },
    {
      "id": "1228588",
      "postDate": "03/06/2021 15:24:03",
      "content": "<p>I also faced the similar issue where the Windows explorer freezes when I extract the zip file. I used 7-zip from <a href=\"https://www.7-zip.org/\" target=\"_blank\">https://www.7-zip.org/</a>. It worked:)</p>",
      "rawMarkdown": "I also faced the similar issue where the Windows explorer freezes when I extract the zip file. I used 7-zip from https://www.7-zip.org/. It worked:)",
      "votes": null
    },
    {
      "id": "1229419",
      "postDate": "03/07/2021 10:33:03",
      "content": "<p>Thank you very much for your kind message, Matsumura-san.😄</p>",
      "rawMarkdown": "Thank you very much for your kind message, Matsumura-san.😄",
      "votes": null
    },
    {
      "id": "1239604",
      "postDate": "03/15/2021 21:03:39",
      "content": "<p>How long did it take for the unzip to finish? I am running <code>unzip -qq -o ./input/*.zip -d ./input</code> and in <code>glances</code> I see writes per second of 3M. This is in bytes per second -- 3 million bytes per second. Does this make sense? It has been running for 1hr 20 minutes. CPU usage is not that high… just about 5%.</p>\n<p>Per <a href=\"https://www.peterbe.com/plog/unzip-benchmark-on-aws-ec2-c3.large-vs-c4.large\" target=\"_blank\">this post</a> I think I should be seeing about 100MB per second, or 104 million bytes per second.</p>",
      "rawMarkdown": "How long did it take for the unzip to finish? I am running `unzip -qq -o ./input/*.zip -d ./input` and in `glances` I see writes per second of 3M. This is in bytes per second -- 3 million bytes per second. Does this make sense? It has been running for 1hr 20 minutes. CPU usage is not that high... just about 5%.\n\nPer [this post](https://www.peterbe.com/plog/unzip-benchmark-on-aws-ec2-c3.large-vs-c4.large) I think I should be seeing about 100MB per second, or 104 million bytes per second.",
      "votes": null
    },
    {
      "id": "1239643",
      "postDate": "03/15/2021 21:56:36",
      "content": "<p>not long… I don't remember exact time… but maybe <code>1h</code> max </p>",
      "rawMarkdown": "not long... I don't remember exact time... but maybe `1h` max",
      "votes": null
    },
    {
      "id": "1240573",
      "postDate": "03/16/2021 14:02:29",
      "content": "<p>Something must be wrong with my instance. I am using Paperspace Gradient. I've been running the unzip for 22hrs… and still going. I've asked Paperspace support so we will see. Does anyone use Paperspace Gradient?</p>",
      "rawMarkdown": "Something must be wrong with my instance. I am using Paperspace Gradient. I've been running the unzip for 22hrs... and still going. I've asked Paperspace support so we will see. Does anyone use Paperspace Gradient?",
      "votes": null
    },
    {
      "id": "1240580",
      "postDate": "03/16/2021 14:08:53",
      "content": "<p>Do you extract it to an HDD?</p>",
      "rawMarkdown": "Do you extract it to an HDD?",
      "votes": null
    },
    {
      "id": "1240628",
      "postDate": "03/16/2021 14:41:00",
      "content": "<p>Good question. I am trying to figure that out. The instance is running Ubuntu 16.04. When I try to inspect with <code>df</code> I get</p>\n<p><img src=\"https://i.imgur.com/gyn5fLy.jpeg\" alt=\"\"></p>\n<p>It's odd to see this URL as the device. This makes me think reads and writes are going over tcp (?!). Anyhow, I've asked Paperspace support.</p>\n<p>It's so frustrating to find a good cloud VM host. I'd happily use GCP but I can't seem to get a GPU quota increase.</p>",
      "rawMarkdown": "Good question. I am trying to figure that out. The instance is running Ubuntu 16.04. When I try to inspect with `df` I get\n\n![](https://i.imgur.com/gyn5fLy.jpeg)\n\nIt's odd to see this URL as the device. This makes me think reads and writes are going over tcp (?!). Anyhow, I've asked Paperspace support.\n\nIt's so frustrating to find a good cloud VM host. I'd happily use GCP but I can't seem to get a GPU quota increase.",
      "votes": null
    },
    {
      "id": "1240677",
      "postDate": "03/16/2021 15:12:52",
      "content": "<p>use command line instead of gui</p>",
      "rawMarkdown": "use command line instead of gui",
      "votes": null
    },
    {
      "id": "1242338",
      "postDate": "03/17/2021 14:54:55",
      "content": "<p>on an AWS p2.xlarge ubuntu deep learning AMI I was able to unzip with <code>unzip -qq</code> in 6 minutes. The zipped size is 7GB and the unzipped size is 17GB.</p>",
      "rawMarkdown": "on an AWS p2.xlarge ubuntu deep learning AMI I was able to unzip with `unzip -qq` in 6 minutes. The zipped size is 7GB and the unzipped size is 17GB.",
      "votes": null
    },
    {
      "id": "1242365",
      "postDate": "03/17/2021 15:11:21",
      "content": "<p>Does that mean you can start the fun part now? :)</p>",
      "rawMarkdown": "Does that mean you can start the fun part now? :)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1226608,
      "author_name": "nofreewill",
      "author_url": "",
      "post_date": "03/04/2021 17:21:01",
      "content": "<p>I'm on Ubuntu. It seems that the archive manager is the weak link.<br>\nI simple <code>unzip bms-molecular-translation.zip</code> from terminal makes the difference.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1226756,
      "author_name": "drhabib",
      "author_url": "",
      "post_date": "03/04/2021 20:16:42",
      "content": "<p><code>unzip -qq  bms-molecular-translation.zip</code></p>\n<pre><code>   -q     perform  operations  quietly  (-qq  = even quieter).  Ordinarily\n          unzip prints the names of the files it's extracting or  testing,\n          the extraction methods, any file or zipfile comments that may be\n          stored in the archive, and possibly a summary when finished with\n          each  archive.   The -q[q] options suppress the printing of some\n          or all of these messages.\n</code></pre>\n<p>this should speed up…<br>\n<a href=\"https://stackoverflow.com/questions/19279038/how-to-minimize-the-time-for-unzipping-zipping-the-files-in-unix\" target=\"_blank\">https://stackoverflow.com/questions/19279038/how-to-minimize-the-time-for-unzipping-zipping-the-files-in-unix</a></p>",
      "votes": null,
      "replies": [
        {
          "id": 1239604,
          "author_name": "marketneutral",
          "author_url": "",
          "post_date": "03/15/2021 21:03:39",
          "content": "<p>How long did it take for the unzip to finish? I am running <code>unzip -qq -o ./input/*.zip -d ./input</code> and in <code>glances</code> I see writes per second of 3M. This is in bytes per second -- 3 million bytes per second. Does this make sense? It has been running for 1hr 20 minutes. CPU usage is not that high… just about 5%.</p>\n<p>Per <a href=\"https://www.peterbe.com/plog/unzip-benchmark-on-aws-ec2-c3.large-vs-c4.large\" target=\"_blank\">this post</a> I think I should be seeing about 100MB per second, or 104 million bytes per second.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1239643,
          "author_name": "drhabib",
          "author_url": "",
          "post_date": "03/15/2021 21:56:36",
          "content": "<p>not long… I don't remember exact time… but maybe <code>1h</code> max </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1240573,
          "author_name": "marketneutral",
          "author_url": "",
          "post_date": "03/16/2021 14:02:29",
          "content": "<p>Something must be wrong with my instance. I am using Paperspace Gradient. I've been running the unzip for 22hrs… and still going. I've asked Paperspace support so we will see. Does anyone use Paperspace Gradient?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1240580,
          "author_name": "nofreewill",
          "author_url": "",
          "post_date": "03/16/2021 14:08:53",
          "content": "<p>Do you extract it to an HDD?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1240628,
          "author_name": "marketneutral",
          "author_url": "",
          "post_date": "03/16/2021 14:41:00",
          "content": "<p>Good question. I am trying to figure that out. The instance is running Ubuntu 16.04. When I try to inspect with <code>df</code> I get</p>\n<p><img src=\"https://i.imgur.com/gyn5fLy.jpeg\" alt=\"\"></p>\n<p>It's odd to see this URL as the device. This makes me think reads and writes are going over tcp (?!). Anyhow, I've asked Paperspace support.</p>\n<p>It's so frustrating to find a good cloud VM host. I'd happily use GCP but I can't seem to get a GPU quota increase.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1242338,
          "author_name": "marketneutral",
          "author_url": "",
          "post_date": "03/17/2021 14:54:55",
          "content": "<p>on an AWS p2.xlarge ubuntu deep learning AMI I was able to unzip with <code>unzip -qq</code> in 6 minutes. The zipped size is 7GB and the unzipped size is 17GB.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1242365,
          "author_name": "nofreewill",
          "author_url": "",
          "post_date": "03/17/2021 15:11:21",
          "content": "<p>Does that mean you can start the fun part now? :)</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1227094,
      "author_name": "eladwar",
      "author_url": "",
      "post_date": "03/05/2021 07:36:08",
      "content": "<p>I try to use c++ based applications to unpack data in python. The packages vtk (for images, and plenty of other data-sources) and python datatable (for tabular data), both add considerable speed-up from what I've seen. And they're both already present in the Kaggle VM!    </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1227555,
      "author_name": "hiroshisakiyama",
      "author_url": "",
      "post_date": "03/05/2021 16:43:41",
      "content": "<p>Now, I am extracting the zip file with 7-Zip on a Windows 10 notebook. About 1319000 files have been extracted in one hour. Another one hour seems to be necessary. 😄</p>",
      "votes": null,
      "replies": [
        {
          "id": 1227598,
          "author_name": "nofreewill",
          "author_url": "",
          "post_date": "03/05/2021 17:04:21",
          "content": "<p>On Ubuntu on an SSD with the command <code>unzip bms-molecular-translation.zip</code> it extracted for me in less than 5 minutes.<br>\nIt really is important what you are extracting with.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1228018,
          "author_name": "hiroshisakiyama",
          "author_url": "",
          "post_date": "03/06/2021 03:13:58",
          "content": "<p>Thank you for your reply. I need more to learn. 😄</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1228588,
          "author_name": "matsumuranaoki",
          "author_url": "",
          "post_date": "03/06/2021 15:24:03",
          "content": "<p>I also faced the similar issue where the Windows explorer freezes when I extract the zip file. I used 7-zip from <a href=\"https://www.7-zip.org/\" target=\"_blank\">https://www.7-zip.org/</a>. It worked:)</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1229419,
          "author_name": "hiroshisakiyama",
          "author_url": "",
          "post_date": "03/07/2021 10:33:03",
          "content": "<p>Thank you very much for your kind message, Matsumura-san.😄</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1240677,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "03/16/2021 15:12:52",
      "content": "<p>use command line instead of gui</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1226571": "I'm wondering.. is it normal, that in 5 minutes, only 6k (six thousand) images were extracted from the .zip file?\n\nThe zip file is on a SATA SSD and I tried to extract it first onto an NVMe SSD but was so slow. Now I'm trying to extract it onto the same SATA SSD but it is still that slow.\n\nHow much time did it cost you to extract the data?",
    "1226608": "I'm on Ubuntu. It seems that the archive manager is the weak link.\nI simple `unzip bms-molecular-translation.zip` from terminal makes the difference.",
    "1226756": "`unzip -qq  bms-molecular-translation.zip `\n```\n   -q     perform  operations  quietly  (-qq  = even quieter).  Ordinarily\n          unzip prints the names of the files it's extracting or  testing,\n          the extraction methods, any file or zipfile comments that may be\n          stored in the archive, and possibly a summary when finished with\n          each  archive.   The -q[q] options suppress the printing of some\n          or all of these messages.\n```\n\nthis should speed up...\nhttps://stackoverflow.com/questions/19279038/how-to-minimize-the-time-for-unzipping-zipping-the-files-in-unix",
    "1227094": "I try to use c++ based applications to unpack data in python. The packages vtk (for images, and plenty of other data-sources) and python datatable (for tabular data), both add considerable speed-up from what I've seen. And they're both already present in the Kaggle VM!",
    "1227555": "Now, I am extracting the zip file with 7-Zip on a Windows 10 notebook. About 1319000 files have been extracted in one hour. Another one hour seems to be necessary. 😄",
    "1227598": "On Ubuntu on an SSD with the command `unzip bms-molecular-translation.zip` it extracted for me in less than 5 minutes.\nIt really is important what you are extracting with.",
    "1228018": "Thank you for your reply. I need more to learn. 😄",
    "1228588": "I also faced the similar issue where the Windows explorer freezes when I extract the zip file. I used 7-zip from https://www.7-zip.org/. It worked:)",
    "1229419": "Thank you very much for your kind message, Matsumura-san.😄",
    "1239604": "How long did it take for the unzip to finish? I am running `unzip -qq -o ./input/*.zip -d ./input` and in `glances` I see writes per second of 3M. This is in bytes per second -- 3 million bytes per second. Does this make sense? It has been running for 1hr 20 minutes. CPU usage is not that high... just about 5%.\n\nPer [this post](https://www.peterbe.com/plog/unzip-benchmark-on-aws-ec2-c3.large-vs-c4.large) I think I should be seeing about 100MB per second, or 104 million bytes per second.",
    "1239643": "not long... I don't remember exact time... but maybe `1h` max",
    "1240573": "Something must be wrong with my instance. I am using Paperspace Gradient. I've been running the unzip for 22hrs... and still going. I've asked Paperspace support so we will see. Does anyone use Paperspace Gradient?",
    "1240580": "Do you extract it to an HDD?",
    "1240628": "Good question. I am trying to figure that out. The instance is running Ubuntu 16.04. When I try to inspect with `df` I get\n\n![](https://i.imgur.com/gyn5fLy.jpeg)\n\nIt's odd to see this URL as the device. This makes me think reads and writes are going over tcp (?!). Anyhow, I've asked Paperspace support.\n\nIt's so frustrating to find a good cloud VM host. I'd happily use GCP but I can't seem to get a GPU quota increase.",
    "1240677": "use command line instead of gui",
    "1242338": "on an AWS p2.xlarge ubuntu deep learning AMI I was able to unzip with `unzip -qq` in 6 minutes. The zipped size is 7GB and the unzipped size is 17GB.",
    "1242365": "Does that mean you can start the fun part now? :)"
  },
  "source": "meta"
}