{
  "id": 31587,
  "title": "A Suggestion About Torrent",
  "url": "/competitions/noaa-fisheries-steller-sea-lion-population-count/discussion/31587",
  "author_name": "",
  "post_date": "2017-04-13T18:59:42.054141800Z",
  "votes": null,
  "comment_count": 4,
  "views": 0,
  "content": "<p>I'd like to propose a suggestion about torrent usage. </p>\n\n<p>2 days ago, after starting torrent, I see one big .zip file at about 95.8GB. I left the computer open and somehow my Macbook crashed and that partially downloaded big .zip file ruined. Then I started the torrent again from %0.</p>\n\n<p>That happened twice. I know, it's my problem to have a slower connection or crashing computer, but I wonder if is it possible to upload images and folders as is to the torrent, than uploading one big gigantic .zip file. If there were were many small files in the torrent, some of the files I'd downloaded would stay after the crash and I'd continue to download the rest. Also, I'd start testing my code with partially downloaded files, other than the samples.</p>\n\n<p>And there is one more benefit of downloading the folders and files. I don't have enough HD space on my mac, so I download the .zip to my external drive. Opening a gigantic .zip file from external HD is a headache. It also needs another free 100GB space.</p>\n\n<p>So please do not upload one big montrous 95.8GB .zip file to torrent, if possible. </p>\n\n<p>(Sorry for my bad English :)</p>",
  "messages": [
    {
      "id": "175100",
      "postDate": "04/13/2017 18:59:42",
      "content": "<p>I'd like to propose a suggestion about torrent usage. </p>\n\n<p>2 days ago, after starting torrent, I see one big .zip file at about 95.8GB. I left the computer open and somehow my Macbook crashed and that partially downloaded big .zip file ruined. Then I started the torrent again from %0.</p>\n\n<p>That happened twice. I know, it's my problem to have a slower connection or crashing computer, but I wonder if is it possible to upload images and folders as is to the torrent, than uploading one big gigantic .zip file. If there were were many small files in the torrent, some of the files I'd downloaded would stay after the crash and I'd continue to download the rest. Also, I'd start testing my code with partially downloaded files, other than the samples.</p>\n\n<p>And there is one more benefit of downloading the folders and files. I don't have enough HD space on my mac, so I download the .zip to my external drive. Opening a gigantic .zip file from external HD is a headache. It also needs another free 100GB space.</p>\n\n<p>So please do not upload one big montrous 95.8GB .zip file to torrent, if possible. </p>\n\n<p>(Sorry for my bad English :)</p>",
      "rawMarkdown": "I'd like to propose a suggestion about torrent usage. \n\n2 days ago, after starting torrent, I see one big .zip file at about 95.8GB. I left the computer open and somehow my Macbook crashed and that partially downloaded big .zip file ruined. Then I started the torrent again from %0.\n\nThat happened twice. I know, it's my problem to have a slower connection or crashing computer, but I wonder if is it possible to upload images and folders as is to the torrent, than uploading one big gigantic .zip file. If there were were many small files in the torrent, some of the files I'd downloaded would stay after the crash and I'd continue to download the rest. Also, I'd start testing my code with partially downloaded files, other than the samples.\n\nAnd there is one more benefit of downloading the folders and files. I don't have enough HD space on my mac, so I download the .zip to my external drive. Opening a gigantic .zip file from external HD is a headache. It also needs another free 100GB space.\n\nSo please do not upload one big montrous 95.8GB .zip file to torrent, if possible. \n\n(Sorry for my bad English :)",
      "votes": null
    },
    {
      "id": "175851",
      "postDate": "04/17/2017 21:01:09",
      "content": "<p>i completely agree. serving the images in folder has mostly upsides:</p>\n\n<ul>\n<li>better to continue when something goes wrong</li>\n<li>split data among multiple connections/computers</li>\n<li>start working on a small sample before the rest finishes</li>\n<li>jpgs cant be compressed anyway</li>\n</ul>",
      "rawMarkdown": "i completely agree. serving the images in folder has mostly upsides:\n\n- better to continue when something goes wrong\n- split data among multiple connections/computers\n- start working on a small sample before the rest finishes\n- jpgs cant be compressed anyway",
      "votes": null
    },
    {
      "id": "178230",
      "postDate": "04/27/2017 03:27:48",
      "content": "<p>I just entered this contest and am having problems downloading KaggleNOAASeaLions.7z using Firefox\non an Ubuntu Linux system.  I have sufficient bandwidth, and space on my computer, but the download keeps failing partway through due apparently to interrupted connection.  Once it was well over half complete when it failed.  So far I can find no way to resume the download from where it left off.  It would help me (and others with the same problem) if the organizers would split the file up into more manageable pieces.  It could be done by folders as suggested by armamut, or just in separate .7z files (10 files of about 9.6 GB each would work for me).  This has been done in some previous contests with very large datasets.  It would be unfortunate if this contest ended up being more about how to download the images than how to analyze them.</p>",
      "rawMarkdown": "I just entered this contest and am having problems downloading KaggleNOAASeaLions.7z using Firefox\non an Ubuntu Linux system.  I have sufficient bandwidth, and space on my computer, but the download keeps failing partway through due apparently to interrupted connection.  Once it was well over half complete when it failed.  So far I can find no way to resume the download from where it left off.  It would help me (and others with the same problem) if the organizers would split the file up into more manageable pieces.  It could be done by folders as suggested by armamut, or just in separate .7z files (10 files of about 9.6 GB each would work for me).  This has been done in some previous contests with very large datasets.  It would be unfortunate if this contest ended up being more about how to download the images than how to analyze them.",
      "votes": null
    },
    {
      "id": "178353",
      "postDate": "04/27/2017 11:17:03",
      "content": "<p>After a few failed attempts, I finally succeeded in downloading KaggleNOAASeaLions.7z.</p>",
      "rawMarkdown": "After a few failed attempts, I finally succeeded in downloading KaggleNOAASeaLions.7z.",
      "votes": null
    },
    {
      "id": "186258",
      "postDate": "05/27/2017 09:18:38",
      "content": "<p>Same Issue here. I really spent a lot of time downloading the dataset, a partition by images would be a wise solution. Please consider that option next time.</p>",
      "rawMarkdown": "Same Issue here. I really spent a lot of time downloading the dataset, a partition by images would be a wise solution. Please consider that option next time.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 175851,
      "author_name": "gopietz",
      "author_url": "",
      "post_date": "04/17/2017 21:01:09",
      "content": "<p>i completely agree. serving the images in folder has mostly upsides:</p>\n\n<ul>\n<li>better to continue when something goes wrong</li>\n<li>split data among multiple connections/computers</li>\n<li>start working on a small sample before the rest finishes</li>\n<li>jpgs cant be compressed anyway</li>\n</ul>",
      "votes": null,
      "replies": []
    },
    {
      "id": 178230,
      "author_name": "dslate",
      "author_url": "",
      "post_date": "04/27/2017 03:27:48",
      "content": "<p>I just entered this contest and am having problems downloading KaggleNOAASeaLions.7z using Firefox\non an Ubuntu Linux system.  I have sufficient bandwidth, and space on my computer, but the download keeps failing partway through due apparently to interrupted connection.  Once it was well over half complete when it failed.  So far I can find no way to resume the download from where it left off.  It would help me (and others with the same problem) if the organizers would split the file up into more manageable pieces.  It could be done by folders as suggested by armamut, or just in separate .7z files (10 files of about 9.6 GB each would work for me).  This has been done in some previous contests with very large datasets.  It would be unfortunate if this contest ended up being more about how to download the images than how to analyze them.</p>",
      "votes": null,
      "replies": [
        {
          "id": 178353,
          "author_name": "dslate",
          "author_url": "",
          "post_date": "04/27/2017 11:17:03",
          "content": "<p>After a few failed attempts, I finally succeeded in downloading KaggleNOAASeaLions.7z.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 186258,
      "author_name": "alviur",
      "author_url": "",
      "post_date": "05/27/2017 09:18:38",
      "content": "<p>Same Issue here. I really spent a lot of time downloading the dataset, a partition by images would be a wise solution. Please consider that option next time.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "175100": "I'd like to propose a suggestion about torrent usage. \n\n2 days ago, after starting torrent, I see one big .zip file at about 95.8GB. I left the computer open and somehow my Macbook crashed and that partially downloaded big .zip file ruined. Then I started the torrent again from %0.\n\nThat happened twice. I know, it's my problem to have a slower connection or crashing computer, but I wonder if is it possible to upload images and folders as is to the torrent, than uploading one big gigantic .zip file. If there were were many small files in the torrent, some of the files I'd downloaded would stay after the crash and I'd continue to download the rest. Also, I'd start testing my code with partially downloaded files, other than the samples.\n\nAnd there is one more benefit of downloading the folders and files. I don't have enough HD space on my mac, so I download the .zip to my external drive. Opening a gigantic .zip file from external HD is a headache. It also needs another free 100GB space.\n\nSo please do not upload one big montrous 95.8GB .zip file to torrent, if possible. \n\n(Sorry for my bad English :)",
    "175851": "i completely agree. serving the images in folder has mostly upsides:\n\n- better to continue when something goes wrong\n- split data among multiple connections/computers\n- start working on a small sample before the rest finishes\n- jpgs cant be compressed anyway",
    "178230": "I just entered this contest and am having problems downloading KaggleNOAASeaLions.7z using Firefox\non an Ubuntu Linux system.  I have sufficient bandwidth, and space on my computer, but the download keeps failing partway through due apparently to interrupted connection.  Once it was well over half complete when it failed.  So far I can find no way to resume the download from where it left off.  It would help me (and others with the same problem) if the organizers would split the file up into more manageable pieces.  It could be done by folders as suggested by armamut, or just in separate .7z files (10 files of about 9.6 GB each would work for me).  This has been done in some previous contests with very large datasets.  It would be unfortunate if this contest ended up being more about how to download the images than how to analyze them.",
    "178353": "After a few failed attempts, I finally succeeded in downloading KaggleNOAASeaLions.7z.",
    "186258": "Same Issue here. I really spent a lot of time downloading the dataset, a partition by images would be a wise solution. Please consider that option next time."
  },
  "source": "meta"
}