{
  "id": 10622,
  "title": "3 weeks... still downloading data",
  "url": "/competitions/seizure-prediction/discussion/10622",
  "author_name": "",
  "post_date": "2014-10-17T09:41:42.703Z",
  "votes": null,
  "comment_count": 17,
  "views": 3635,
  "content": "<p>I started downloading data 3 weeks ago. Now, I am still downloading Patient_2.tar, the speed is around 20kb...&nbsp;</p>\n<p>Downloading fails many times this week.</p>\n<p>Maybe my network has problem or that file server has problem.</p>\n<p>Can anyone help?</p>\n<p>If there is a ftp server or whatever ,I would greatly appreciate it.</p>",
  "messages": [
    {
      "id": "56172",
      "postDate": "10/17/2014 09:41:42",
      "content": "<p>I started downloading data 3 weeks ago. Now, I am still downloading Patient_2.tar, the speed is around 20kb...&nbsp;</p>\n<p>Downloading fails many times this week.</p>\n<p>Maybe my network has problem or that file server has problem.</p>\n<p>Can anyone help?</p>\n<p>If there is a ftp server or whatever ,I would greatly appreciate it.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "56178",
      "postDate": "10/17/2014 13:45:00",
      "content": "<p>I got the same situation. I used lynx and tried six times for this file. Just wanted you know, you are not the only one has this problem. good luck.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "56188",
      "postDate": "10/17/2014 21:48:59",
      "content": "<p>maybe something has changed, but it did not take more than a couple of hours per file.</p>\n\n<p>try using a download manager.&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "56237",
      "postDate": "10/18/2014 21:47:25",
      "content": "<p>If you use an Amazon EC2 instance (e.g. m3.2xlarge for this problem) you can download the files onto it using&nbsp;wget in minutes. Note that you need to load cookies (instructions are discussed in other kaggle forums) and you will need to pay around $0.30-$0.50 per hour of use on Amazon. You also will need to&nbsp;attach an&nbsp;extra storage of 200GB to hold the raw data uncompressed.</p>\n<p>Another&nbsp;option is Microsoft Azure which gives&nbsp;a first month free and a $200 credit. Last time I used it I found it&nbsp;less easy to use than Amazon and download speeds are much slower.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "56256",
      "postDate": "10/19/2014 14:40:28",
      "content": "<p>could anyone suggest a download manager? i have never downloaded such large files before.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "56285",
      "postDate": "10/20/2014 13:35:48",
      "content": "<p>You may try one or more of the following:</p>\n<p>Google Chrome or Mozilla Firefox browser</p>\n<p>Direct wired connection of computer with router with nothing else attached to router</p>\n<p>Increase in Virtual Memory of the computer</p>\n<p>No other programs running on the computer</p>\n<p>[Added later: Cookies removed from the browser]</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "56286",
      "postDate": "10/20/2014 13:53:16",
      "content": "<p>Finally I&nbsp;manage to&nbsp;download these&nbsp;data.&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "56598",
      "postDate": "10/23/2014 12:54:05",
      "content": "<p>[quote=Steven Du;56286]</p>\n<p>Finally I&nbsp;manage to&nbsp;download these&nbsp;data.&nbsp;</p>\n<p>[/quote]</p>\n<p>how?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "56638",
      "postDate": "10/23/2014 20:14:55",
      "content": "<p>I think it will be better If we can download the file in a P2P way.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "56644",
      "postDate": "10/24/2014 00:12:29",
      "content": "<p>[quote=Abhishek;56598]</p>\n<p>[quote=Steven Du;56286]</p>\n<p>Finally I&nbsp;manage to&nbsp;download these&nbsp;data.&nbsp;</p>\n<p>[/quote]</p>\n<p>how?</p>\n<p>[/quote]</p>\n<p>Firefox default download manager, then I just wait...</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "56645",
      "postDate": "10/24/2014 00:32:27",
      "content": "<p>I would suggest Kaggle could come out an App that mange competitions ,data downloading, benchmarking, submission.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "56646",
      "postDate": "10/24/2014 01:11:17",
      "content": "<p>While I&nbsp;could somehow manage to download huge data files of Kaggle competitions, I&nbsp;can understand the pain of many other people.</p>\n<p>It will be nice if there existed a DataPlayer app, which will download a huge&nbsp;unzipped or zipped&nbsp;data file in&nbsp;sequential chunks and allow the person to&nbsp;review the data as they are being downloaded.</p>\n<p>If such an app does not exist, I might attempt to&nbsp;do something about it.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "56655",
      "postDate": "10/24/2014 04:14:51",
      "content": "<p>In case anyone's still having problems&nbsp;with this, here's one solution:</p>\n<p>1) login to kaggle.com from your browser</p>\n<p>2) export your kaggle.com cookies to a file. &nbsp;For example, if you are using Chrome, this extension makes it easy:&nbsp;https://chrome.google.com/webstore/detail/cookietxt-export/lopabhfecdfhgogdbojmaicoicjekelh</p>\n<p>3) use wget to download, taking advantage of&nbsp;its --continue and --load-cookies options. &nbsp;For example:</p>\n<p>attempt 1:</p>\n<p>wget --load-cookies=cookies.txt http://www.kaggle.com/c/seizure-prediction/download/Dog_1.tar.gz</p>\n<p>attempt 2 (if attempt 1 fails):</p>\n<p>wget --continue --load-cookies=cookies.txt http://www.kaggle.com/c/seizure-prediction/download/Dog_1.tar.gz</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "56658",
      "postDate": "10/24/2014 04:56:22",
      "content": "<p>@d00: Thanks for your nice writeup. It has taught me something useful.</p>\n<p>I am looking for a DataPlayer as described in my previous post. If you know something like that, please post it.</p>\n<p>Thanks, Lalit</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "56659",
      "postDate": "10/24/2014 05:15:30",
      "content": "<p>Lalit, the answer to your question depends on the binary format of the file being downloaded (in addition to the &quot;player&quot; being used to play/inspect&nbsp;the file, and the app being used to download it).</p>\n<p>For example, you can&nbsp;usually&nbsp;listen to a&nbsp;partially downloaded .mp3 file in a typical media player before it has finished downloading, due to the nature of the .mp3 file format. &nbsp;However, a .wav file usually won't work like that, unless the app being used to download&nbsp;the .wav file pre-allocated space for the entire .wav file ahead of time, or unless the app being used to play the .wav file didn't mind if the file size didn't match the data size described in the .wav file header.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "56689",
      "postDate": "10/24/2014 16:44:30",
      "content": "<p>@d00: Thanks. You understood my wish-list and I understood your &quot;miss-list&quot;.</p>\n<p><span style=\"line-height: 1.4\">@d00, Abhishek, and others: Let a few of us work on making this DataPlayer (or whatever you call it). It has a good demand with Big Data booming. Aside from wishing, my coding knowledge is limited.</span></p>\n<p><span style=\"line-height: 1.4\">Thanks, Lalit</span></p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "56701",
      "postDate": "10/25/2014 00:35:42",
      "content": "",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "56720",
      "postDate": "10/25/2014 09:08:01",
      "content": "<p>@d00: do you know what to do if following your instructions I get redirect status 302?</p>\n<p>It seems to me as if the cookies were not used.</p>\n\n<p>Update:</p>\n<p>Solved by logging in via wget and saving cookies. I do not understand why cookies from the browser do not work.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 56178,
      "author_name": "xinyulrsm",
      "author_url": "",
      "post_date": "10/17/2014 13:45:00",
      "content": "<p>I got the same situation. I used lynx and tried six times for this file. Just wanted you know, you are not the only one has this problem. good luck.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 56188,
      "author_name": "franklyn",
      "author_url": "",
      "post_date": "10/17/2014 21:48:59",
      "content": "<p>maybe something has changed, but it did not take more than a couple of hours per file.</p>\n\n<p>try using a download manager.&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 56237,
      "author_name": "lawrencechernin",
      "author_url": "",
      "post_date": "10/18/2014 21:47:25",
      "content": "<p>If you use an Amazon EC2 instance (e.g. m3.2xlarge for this problem) you can download the files onto it using&nbsp;wget in minutes. Note that you need to load cookies (instructions are discussed in other kaggle forums) and you will need to pay around $0.30-$0.50 per hour of use on Amazon. You also will need to&nbsp;attach an&nbsp;extra storage of 200GB to hold the raw data uncompressed.</p>\n<p>Another&nbsp;option is Microsoft Azure which gives&nbsp;a first month free and a $200 credit. Last time I used it I found it&nbsp;less easy to use than Amazon and download speeds are much slower.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 56256,
      "author_name": "siddjain",
      "author_url": "",
      "post_date": "10/19/2014 14:40:28",
      "content": "<p>could anyone suggest a download manager? i have never downloaded such large files before.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 56285,
      "author_name": "lalitapatel",
      "author_url": "",
      "post_date": "10/20/2014 13:35:48",
      "content": "<p>You may try one or more of the following:</p>\n<p>Google Chrome or Mozilla Firefox browser</p>\n<p>Direct wired connection of computer with router with nothing else attached to router</p>\n<p>Increase in Virtual Memory of the computer</p>\n<p>No other programs running on the computer</p>\n<p>[Added later: Cookies removed from the browser]</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 56286,
      "author_name": "stevendu",
      "author_url": "",
      "post_date": "10/20/2014 13:53:16",
      "content": "<p>Finally I&nbsp;manage to&nbsp;download these&nbsp;data.&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 56598,
      "author_name": "abhishek",
      "author_url": "",
      "post_date": "10/23/2014 12:54:05",
      "content": "<p>[quote=Steven Du;56286]</p>\n<p>Finally I&nbsp;manage to&nbsp;download these&nbsp;data.&nbsp;</p>\n<p>[/quote]</p>\n<p>how?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 56638,
      "author_name": "phhuang",
      "author_url": "",
      "post_date": "10/23/2014 20:14:55",
      "content": "<p>I think it will be better If we can download the file in a P2P way.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 56644,
      "author_name": "stevendu",
      "author_url": "",
      "post_date": "10/24/2014 00:12:29",
      "content": "<p>[quote=Abhishek;56598]</p>\n<p>[quote=Steven Du;56286]</p>\n<p>Finally I&nbsp;manage to&nbsp;download these&nbsp;data.&nbsp;</p>\n<p>[/quote]</p>\n<p>how?</p>\n<p>[/quote]</p>\n<p>Firefox default download manager, then I just wait...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 56645,
      "author_name": "stevendu",
      "author_url": "",
      "post_date": "10/24/2014 00:32:27",
      "content": "<p>I would suggest Kaggle could come out an App that mange competitions ,data downloading, benchmarking, submission.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 56646,
      "author_name": "lalitapatel",
      "author_url": "",
      "post_date": "10/24/2014 01:11:17",
      "content": "<p>While I&nbsp;could somehow manage to download huge data files of Kaggle competitions, I&nbsp;can understand the pain of many other people.</p>\n<p>It will be nice if there existed a DataPlayer app, which will download a huge&nbsp;unzipped or zipped&nbsp;data file in&nbsp;sequential chunks and allow the person to&nbsp;review the data as they are being downloaded.</p>\n<p>If such an app does not exist, I might attempt to&nbsp;do something about it.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 56655,
      "author_name": "drewabbot",
      "author_url": "",
      "post_date": "10/24/2014 04:14:51",
      "content": "<p>In case anyone's still having problems&nbsp;with this, here's one solution:</p>\n<p>1) login to kaggle.com from your browser</p>\n<p>2) export your kaggle.com cookies to a file. &nbsp;For example, if you are using Chrome, this extension makes it easy:&nbsp;https://chrome.google.com/webstore/detail/cookietxt-export/lopabhfecdfhgogdbojmaicoicjekelh</p>\n<p>3) use wget to download, taking advantage of&nbsp;its --continue and --load-cookies options. &nbsp;For example:</p>\n<p>attempt 1:</p>\n<p>wget --load-cookies=cookies.txt http://www.kaggle.com/c/seizure-prediction/download/Dog_1.tar.gz</p>\n<p>attempt 2 (if attempt 1 fails):</p>\n<p>wget --continue --load-cookies=cookies.txt http://www.kaggle.com/c/seizure-prediction/download/Dog_1.tar.gz</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 56658,
      "author_name": "lalitapatel",
      "author_url": "",
      "post_date": "10/24/2014 04:56:22",
      "content": "<p>@d00: Thanks for your nice writeup. It has taught me something useful.</p>\n<p>I am looking for a DataPlayer as described in my previous post. If you know something like that, please post it.</p>\n<p>Thanks, Lalit</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 56659,
      "author_name": "drewabbot",
      "author_url": "",
      "post_date": "10/24/2014 05:15:30",
      "content": "<p>Lalit, the answer to your question depends on the binary format of the file being downloaded (in addition to the &quot;player&quot; being used to play/inspect&nbsp;the file, and the app being used to download it).</p>\n<p>For example, you can&nbsp;usually&nbsp;listen to a&nbsp;partially downloaded .mp3 file in a typical media player before it has finished downloading, due to the nature of the .mp3 file format. &nbsp;However, a .wav file usually won't work like that, unless the app being used to download&nbsp;the .wav file pre-allocated space for the entire .wav file ahead of time, or unless the app being used to play the .wav file didn't mind if the file size didn't match the data size described in the .wav file header.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 56689,
      "author_name": "lalitapatel",
      "author_url": "",
      "post_date": "10/24/2014 16:44:30",
      "content": "<p>@d00: Thanks. You understood my wish-list and I understood your &quot;miss-list&quot;.</p>\n<p><span style=\"line-height: 1.4\">@d00, Abhishek, and others: Let a few of us work on making this DataPlayer (or whatever you call it). It has a good demand with Big Data booming. Aside from wishing, my coding knowledge is limited.</span></p>\n<p><span style=\"line-height: 1.4\">Thanks, Lalit</span></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 56701,
      "author_name": "lalitapatel",
      "author_url": "",
      "post_date": "10/25/2014 00:35:42",
      "content": "",
      "votes": null,
      "replies": []
    },
    {
      "id": 56720,
      "author_name": "aydarkhan",
      "author_url": "",
      "post_date": "10/25/2014 09:08:01",
      "content": "<p>@d00: do you know what to do if following your instructions I get redirect status 302?</p>\n<p>It seems to me as if the cookies were not used.</p>\n\n<p>Update:</p>\n<p>Solved by logging in via wget and saving cookies. I do not understand why cookies from the browser do not work.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "56172": "",
    "56178": "",
    "56188": "",
    "56237": "",
    "56256": "",
    "56285": "",
    "56286": "",
    "56598": "",
    "56638": "",
    "56644": "",
    "56645": "",
    "56646": "",
    "56655": "",
    "56658": "",
    "56659": "",
    "56689": "",
    "56701": "",
    "56720": ""
  },
  "source": "meta"
}