{
  "id": 18803,
  "title": "downloading from command line (wget)",
  "url": "/competitions/second-annual-data-science-bowl/discussion/18803",
  "author_name": "",
  "post_date": "2016-02-06T23:35:07.370Z",
  "votes": 1,
  "comment_count": 8,
  "views": 2060,
  "content": "<p>Has anyone been able to successfully download the data using wget? I exported my cookies and tried the following command: wget --load-cookies=cookies.txt -P data <a href=\"https://www.kaggle.com/c/second-annual-data-science-bowl/download/train.zip\">https://www.kaggle.com/c/second-annual-data-science-bowl/download/train.zip</a>\nHowever, it seems to only be getting me some generic Kaggle HTML page. Any suggestions? </p>",
  "messages": [
    {
      "id": "107109",
      "postDate": "02/06/2016 23:35:07",
      "content": "<p>Has anyone been able to successfully download the data using wget? I exported my cookies and tried the following command: wget --load-cookies=cookies.txt -P data <a href=\"https://www.kaggle.com/c/second-annual-data-science-bowl/download/train.zip\">https://www.kaggle.com/c/second-annual-data-science-bowl/download/train.zip</a>\nHowever, it seems to only be getting me some generic Kaggle HTML page. Any suggestions? </p>",
      "rawMarkdown": "Has anyone been able to successfully download the data using wget? I exported my cookies and tried the following command: wget --load-cookies=cookies.txt -P data https://www.kaggle.com/c/second-annual-data-science-bowl/download/train.zip\r\nHowever, it seems to only be getting me some generic Kaggle HTML page. Any suggestions?",
      "votes": null
    },
    {
      "id": "107209",
      "postDate": "02/08/2016 12:58:41",
      "content": "<p>try wget -x --load-cookies cookies.txt -nH <a href=\"https://www.kaggle.com/c/second-annual-data-science-bowl/download/train.zip\">https://www.kaggle.com/c/second-annual-data-science-bowl/download/train.zip</a></p>",
      "rawMarkdown": "try wget -x --load-cookies cookies.txt -nH https://www.kaggle.com/c/second-annual-data-science-bowl/download/train.zip",
      "votes": null
    },
    {
      "id": "107210",
      "postDate": "02/08/2016 12:58:58",
      "content": "<p>try wget -x --load-cookies cookies.txt -nH <a href=\"https://www.kaggle.com/c/second-annual-data-science-bowl/download/train.zip\">https://www.kaggle.com/c/second-annual-data-science-bowl/download/train.zip</a></p>",
      "rawMarkdown": "try wget -x --load-cookies cookies.txt -nH https://www.kaggle.com/c/second-annual-data-science-bowl/download/train.zip",
      "votes": null
    },
    {
      "id": "107233",
      "postDate": "02/08/2016 16:57:39",
      "content": "<p>I had the same problem. I wasn't able to download data even with cookies. For some reason Kaggle server wanted not only cookies, but all other headers.</p>\n\n<p>You can learn how to provide wget with required headers at this page: <a href=\"http://stackoverflow.com/questions/7618155/how-to-duplicate-a-request-using-wget-or-curl-with-raw-headers\">http://stackoverflow.com/questions/7618155/how-to-duplicate-a-request-using-wget-or-curl-with-raw-headers</a></p>\n\n<p>If you are using Firefox, you can find, which headers are passed using Firefox Developer Tools. Open new tab, then press F12 to open developer tools and select &quot;Network&quot; tab. Then open <a href=\"https://www.kaggle.com/c/second-annual-data-science-bowl/download/train.zip\">https://www.kaggle.com/c/second-annual-data-science-bowl/download/train.zip</a> Requests will appear in &quot;Network&quot; tab. Select request for train.zip and you will see all needed headers on the right.</p>",
      "rawMarkdown": "I had the same problem. I wasn't able to download data even with cookies. For some reason Kaggle server wanted not only cookies, but all other headers.\r\n\r\nYou can learn how to provide wget with required headers at this page: http://stackoverflow.com/questions/7618155/how-to-duplicate-a-request-using-wget-or-curl-with-raw-headers\r\n\r\nIf you are using Firefox, you can find, which headers are passed using Firefox Developer Tools. Open new tab, then press F12 to open developer tools and select \"Network\" tab. Then open https://www.kaggle.com/c/second-annual-data-science-bowl/download/train.zip Requests will appear in \"Network\" tab. Select request for train.zip and you will see all needed headers on the right.",
      "votes": null
    },
    {
      "id": "108957",
      "postDate": "02/21/2016 23:08:07",
      "content": "<p>Hi guys - I seemed to have been able to use wget to get the data but when I unzip it, I get the following error: </p>\n\n<p>End-of-central-directory signature not found.  Either this file is not\n  a zipfile, or it constitutes one disk of a multi-part archive.  In the\n  latter case the central directory and zipfile comment will be found on\n  the last disk(s) of this archive.\nunzip:  cannot find zipfile directory in one of /home/ubuntu/data/train/c/second-annual-data-science-bowl/download/train.zip.1 or\n        /home/ubuntu/data/train/c/second-annual-data-science-bowl/download/train.zip.1.zip, and cannot find /home/ubuntu/data/train/c/second-annual-data-science-bowl/download/train.zip.1.ZIP, period.</p>\n\n<p>Has anyone run into the same issue?  </p>\n\n<p>Any help would be appreciated! </p>\n\n<p>Thanks! </p>\n\n<p>John </p>",
      "rawMarkdown": "Hi guys - I seemed to have been able to use wget to get the data but when I unzip it, I get the following error: \r\n\r\n End-of-central-directory signature not found.  Either this file is not\r\n  a zipfile, or it constitutes one disk of a multi-part archive.  In the\r\n  latter case the central directory and zipfile comment will be found on\r\n  the last disk(s) of this archive.\r\nunzip:  cannot find zipfile directory in one of /home/ubuntu/data/train/c/second-annual-data-science-bowl/download/train.zip.1 or\r\n        /home/ubuntu/data/train/c/second-annual-data-science-bowl/download/train.zip.1.zip, and cannot find /home/ubuntu/data/train/c/second-annual-data-science-bowl/download/train.zip.1.ZIP, period.\r\n\r\n\r\nHas anyone run into the same issue?  \r\n\r\nAny help would be appreciated! \r\n\r\nThanks! \r\n\r\nJohn",
      "votes": null
    },
    {
      "id": "108959",
      "postDate": "02/21/2016 23:19:58",
      "content": "<p>probably the file did not fully download</p>",
      "rawMarkdown": "probably the file did not fully download",
      "votes": null
    },
    {
      "id": "108967",
      "postDate": "02/22/2016 01:40:45",
      "content": "<p>Thanks, WD!  </p>",
      "rawMarkdown": "Thanks, WD!",
      "votes": null
    },
    {
      "id": "109958",
      "postDate": "03/01/2016 18:29:49",
      "content": "<p>Same issue as @john. Using the command with cookies exported:</p>\n\n<p><code>wget -x --load-cookies cookies.txt -nH https://www.kaggle.com/c/second-annual-data-science-bowl/download/train.zip</code></p>\n\n<p>returns a file:</p>\n\n<p>login?ReturnUrl=%2Fc%2Fsecond-annual-data-science-bowl%2Fdownload%2Fvalidate.zip</p>\n\n<p>with contents:</p>\n\n<p>Archive:  login?ReturnUrl=%2Fc%2Fsecond-annual-data-science-bowl%2Fdownload%2Fvalidate.zip\n[login?ReturnUrl=%2Fc%2Fsecond-annual-data-science-bowl%2Fdownload%2Fvalidate.zip]\n  End-of-central-directory signature not found.  Either this file is not\n  a zipfile, or it constitutes one disk of a multi-part archive.  In the\n  latter case the central directory and zipfile comment will be found on\n  the last disk(s) of this archive.\nzipinfo:  cannot find zipfile directory in one of login?ReturnUrl=%2Fc%2Fsecond-annual-data-science-bowl%2Fdownload%2Fvalidate.zip or\n          login?ReturnUrl=%2Fc%2Fsecond-annual-data-science-bowl%2Fdownload%2Fvalidate.zip.zip, and cannot find login?ReturnUrl=%2Fc%2Fsecond-annual-data-science-bowl%2Fdownload%2Fvalidate.zip.ZIP, period.</p>\n\n<p>No zipfiles found.</p>",
      "rawMarkdown": "Same issue as @john. Using the command with cookies exported:\r\n\r\n`wget -x --load-cookies cookies.txt -nH https://www.kaggle.com/c/second-annual-data-science-bowl/download/train.zip`\r\n\r\nreturns a file:\r\n\r\nlogin?ReturnUrl=%2Fc%2Fsecond-annual-data-science-bowl%2Fdownload%2Fvalidate.zip\r\n\r\nwith contents:\r\n\r\nArchive:  login?ReturnUrl=%2Fc%2Fsecond-annual-data-science-bowl%2Fdownload%2Fvalidate.zip\r\n[login?ReturnUrl=%2Fc%2Fsecond-annual-data-science-bowl%2Fdownload%2Fvalidate.zip]\r\n  End-of-central-directory signature not found.  Either this file is not\r\n  a zipfile, or it constitutes one disk of a multi-part archive.  In the\r\n  latter case the central directory and zipfile comment will be found on\r\n  the last disk(s) of this archive.\r\nzipinfo:  cannot find zipfile directory in one of login?ReturnUrl=%2Fc%2Fsecond-annual-data-science-bowl%2Fdownload%2Fvalidate.zip or\r\n          login?ReturnUrl=%2Fc%2Fsecond-annual-data-science-bowl%2Fdownload%2Fvalidate.zip.zip, and cannot find login?ReturnUrl=%2Fc%2Fsecond-annual-data-science-bowl%2Fdownload%2Fvalidate.zip.ZIP, period.\r\n\r\nNo zipfiles found.",
      "votes": null
    },
    {
      "id": "125629",
      "postDate": "07/01/2016 00:14:34",
      "content": "<p>I am running into this as well.  Any ideas?  Has the data distribution changed?</p>",
      "rawMarkdown": "I am running into this as well.  Any ideas?  Has the data distribution changed?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 107209,
      "author_name": "wouterd1",
      "author_url": "",
      "post_date": "02/08/2016 12:58:41",
      "content": "<p>try wget -x --load-cookies cookies.txt -nH <a href=\"https://www.kaggle.com/c/second-annual-data-science-bowl/download/train.zip\">https://www.kaggle.com/c/second-annual-data-science-bowl/download/train.zip</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 107210,
      "author_name": "wouterd1",
      "author_url": "",
      "post_date": "02/08/2016 12:58:58",
      "content": "<p>try wget -x --load-cookies cookies.txt -nH <a href=\"https://www.kaggle.com/c/second-annual-data-science-bowl/download/train.zip\">https://www.kaggle.com/c/second-annual-data-science-bowl/download/train.zip</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 107233,
      "author_name": "timmmy",
      "author_url": "",
      "post_date": "02/08/2016 16:57:39",
      "content": "<p>I had the same problem. I wasn't able to download data even with cookies. For some reason Kaggle server wanted not only cookies, but all other headers.</p>\n\n<p>You can learn how to provide wget with required headers at this page: <a href=\"http://stackoverflow.com/questions/7618155/how-to-duplicate-a-request-using-wget-or-curl-with-raw-headers\">http://stackoverflow.com/questions/7618155/how-to-duplicate-a-request-using-wget-or-curl-with-raw-headers</a></p>\n\n<p>If you are using Firefox, you can find, which headers are passed using Firefox Developer Tools. Open new tab, then press F12 to open developer tools and select &quot;Network&quot; tab. Then open <a href=\"https://www.kaggle.com/c/second-annual-data-science-bowl/download/train.zip\">https://www.kaggle.com/c/second-annual-data-science-bowl/download/train.zip</a> Requests will appear in &quot;Network&quot; tab. Select request for train.zip and you will see all needed headers on the right.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 108957,
      "author_name": "johngunawan",
      "author_url": "",
      "post_date": "02/21/2016 23:08:07",
      "content": "<p>Hi guys - I seemed to have been able to use wget to get the data but when I unzip it, I get the following error: </p>\n\n<p>End-of-central-directory signature not found.  Either this file is not\n  a zipfile, or it constitutes one disk of a multi-part archive.  In the\n  latter case the central directory and zipfile comment will be found on\n  the last disk(s) of this archive.\nunzip:  cannot find zipfile directory in one of /home/ubuntu/data/train/c/second-annual-data-science-bowl/download/train.zip.1 or\n        /home/ubuntu/data/train/c/second-annual-data-science-bowl/download/train.zip.1.zip, and cannot find /home/ubuntu/data/train/c/second-annual-data-science-bowl/download/train.zip.1.ZIP, period.</p>\n\n<p>Has anyone run into the same issue?  </p>\n\n<p>Any help would be appreciated! </p>\n\n<p>Thanks! </p>\n\n<p>John </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 108959,
      "author_name": "wouterd1",
      "author_url": "",
      "post_date": "02/21/2016 23:19:58",
      "content": "<p>probably the file did not fully download</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 108967,
      "author_name": "johngunawan",
      "author_url": "",
      "post_date": "02/22/2016 01:40:45",
      "content": "<p>Thanks, WD!  </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 109958,
      "author_name": "darylwagg",
      "author_url": "",
      "post_date": "03/01/2016 18:29:49",
      "content": "<p>Same issue as @john. Using the command with cookies exported:</p>\n\n<p><code>wget -x --load-cookies cookies.txt -nH https://www.kaggle.com/c/second-annual-data-science-bowl/download/train.zip</code></p>\n\n<p>returns a file:</p>\n\n<p>login?ReturnUrl=%2Fc%2Fsecond-annual-data-science-bowl%2Fdownload%2Fvalidate.zip</p>\n\n<p>with contents:</p>\n\n<p>Archive:  login?ReturnUrl=%2Fc%2Fsecond-annual-data-science-bowl%2Fdownload%2Fvalidate.zip\n[login?ReturnUrl=%2Fc%2Fsecond-annual-data-science-bowl%2Fdownload%2Fvalidate.zip]\n  End-of-central-directory signature not found.  Either this file is not\n  a zipfile, or it constitutes one disk of a multi-part archive.  In the\n  latter case the central directory and zipfile comment will be found on\n  the last disk(s) of this archive.\nzipinfo:  cannot find zipfile directory in one of login?ReturnUrl=%2Fc%2Fsecond-annual-data-science-bowl%2Fdownload%2Fvalidate.zip or\n          login?ReturnUrl=%2Fc%2Fsecond-annual-data-science-bowl%2Fdownload%2Fvalidate.zip.zip, and cannot find login?ReturnUrl=%2Fc%2Fsecond-annual-data-science-bowl%2Fdownload%2Fvalidate.zip.ZIP, period.</p>\n\n<p>No zipfiles found.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 125629,
      "author_name": "brocktibert",
      "author_url": "",
      "post_date": "07/01/2016 00:14:34",
      "content": "<p>I am running into this as well.  Any ideas?  Has the data distribution changed?</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "107109": "Has anyone been able to successfully download the data using wget? I exported my cookies and tried the following command: wget --load-cookies=cookies.txt -P data https://www.kaggle.com/c/second-annual-data-science-bowl/download/train.zip\r\nHowever, it seems to only be getting me some generic Kaggle HTML page. Any suggestions?",
    "107209": "try wget -x --load-cookies cookies.txt -nH https://www.kaggle.com/c/second-annual-data-science-bowl/download/train.zip",
    "107210": "try wget -x --load-cookies cookies.txt -nH https://www.kaggle.com/c/second-annual-data-science-bowl/download/train.zip",
    "107233": "I had the same problem. I wasn't able to download data even with cookies. For some reason Kaggle server wanted not only cookies, but all other headers.\r\n\r\nYou can learn how to provide wget with required headers at this page: http://stackoverflow.com/questions/7618155/how-to-duplicate-a-request-using-wget-or-curl-with-raw-headers\r\n\r\nIf you are using Firefox, you can find, which headers are passed using Firefox Developer Tools. Open new tab, then press F12 to open developer tools and select \"Network\" tab. Then open https://www.kaggle.com/c/second-annual-data-science-bowl/download/train.zip Requests will appear in \"Network\" tab. Select request for train.zip and you will see all needed headers on the right.",
    "108957": "Hi guys - I seemed to have been able to use wget to get the data but when I unzip it, I get the following error: \r\n\r\n End-of-central-directory signature not found.  Either this file is not\r\n  a zipfile, or it constitutes one disk of a multi-part archive.  In the\r\n  latter case the central directory and zipfile comment will be found on\r\n  the last disk(s) of this archive.\r\nunzip:  cannot find zipfile directory in one of /home/ubuntu/data/train/c/second-annual-data-science-bowl/download/train.zip.1 or\r\n        /home/ubuntu/data/train/c/second-annual-data-science-bowl/download/train.zip.1.zip, and cannot find /home/ubuntu/data/train/c/second-annual-data-science-bowl/download/train.zip.1.ZIP, period.\r\n\r\n\r\nHas anyone run into the same issue?  \r\n\r\nAny help would be appreciated! \r\n\r\nThanks! \r\n\r\nJohn",
    "108959": "probably the file did not fully download",
    "108967": "Thanks, WD!",
    "109958": "Same issue as @john. Using the command with cookies exported:\r\n\r\n`wget -x --load-cookies cookies.txt -nH https://www.kaggle.com/c/second-annual-data-science-bowl/download/train.zip`\r\n\r\nreturns a file:\r\n\r\nlogin?ReturnUrl=%2Fc%2Fsecond-annual-data-science-bowl%2Fdownload%2Fvalidate.zip\r\n\r\nwith contents:\r\n\r\nArchive:  login?ReturnUrl=%2Fc%2Fsecond-annual-data-science-bowl%2Fdownload%2Fvalidate.zip\r\n[login?ReturnUrl=%2Fc%2Fsecond-annual-data-science-bowl%2Fdownload%2Fvalidate.zip]\r\n  End-of-central-directory signature not found.  Either this file is not\r\n  a zipfile, or it constitutes one disk of a multi-part archive.  In the\r\n  latter case the central directory and zipfile comment will be found on\r\n  the last disk(s) of this archive.\r\nzipinfo:  cannot find zipfile directory in one of login?ReturnUrl=%2Fc%2Fsecond-annual-data-science-bowl%2Fdownload%2Fvalidate.zip or\r\n          login?ReturnUrl=%2Fc%2Fsecond-annual-data-science-bowl%2Fdownload%2Fvalidate.zip.zip, and cannot find login?ReturnUrl=%2Fc%2Fsecond-annual-data-science-bowl%2Fdownload%2Fvalidate.zip.ZIP, period.\r\n\r\nNo zipfiles found.",
    "125629": "I am running into this as well.  Any ideas?  Has the data distribution changed?"
  },
  "source": "meta"
}