I want to download file page_views.csv.zip, but my connection fails at some point. Is it possible to upload the file in parts or upload this file to a torrent service?
", "messages": [ { "id": "137973", "postDate": "10/06/2016 11:04:20", "content": "I want to download file page_views.csv.zip, but my connection fails at some point. Is it possible to upload the file in parts or upload this file to a torrent service?
", "rawMarkdown": "I want to download file page_views.csv.zip, but my connection fails at some point. Is it possible to upload the file in parts or upload this file to a torrent service?", "votes": null }, { "id": "138033", "postDate": "10/06/2016 18:29:09", "content": "Hi, I'm afraid torrent is not going to be an option. From the "Welcome" post you can read:
\n\n"Expect a traffic jam to download the 30 GB file during the launch gold rush (sorry!). Use a download manager with resuming capabilities and we promise you'll eventually get the file. There is also a small sample available. Please read our note on torrents before you suggest using torrents."
\n\nCheers!
", "rawMarkdown": "Hi, I'm afraid torrent is not going to be an option. From the \"[Welcome][1]\" post you can read:\r\n\r\n\"Expect a traffic jam to download the 30 GB file during the launch gold rush (sorry!). Use a download manager with resuming capabilities and we promise you'll eventually get the file. There is also a small sample available. Please read our [note][2] on torrents before you suggest using torrents.\"\r\n\r\nCheers!\r\n\r\n\r\n [1]: https://www.kaggle.com/c/outbrain-click-prediction/forums/t/24107/welcome\r\n [2]: https://www.kaggle.com/wiki/ANoteOnTorrents", "votes": null }, { "id": "138185", "postDate": "10/07/2016 15:33:16", "content": "I'm pretty sure one could torrent an encrypted version of the data and then force users to download the key directly from kaggle. Versioning could be handled in this way too. Given the failure rate, it will take me probably a week to d/l this file.
\n\nMaybe we should make a "kaggle" a verb for something that's impossible to d/l, a la "I tried to download the new iOS on launch but download system is totally kaggled".
", "rawMarkdown": "I'm pretty sure one could torrent an encrypted version of the data and then force users to download the key directly from kaggle. Versioning could be handled in this way too. Given the failure rate, it will take me probably a week to d/l this file. \r\n\r\nMaybe we should make a \"kaggle\" a verb for something that's impossible to d/l, a la \"I tried to download the new iOS on launch but download system is totally kaggled\".", "votes": null }, { "id": "138247", "postDate": "10/07/2016 18:50:43", "content": "Well, I've been able to download it via curl:
cookiescopy as cURLcurl commandI was able to download it at once, with a pretty shitty connexion. But the nice thing about curl is that you can resume the download by setting an offset:
\n\ncurl -C offset url\nSo if it has stopped, you can check for the size of the current file, and set it (+1 I think) as the offset parameter.
\n\nHope this is helpful to somebody, I know it shouldn't be that elaborated, but it's feasible ;)
", "rawMarkdown": "Well, I've been able to download it via `curl`:\r\n\r\n 1. Download some of the light files so in your browser you have to accept the data agreement and this action is stored in your `cookies`\r\n 2. In your browser (in my case it is Chrome) right click and open the Inspector\r\n 3. Click on the super heavy file and in the Inspector check the petition in the Network tab\r\n 4. Right click on the network petition and `copy as cURL`\r\n 5. Finally you can open the terminal and paste the `curl` command\r\n\r\nI was able to download it at once, with a pretty shitty connexion. But the nice thing about curl is that you can resume the download by setting an offset:\r\n\r\n curl -C offset url\r\n\r\nSo if it has stopped, you can check for the size of the current file, and set it (+1 I think) as the offset parameter.\r\n\r\nHope this is helpful to somebody, I know it shouldn't be that elaborated, but it's feasible ;)", "votes": null }, { "id": "138585", "postDate": "10/09/2016 17:00:59", "content": "Most likely, that's against the contest rules. But just as a test, no, it's not possible. After several days of retrying:
\n\n"\nwget 'https://kaggle2.blob.core.windows.net/competitions-data/kaggle/5497/page_views.csv.zip? ...'
\n\nResolving kaggle2.blob.core.windows.net (kaggle2.blob.core.windows.net)... 40.116.120.24\nConnecting to kaggle2.blob.core.windows.net (kaggle2.blob.core.windows.net)|40.116.120.24|:443... connected.\nHTTP request sent, awaiting response... 200 OK\nLength: 31905475875 (30G) [application/zip]\nSaving to: 'page_views.csv.zip... '\npage_views.csv.zip@sv=2012-02-12&se=201 35%[==========================> ] 10.45G 277KB/s in 14h 12m
\n\n\n\n2016-10-08 07:46:50 (214 KB/s) - Read error at byte 11221434368/31905475875 (Resource temporarily unavailable, try again.). Retrying.
\n\nHTTP request sent, awaiting response... 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\n2016-10-09 09:22:35 ERROR 403: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature..\n"
", "rawMarkdown": "Most likely, that's against the contest rules. But just as a test, no, it's not possible. After several days of retrying:\r\n\r\n\"\r\nwget 'https://kaggle2.blob.core.windows.net/competitions-data/kaggle/5497/page_views.csv.zip? ...'Firefox allows restarting if the download fails. I found I couldn't use a command-line utility because I needed to have an authenticated browser session, and I couldn't work out how to do that.
\n\nI've been able to resume the download four times so far...
", "rawMarkdown": "Firefox allows restarting if the download fails. I found I couldn't use a command-line utility because I needed to have an authenticated browser session, and I couldn't work out how to do that.\r\n\r\nI've been able to resume the download four times so far...", "votes": null }, { "id": "138841", "postDate": "10/11/2016 06:35:27", "content": "Chrome has the same. However it seems the azure-hosted files have a token that times out after a few days? So even if you keep restarting, you have to get it in that period or the token (on their end) will timeout, apparently. Retrying therefore may not work in many circumstances.
\n\nAlso the dl speed is limited to about 200k/s (observed) which further increases the likelihood of failure.
\n\nI finally d/l'ed the file with some wget-foo.
", "rawMarkdown": "Chrome has the same. However it seems the azure-hosted files have a token that times out after a few days? So even if you keep restarting, you have to get it in that period or the token (on their end) will timeout, apparently. Retrying therefore may not work in many circumstances. \r\n\r\nAlso the dl speed is limited to about 200k/s (observed) which further increases the likelihood of failure.\r\n\r\nI finally d/l'ed the file with some wget-foo.", "votes": null } ], "comments": [ { "id": 138033, "author_name": "guiemb", "author_url": "", "post_date": "10/06/2016 18:29:09", "content": "Hi, I'm afraid torrent is not going to be an option. From the "Welcome" post you can read:
\n\n"Expect a traffic jam to download the 30 GB file during the launch gold rush (sorry!). Use a download manager with resuming capabilities and we promise you'll eventually get the file. There is also a small sample available. Please read our note on torrents before you suggest using torrents."
\n\nCheers!
", "votes": null, "replies": [] }, { "id": 138185, "author_name": "rgwood", "author_url": "", "post_date": "10/07/2016 15:33:16", "content": "I'm pretty sure one could torrent an encrypted version of the data and then force users to download the key directly from kaggle. Versioning could be handled in this way too. Given the failure rate, it will take me probably a week to d/l this file.
\n\nMaybe we should make a "kaggle" a verb for something that's impossible to d/l, a la "I tried to download the new iOS on launch but download system is totally kaggled".
", "votes": null, "replies": [] }, { "id": 138247, "author_name": "guiemb", "author_url": "", "post_date": "10/07/2016 18:50:43", "content": "Well, I've been able to download it via curl:
cookiescopy as cURLcurl commandI was able to download it at once, with a pretty shitty connexion. But the nice thing about curl is that you can resume the download by setting an offset:
\n\ncurl -C offset url\nSo if it has stopped, you can check for the size of the current file, and set it (+1 I think) as the offset parameter.
\n\nHope this is helpful to somebody, I know it shouldn't be that elaborated, but it's feasible ;)
", "votes": null, "replies": [] }, { "id": 138585, "author_name": "rgwood", "author_url": "", "post_date": "10/09/2016 17:00:59", "content": "Most likely, that's against the contest rules. But just as a test, no, it's not possible. After several days of retrying:
\n\n"\nwget 'https://kaggle2.blob.core.windows.net/competitions-data/kaggle/5497/page_views.csv.zip? ...'
\n\nResolving kaggle2.blob.core.windows.net (kaggle2.blob.core.windows.net)... 40.116.120.24\nConnecting to kaggle2.blob.core.windows.net (kaggle2.blob.core.windows.net)|40.116.120.24|:443... connected.\nHTTP request sent, awaiting response... 200 OK\nLength: 31905475875 (30G) [application/zip]\nSaving to: 'page_views.csv.zip... '\npage_views.csv.zip@sv=2012-02-12&se=201 35%[==========================> ] 10.45G 277KB/s in 14h 12m
\n\n\n\n2016-10-08 07:46:50 (214 KB/s) - Read error at byte 11221434368/31905475875 (Resource temporarily unavailable, try again.). Retrying.
\n\nHTTP request sent, awaiting response... 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\n2016-10-09 09:22:35 ERROR 403: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature..\n"
", "votes": null, "replies": [] }, { "id": 138648, "author_name": "terribletadpole", "author_url": "", "post_date": "10/10/2016 05:13:12", "content": "Firefox allows restarting if the download fails. I found I couldn't use a command-line utility because I needed to have an authenticated browser session, and I couldn't work out how to do that.
\n\nI've been able to resume the download four times so far...
", "votes": null, "replies": [] }, { "id": 138841, "author_name": "rgwood", "author_url": "", "post_date": "10/11/2016 06:35:27", "content": "Chrome has the same. However it seems the azure-hosted files have a token that times out after a few days? So even if you keep restarting, you have to get it in that period or the token (on their end) will timeout, apparently. Retrying therefore may not work in many circumstances.
\n\nAlso the dl speed is limited to about 200k/s (observed) which further increases the likelihood of failure.
\n\nI finally d/l'ed the file with some wget-foo.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "137973": "I want to download file page_views.csv.zip, but my connection fails at some point. Is it possible to upload the file in parts or upload this file to a torrent service?", "138033": "Hi, I'm afraid torrent is not going to be an option. From the \"[Welcome][1]\" post you can read:\r\n\r\n\"Expect a traffic jam to download the 30 GB file during the launch gold rush (sorry!). Use a download manager with resuming capabilities and we promise you'll eventually get the file. There is also a small sample available. Please read our [note][2] on torrents before you suggest using torrents.\"\r\n\r\nCheers!\r\n\r\n\r\n [1]: https://www.kaggle.com/c/outbrain-click-prediction/forums/t/24107/welcome\r\n [2]: https://www.kaggle.com/wiki/ANoteOnTorrents", "138185": "I'm pretty sure one could torrent an encrypted version of the data and then force users to download the key directly from kaggle. Versioning could be handled in this way too. Given the failure rate, it will take me probably a week to d/l this file. \r\n\r\nMaybe we should make a \"kaggle\" a verb for something that's impossible to d/l, a la \"I tried to download the new iOS on launch but download system is totally kaggled\".", "138247": "Well, I've been able to download it via `curl`:\r\n\r\n 1. Download some of the light files so in your browser you have to accept the data agreement and this action is stored in your `cookies`\r\n 2. In your browser (in my case it is Chrome) right click and open the Inspector\r\n 3. Click on the super heavy file and in the Inspector check the petition in the Network tab\r\n 4. Right click on the network petition and `copy as cURL`\r\n 5. Finally you can open the terminal and paste the `curl` command\r\n\r\nI was able to download it at once, with a pretty shitty connexion. But the nice thing about curl is that you can resume the download by setting an offset:\r\n\r\n curl -C offset url\r\n\r\nSo if it has stopped, you can check for the size of the current file, and set it (+1 I think) as the offset parameter.\r\n\r\nHope this is helpful to somebody, I know it shouldn't be that elaborated, but it's feasible ;)", "138585": "Most likely, that's against the contest rules. But just as a test, no, it's not possible. After several days of retrying:\r\n\r\n\"\r\nwget 'https://kaggle2.blob.core.windows.net/competitions-data/kaggle/5497/page_views.csv.zip? ...'