{ "id": 6060, "title": "download data", "url": "/competitions/yandex-personalized-web-search-challenge/discussion/6060", "author_name": "", "post_date": "2013-10-20T12:17:59.837Z", "votes": 1, "comment_count": 15, "views": 5844, "content": "

how did you manage to download the data? i tried multiple time via browser or wget, but without success...

", "messages": [ { "id": "32459", "postDate": "10/20/2013 12:17:59", "content": "

how did you manage to download the data? i tried multiple time via browser or wget, but without success...

", "rawMarkdown": "", "votes": null }, { "id": "32460", "postDate": "10/20/2013 12:56:01", "content": "

I used DownThemAll!, a firefox extension quite similar to a download manager.

", "rawMarkdown": "", "votes": null }, { "id": "32465", "postDate": "10/20/2013 16:47:10", "content": "

Many thanks for this hint. I will give it a try

", "rawMarkdown": "", "votes": null }, { "id": "32473", "postDate": "10/21/2013 02:36:13", "content": "

Thanks so much EGO for the tip on using "DownLoadThemAll!". I am in Australia and had tried half a dozen times unsuccessfully to download the data but finally got it working with this tool.

", "rawMarkdown": "", "votes": null }, { "id": "32734", "postDate": "10/27/2013 11:07:40", "content": "

Hi,

\n

I cannot download the dataset due to two problems: download managers cannot resume and I have low internet connection to your site.  Can you please provide an ftp or Torent link for data?

", "rawMarkdown": "", "votes": null }, { "id": "32745", "postDate": "10/27/2013 19:50:30", "content": "

I am not sure who you are asking for ftp or torrent link to. In my case, considering I have small upload speed and the time I can keep my computer on, it would take more time than the rest of the competition. So, it does make sense. Hope admins can help you.

", "rawMarkdown": "", "votes": null }, { "id": "32776", "postDate": "10/28/2013 13:46:34", "content": "

What I was going to try, is to use wget in screen on a Linux server. But then I need the direct link to the train.gz file, and apparently this is not it: https://www.kaggle.com/c/yandex-personalized-web-search-challenge/download/train.gz Did anyone find the direct link to the file?

", "rawMarkdown": "", "votes": null }, { "id": "32777", "postDate": "10/28/2013 15:00:14", "content": "

We don't have support for torrents at this time. This suggestion comes up often, but torrents are not an ideal mechanism for us for two reasons:

\n\n

You should be able to resume downloads for up to 3 days after starting them, regardless of browser. There may be combinations of browsers/managers where this doesn't work, but it should work in most cases.

\n

If you want to use a server or the command line to download a file, you must export your Kaggle cookies from your browser (this chrome extension is the easiest way) and then call wget's --load-cookies option.  Because of the rules clause above, we cannot have naked download links - you need to be logged in and have accepted the rules. After passing your Kaggle cookies, wget should work fine.  Note that the file download links will redirect to something like "https://kaggle2.blob.core.windows.net" after you've clicked the download link. This is the URL you should give to wget.

", "rawMarkdown": "", "votes": null }, { "id": "32823", "postDate": "10/29/2013 11:21:23", "content": "

Thanks, I succeeded in downloading the data!

", "rawMarkdown": "", "votes": null }, { "id": "33738", "postDate": "11/15/2013 04:48:39", "content": "

How can one open this dataset file?

\n

Any software or do we have to write a code?

\n

I am new in this field so might ask stupid questions!

\n

Please help

", "rawMarkdown": "", "votes": null }, { "id": "33743", "postDate": "11/15/2013 07:52:45", "content": "

You can use gunzip on linux or 7-zip on Windows to extract the .gz files.

", "rawMarkdown": "", "votes": null }, { "id": "33755", "postDate": "11/15/2013 16:30:06", "content": "

[quote=Suzan Verberne;33743]

\n

You can use gunzip on linux or 7-zip on Windows to extract the .gz files.

\n

[/quote]

\n

 

\n

I have extracted the file using Winrar. But now how can I manipulate or use this data.

\n

Kindly help.

\n

I am doing a research on click modelling.

", "rawMarkdown": "", "votes": null }, { "id": "33790", "postDate": "11/16/2013 05:00:47", "content": "

[quote=William Cukierski;32777]

\n

Note that the file download links will redirect to something like "https://kaggle2.blob.core.windows.net" after you've clicked the download link. This is the URL you should give to wget.

\n

[/quote]

\n

 

\n

Just a quick note to others in this situation - using the ".windows.net" addresses gave me 404's for some reason, but using the "http://www.kaggle.com/c/yandex-personalized-web-search-challenge/download/xyz.gz" urls from the download page did work.

", "rawMarkdown": "", "votes": null }, { "id": "34849", "postDate": "11/18/2013 10:00:57", "content": "

[quote=Sheikh Adnan Ahmed Usmani;33755]

\n

[quote=Suzan Verberne;33743]

\n

You can use gunzip on linux or 7-zip on Windows to extract the .gz files.

\n

[/quote]

\n

 

\n

I have extracted the file using Winrar. But now how can I manipulate or use this data.

\n

Kindly help.

\n

I am doing a research on click modelling.

\n

[/quote]

\n

 

\n

I am not sure what kind of answer you expect. Here is a description of the data: https://www.kaggle.com/c/yandex-personalized-web-search-challenge/details/logs-format 

", "rawMarkdown": "", "votes": null }, { "id": "36061", "postDate": "12/11/2013 12:53:31", "content": "

I frequently run into the wget problem, and I am tired of relying on workarounds each time. The reason for this is that - though I *can wait*  for a disproportionate time for the 1st download, the next time round when I run my code against another machine, I need the train file again - and having to transfer the (huge) train file from one machine to another is pain !

\n

Here's what I am doing. Would appreciate if someone could point out what am I missing ? 

\n

# Log in to the server and save the cookies the traditional way- this can also be done by Chrome extension as mentioned by @William Cukierski  above.

\n

# username & pwd masked obviously

\n

wget --save-cookies cookies.txt --post-data 'user=masked_kaggle_email_address&password=masked' http://www.kaggle.com/

\n


# Grab the download page
wget --load-cookies cookies.txt \\
-p http://www.kaggle.com/c/yandex-personalized-web-search-challenge/forums/t/6060/download-data

\n

I Also tried the link https://kaggle2.blob.core.windows.net and http://www.kaggle.com/c/yandex-personalized-web-search-challenge/download/train.gz - neither of them works for downloading the data-set.
All this is saving is some js, html & png files - what am I doing wrong ?
following is the structure of the folders it is saving , and the files thereof (See the link to the saved files) 

\n

Link to directory structure fetched using wget with load-cookies 

\n

 

\n

PPS : On giving the redirect link as https://kaggle2.blob.core.windows.net , I see the following. 

\n

-- https://kaggle2.blob.core.windows.net/
Resolving kaggle2.blob.core.windows.net (kaggle2.blob.core.windows.net)... 65.52.106.46
Connecting to kaggle2.blob.core.windows.net (kaggle2.blob.core.windows.net)|65.52.106.46|:443... connected.
HTTP request sent, awaiting response... 400 Value for one of the query parameters specified in the request URI is invalid.
2013-12-11 18:18:41 ERROR 400: Value for one of the query parameters specified in the request URI is invalid..

\n

 

\n

btw, if I plainly try to access the link - https://kaggle2.blob.core.windows.net/ - it reads "This XML file does not appear to have any style information associated with it. The document tree is shown below." 

", "rawMarkdown": "", "votes": null }, { "id": "36072", "postDate": "12/11/2013 15:57:21", "content": "

 use DownloadThemAll!, a firefox extension quite similar to a download manager. It will take five to six hours to get dowlnoad.

\n

But extracting your downloaded train file will take exactly 14 to 15 minutes :)

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 32460, "author_name": "emiliogortizg", "author_url": "", "post_date": "10/20/2013 12:56:01", "content": "

I used DownThemAll!, a firefox extension quite similar to a download manager.

", "votes": null, "replies": [] }, { "id": 32465, "author_name": "mabehles", "author_url": "", "post_date": "10/20/2013 16:47:10", "content": "

Many thanks for this hint. I will give it a try

", "votes": null, "replies": [] }, { "id": 32473, "author_name": "soates", "author_url": "", "post_date": "10/21/2013 02:36:13", "content": "

Thanks so much EGO for the tip on using "DownLoadThemAll!". I am in Australia and had tried half a dozen times unsuccessfully to download the data but finally got it working with this tool.

", "votes": null, "replies": [] }, { "id": 32734, "author_name": "ashkansami", "author_url": "", "post_date": "10/27/2013 11:07:40", "content": "

Hi,

\n

I cannot download the dataset due to two problems: download managers cannot resume and I have low internet connection to your site.  Can you please provide an ftp or Torent link for data?

", "votes": null, "replies": [] }, { "id": 32745, "author_name": "emiliogortizg", "author_url": "", "post_date": "10/27/2013 19:50:30", "content": "

I am not sure who you are asking for ftp or torrent link to. In my case, considering I have small upload speed and the time I can keep my computer on, it would take more time than the rest of the competition. So, it does make sense. Hope admins can help you.

", "votes": null, "replies": [] }, { "id": 32776, "author_name": "suzanv", "author_url": "", "post_date": "10/28/2013 13:46:34", "content": "

What I was going to try, is to use wget in screen on a Linux server. But then I need the direct link to the train.gz file, and apparently this is not it: https://www.kaggle.com/c/yandex-personalized-web-search-challenge/download/train.gz Did anyone find the direct link to the file?

", "votes": null, "replies": [] }, { "id": 32777, "author_name": "wcukierski", "author_url": "", "post_date": "10/28/2013 15:00:14", "content": "

We don't have support for torrents at this time. This suggestion comes up often, but torrents are not an ideal mechanism for us for two reasons:

\n\n

You should be able to resume downloads for up to 3 days after starting them, regardless of browser. There may be combinations of browsers/managers where this doesn't work, but it should work in most cases.

\n

If you want to use a server or the command line to download a file, you must export your Kaggle cookies from your browser (this chrome extension is the easiest way) and then call wget's --load-cookies option.  Because of the rules clause above, we cannot have naked download links - you need to be logged in and have accepted the rules. After passing your Kaggle cookies, wget should work fine.  Note that the file download links will redirect to something like "https://kaggle2.blob.core.windows.net" after you've clicked the download link. This is the URL you should give to wget.

", "votes": null, "replies": [] }, { "id": 32823, "author_name": "suzanv", "author_url": "", "post_date": "10/29/2013 11:21:23", "content": "

Thanks, I succeeded in downloading the data!

", "votes": null, "replies": [] }, { "id": 33738, "author_name": "sheikhadnanahmedusmani", "author_url": "", "post_date": "11/15/2013 04:48:39", "content": "

How can one open this dataset file?

\n

Any software or do we have to write a code?

\n

I am new in this field so might ask stupid questions!

\n

Please help

", "votes": null, "replies": [] }, { "id": 33743, "author_name": "suzanv", "author_url": "", "post_date": "11/15/2013 07:52:45", "content": "

You can use gunzip on linux or 7-zip on Windows to extract the .gz files.

", "votes": null, "replies": [] }, { "id": 33755, "author_name": "sheikhadnanahmedusmani", "author_url": "", "post_date": "11/15/2013 16:30:06", "content": "

[quote=Suzan Verberne;33743]

\n

You can use gunzip on linux or 7-zip on Windows to extract the .gz files.

\n

[/quote]

\n

 

\n

I have extracted the file using Winrar. But now how can I manipulate or use this data.

\n

Kindly help.

\n

I am doing a research on click modelling.

", "votes": null, "replies": [] }, { "id": 33790, "author_name": "richardwalsh", "author_url": "", "post_date": "11/16/2013 05:00:47", "content": "

[quote=William Cukierski;32777]

\n

Note that the file download links will redirect to something like "https://kaggle2.blob.core.windows.net" after you've clicked the download link. This is the URL you should give to wget.

\n

[/quote]

\n

 

\n

Just a quick note to others in this situation - using the ".windows.net" addresses gave me 404's for some reason, but using the "http://www.kaggle.com/c/yandex-personalized-web-search-challenge/download/xyz.gz" urls from the download page did work.

", "votes": null, "replies": [] }, { "id": 34849, "author_name": "suzanv", "author_url": "", "post_date": "11/18/2013 10:00:57", "content": "

[quote=Sheikh Adnan Ahmed Usmani;33755]

\n

[quote=Suzan Verberne;33743]

\n

You can use gunzip on linux or 7-zip on Windows to extract the .gz files.

\n

[/quote]

\n

 

\n

I have extracted the file using Winrar. But now how can I manipulate or use this data.

\n

Kindly help.

\n

I am doing a research on click modelling.

\n

[/quote]

\n

 

\n

I am not sure what kind of answer you expect. Here is a description of the data: https://www.kaggle.com/c/yandex-personalized-web-search-challenge/details/logs-format 

", "votes": null, "replies": [] }, { "id": 36061, "author_name": "ektagrover1", "author_url": "", "post_date": "12/11/2013 12:53:31", "content": "

I frequently run into the wget problem, and I am tired of relying on workarounds each time. The reason for this is that - though I *can wait*  for a disproportionate time for the 1st download, the next time round when I run my code against another machine, I need the train file again - and having to transfer the (huge) train file from one machine to another is pain !

\n

Here's what I am doing. Would appreciate if someone could point out what am I missing ? 

\n

# Log in to the server and save the cookies the traditional way- this can also be done by Chrome extension as mentioned by @William Cukierski  above.

\n

# username & pwd masked obviously

\n

wget --save-cookies cookies.txt --post-data 'user=masked_kaggle_email_address&password=masked' http://www.kaggle.com/

\n


# Grab the download page
wget --load-cookies cookies.txt \\
-p http://www.kaggle.com/c/yandex-personalized-web-search-challenge/forums/t/6060/download-data

\n

I Also tried the link https://kaggle2.blob.core.windows.net and http://www.kaggle.com/c/yandex-personalized-web-search-challenge/download/train.gz - neither of them works for downloading the data-set.
All this is saving is some js, html & png files - what am I doing wrong ?
following is the structure of the folders it is saving , and the files thereof (See the link to the saved files) 

\n

Link to directory structure fetched using wget with load-cookies 

\n

 

\n

PPS : On giving the redirect link as https://kaggle2.blob.core.windows.net , I see the following. 

\n

-- https://kaggle2.blob.core.windows.net/
Resolving kaggle2.blob.core.windows.net (kaggle2.blob.core.windows.net)... 65.52.106.46
Connecting to kaggle2.blob.core.windows.net (kaggle2.blob.core.windows.net)|65.52.106.46|:443... connected.
HTTP request sent, awaiting response... 400 Value for one of the query parameters specified in the request URI is invalid.
2013-12-11 18:18:41 ERROR 400: Value for one of the query parameters specified in the request URI is invalid..

\n

 

\n

btw, if I plainly try to access the link - https://kaggle2.blob.core.windows.net/ - it reads "This XML file does not appear to have any style information associated with it. The document tree is shown below." 

", "votes": null, "replies": [] }, { "id": 36072, "author_name": "parthibangowthaman", "author_url": "", "post_date": "12/11/2013 15:57:21", "content": "

 use DownloadThemAll!, a firefox extension quite similar to a download manager. It will take five to six hours to get dowlnoad.

\n

But extracting your downloaded train file will take exactly 14 to 15 minutes :)

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "32459": "", "32460": "", "32465": "", "32473": "", "32734": "", "32745": "", "32776": "", "32777": "", "32823": "", "33738": "", "33743": "", "33755": "", "33790": "", "34849": "", "36061": "", "36072": "" }, "source": "meta" }