Thanks, Skylord - Each file download took less than 1 second!
\n\nEdit: maybe because something isn't quite right...files won't unzip :(
\n\nEdit2: It was a problem with my cookies file. Fixed now, and train file took a mere 85 seconds.
\n\n![\"enter](\"https://dl.dropboxusercontent.com/u/15775348/ffd.PNG\")
JohnM - This was shared by someone in an earlier competitions. Just repeating it over here. Glad you found it useful.
", "rawMarkdown": "JohnM - This was shared by someone in an earlier competitions. Just repeating it over here. Glad you found it useful.", "votes": null }, { "id": "119317", "postDate": "05/09/2016 06:54:05", "content": "You can simply use a cookie-export extension in Chrome. I use the following:\nhttps://chrome.google.com/webstore/detail/cookietxt-export/lopabhfecdfhgogdbojmaicoicjekelh
\n\nThen just load the cookies as SkyLord commented above.
", "rawMarkdown": "You can simply use a cookie-export extension in Chrome. I use the following:\r\nhttps://chrome.google.com/webstore/detail/cookietxt-export/lopabhfecdfhgogdbojmaicoicjekelh\r\n\r\nThen just load the cookies as SkyLord commented above.", "votes": null }, { "id": "119357", "postDate": "05/09/2016 14:03:12", "content": "for a firefox add-in - \nhttps://addons.mozilla.org/en-US/firefox/addon/export-cookies/
", "rawMarkdown": "for a firefox add-in - \r\nhttps://addons.mozilla.org/en-US/firefox/addon/export-cookies/", "votes": null } ], "comments": [ { "id": 117327, "author_name": "bguberfain", "author_url": "", "post_date": "04/28/2016 15:21:53", "content": "It's a little bit tricky, but you can open the Network tab on Chrome developer tools, click to download your train file (on Kaggle website) and watch for the url it uses. Than stop the download, right click the request on the Network tab and choose "Copy as cUrl". This copies the exact command you can use on a remote machine to download the file (just remember to pipe the output the a file).
\n\nI used this method to get files on a cloud computer.
", "votes": null, "replies": [] }, { "id": 117344, "author_name": "chinta", "author_url": "", "post_date": "04/28/2016 16:17:13", "content": "Thanks Bruno. Let me try .
", "votes": null, "replies": [] }, { "id": 117352, "author_name": "wallych46", "author_url": "", "post_date": "04/28/2016 17:12:34", "content": "Have you seen this?
\n\nhttp://www.louisaslett.com/RStudio_AMI/
\n\nI owe Louis money for the time he's saved me.
\n\nI start up one of these instances, and while it's installing xgboost etc, I scp the files I need over. It's quick and easy.
", "votes": null, "replies": [] }, { "id": 117491, "author_name": "chinta", "author_url": "", "post_date": "04/29/2016 11:25:40", "content": "Thanks wally. yes, I did see.
\n\nI have uploaded using filezilla. It was easy as well.
\n\nhttp://angus.readthedocs.io/en/2014/amazon/transfer-files-between-instance.html
\n\nwhen I read test set, it has only 1,875,015 rows, is it not 2,528,243 rows are expected ?
", "votes": null, "replies": [] }, { "id": 117498, "author_name": "wallych46", "author_url": "", "post_date": "04/29/2016 12:43:38", "content": "Yes, 2.5... is what I have.
", "votes": null, "replies": [] }, { "id": 117613, "author_name": "dotank", "author_url": "", "post_date": "04/29/2016 20:43:19", "content": "How is it done if you develop in python?
", "votes": null, "replies": [] }, { "id": 117659, "author_name": "chinta", "author_url": "", "post_date": "04/30/2016 02:04:37", "content": "read the document @ http://angus.readthedocs.io/en/2014/amazon/transfer-files-between-instance.html, it talks about transferring files from local PC to AWS Instance, it has nothing to do with R or python.
", "votes": null, "replies": [] }, { "id": 117734, "author_name": "mafux777", "author_url": "", "post_date": "04/30/2016 16:05:22", "content": "@Bruno - excellent recommendation from you. It took less than one minute to download the .5GB file to my instance on AWS (US-EAST). Also:\nexpedia_train <- fread("gunzip -c train.csv.gz", header=TRUE)\nRead 37670293 rows and 24 (of 24) columns from 3.791 GB file in 00:02:02
\n\n[quote=Bruno G. do Amaral;117327]
\n\nIt's a little bit tricky, but you can open the Network tab on Chrome developer tools, click to download your train file (on Kaggle website) and watch for the url it uses. Than stop the download, right click the request on the Network tab and choose "Copy as cUrl". This copies the exact command you can use on a remote machine to download the file (just remember to pipe the output the a file).
\n\nI used this method to get files on a cloud computer.
\n\n[/quote]
", "votes": null, "replies": [] }, { "id": 118302, "author_name": "bahramimaryam", "author_url": "", "post_date": "05/03/2016 07:53:18", "content": "Dear Friends\nthis form of reading data does not work on windows system. does it?
", "votes": null, "replies": [] }, { "id": 118307, "author_name": "chinta", "author_url": "", "post_date": "05/03/2016 08:15:12", "content": "@Maryam, I use windows, and I have uploaded using Filezilla, I understand you are asking about the other method, but I did not try that.
", "votes": null, "replies": [] }, { "id": 119160, "author_name": "skylord", "author_url": "", "post_date": "05/07/2016 17:49:19", "content": "If I understood the question correctly, you have downloaded the data on your local machine. And now you were trying to upload it to an AWS instance. Instead why not download it from kaggle directly to the aws instance:
\n\nwget --load-cookies=cookies.txt <url to file to download>\ncookies.txt >> has the cookie data from the kaggle website
", "votes": null, "replies": [] }, { "id": 119203, "author_name": "jpmiller", "author_url": "", "post_date": "05/08/2016 01:42:41", "content": "Thanks, Skylord - Each file download took less than 1 second!
\n\nEdit: maybe because something isn't quite right...files won't unzip :(
\n\nEdit2: It was a problem with my cookies file. Fixed now, and train file took a mere 85 seconds.
\n\n
JohnM - This was shared by someone in an earlier competitions. Just repeating it over here. Glad you found it useful.
", "votes": null, "replies": [] }, { "id": 119317, "author_name": "alirezaa", "author_url": "", "post_date": "05/09/2016 06:54:05", "content": "You can simply use a cookie-export extension in Chrome. I use the following:\nhttps://chrome.google.com/webstore/detail/cookietxt-export/lopabhfecdfhgogdbojmaicoicjekelh
\n\nThen just load the cookies as SkyLord commented above.
", "votes": null, "replies": [] }, { "id": 119357, "author_name": "skylord", "author_url": "", "post_date": "05/09/2016 14:03:12", "content": "for a firefox add-in - \nhttps://addons.mozilla.org/en-US/firefox/addon/export-cookies/
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "117318": "Hello,\r\n\r\nI have set up R-Studio on AWS and have been able to upload destinations.csv file, using the option to upload file in R studio, it uploaded fairly quickly. However train.csv upload is taking forever.\r\n\r\nCan you please let me know, if there is a quicker way to upload files.\r\n\r\nThanks,", "117327": "It's a little bit tricky, but you can open the Network tab on Chrome developer tools, click to download your train file (on Kaggle website) and watch for the url it uses. Than stop the download, right click the request on the Network tab and choose \"Copy as cUrl\". This copies the exact command you can use on a remote machine to download the file (just remember to pipe the output the a file).\r\n\r\nI used this method to get files on a cloud computer.", "117344": "Thanks Bruno. Let me try .", "117352": "Have you seen this?\r\n\r\nhttp://www.louisaslett.com/RStudio_AMI/\r\n\r\nI owe Louis money for the time he's saved me.\r\n\r\nI start up one of these instances, and while it's installing xgboost etc, I scp the files I need over. It's quick and easy.", "117491": "Thanks wally. yes, I did see.\r\n\r\nI have uploaded using filezilla. It was easy as well. \r\n\r\nhttp://angus.readthedocs.io/en/2014/amazon/transfer-files-between-instance.html\r\n\r\nwhen I read test set, it has only 1,875,015 rows, is it not 2,528,243 rows are expected ?", "117498": "Yes, 2.5... is what I have.", "117613": "How is it done if you develop in python?", "117659": "read the document @ http://angus.readthedocs.io/en/2014/amazon/transfer-files-between-instance.html, it talks about transferring files from local PC to AWS Instance, it has nothing to do with R or python.", "117734": "Bruno - excellent recommendation from you. It took less than one minute to download the .5GB file to my instance on AWS (US-EAST). Also:\r\nexpedia_train <- fread(\"gunzip -c train.csv.gz\", header=TRUE)\r\nRead 37670293 rows and 24 (of 24) columns from 3.791 GB file in 00:02:02\r\n\r\n[quote=Bruno G. do Amaral;117327]\r\n\r\nIt's a little bit tricky, but you can open the Network tab on Chrome developer tools, click to download your train file (on Kaggle website) and watch for the url it uses. Than stop the download, right click the request on the Network tab and choose \"Copy as cUrl\". This copies the exact command you can use on a remote machine to download the file (just remember to pipe the output the a file).\r\n\r\nI used this method to get files on a cloud computer.\r\n\r\n[/quote]", "118302": "Dear Friends\r\nthis form of reading data does not work on windows system. does it?", "118307": "Maryam, I use windows, and I have uploaded using Filezilla, I understand you are asking about the other method, but I did not try that.", "119160": "If I understood the question correctly, you have downloaded the data on your local machine. And now you were trying to upload it to an AWS instance. Instead why not download it from kaggle directly to the aws instance: \r\n\r\n wget --load-cookies=cookies.txt