I am new to google cloud, so I have a newbie question: I see that some people are running python scripts on their local machine. The data is 3TB and I have it in my own bucket, so is their a method to access these files from my local machine? Or do I need to use cloud tools to do all the work?
", "messages": [ { "id": "250116", "postDate": "11/29/2017 20:27:50", "content": "I am new to google cloud, so I have a newbie question: I see that some people are running python scripts on their local machine. The data is 3TB and I have it in my own bucket, so is their a method to access these files from my local machine? Or do I need to use cloud tools to do all the work?
", "rawMarkdown": "I am new to google cloud, so I have a newbie question: I see that some people are running python scripts on their local machine. The data is 3TB and I have it in my own bucket, so is their a method to access these files from my local machine? Or do I need to use cloud tools to do all the work?", "votes": null }, { "id": "250177", "postDate": "11/29/2017 22:27:42", "content": "You can access your Google Cloud Storage Bucket files directly from your scripts on your local computer. gsutil (https://cloud.google.com/storage/docs/gsutil) and tensorflow.python.lib.io.file_io (https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/lib/io/file_io.py) are two tools you could use. gsutil is actually a commandline tool, to use it from within python you need to use the subprocess module (https://docs.python.org/2/library/subprocess.html). file_io is pretty well hidden, but is part of tensorflow.
\n\nHowever, you will have delays, files don't come fast down through the internet so if you want to train your whole model locally you probably will want to download the files first, also possible with gsutil as well as the webinterface of GCP.
\n\nFinally, it is also possible to do it the other way around, load your code to the cloud and have them run it on the data, then you just download the results and the log files.
\n\nHope that helps, happy coding!\nBastiaan.
", "rawMarkdown": "You can access your Google Cloud Storage Bucket files directly from your scripts on your local computer. gsutil (https://cloud.google.com/storage/docs/gsutil) and tensorflow.python.lib.io.file_io (https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/lib/io/file_io.py) are two tools you could use. gsutil is actually a commandline tool, to use it from within python you need to use the subprocess module (https://docs.python.org/2/library/subprocess.html). file_io is pretty well hidden, but *is* part of tensorflow.\n\nHowever, you will have delays, files don't come fast down through the internet so if you want to train your whole model locally you probably will want to download the files first, also possible with gsutil as well as the webinterface of GCP.\n\nFinally, it is also possible to do it the other way around, load your code to the cloud and have them run it on the data, then you just download the results and the log files.\n\nHope that helps, happy coding!\nBastiaan.", "votes": null } ], "comments": [ { "id": 250177, "author_name": "bastiaanbergman", "author_url": "", "post_date": "11/29/2017 22:27:42", "content": "You can access your Google Cloud Storage Bucket files directly from your scripts on your local computer. gsutil (https://cloud.google.com/storage/docs/gsutil) and tensorflow.python.lib.io.file_io (https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/lib/io/file_io.py) are two tools you could use. gsutil is actually a commandline tool, to use it from within python you need to use the subprocess module (https://docs.python.org/2/library/subprocess.html). file_io is pretty well hidden, but is part of tensorflow.
\n\nHowever, you will have delays, files don't come fast down through the internet so if you want to train your whole model locally you probably will want to download the files first, also possible with gsutil as well as the webinterface of GCP.
\n\nFinally, it is also possible to do it the other way around, load your code to the cloud and have them run it on the data, then you just download the results and the log files.
\n\nHope that helps, happy coding!\nBastiaan.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "250116": "I am new to google cloud, so I have a newbie question: I see that some people are running python scripts on their local machine. The data is 3TB and I have it in my own bucket, so is their a method to access these files from my local machine? Or do I need to use cloud tools to do all the work?", "250177": "You can access your Google Cloud Storage Bucket files directly from your scripts on your local computer. gsutil (https://cloud.google.com/storage/docs/gsutil) and tensorflow.python.lib.io.file_io (https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/lib/io/file_io.py) are two tools you could use. gsutil is actually a commandline tool, to use it from within python you need to use the subprocess module (https://docs.python.org/2/library/subprocess.html). file_io is pretty well hidden, but *is* part of tensorflow.\n\nHowever, you will have delays, files don't come fast down through the internet so if you want to train your whole model locally you probably will want to download the files first, also possible with gsutil as well as the webinterface of GCP.\n\nFinally, it is also possible to do it the other way around, load your code to the cloud and have them run it on the data, then you just download the results and the log files.\n\nHope that helps, happy coding!\nBastiaan." }, "source": "meta" }