Hello I am excited to begin this competition but I do not have the hard drive space for 3 TB so I'll have to do my analysis off the cloud. I am not familiar with working with Cloud Buckets so could anyone give me some advice or point me in the right direction to find how to access this info for analysis from R?
\n\nThanks in advance\n-Velma
", "messages": [ { "id": "196499", "postDate": "06/27/2017 13:59:32", "content": "Hello I am excited to begin this competition but I do not have the hard drive space for 3 TB so I'll have to do my analysis off the cloud. I am not familiar with working with Cloud Buckets so could anyone give me some advice or point me in the right direction to find how to access this info for analysis from R?
\n\nThanks in advance\n-Velma
", "rawMarkdown": "Hello I am excited to begin this competition but I do not have the hard drive space for 3 TB so I'll have to do my analysis off the cloud. I am not familiar with working with Cloud Buckets so could anyone give me some advice or point me in the right direction to find how to access this info for analysis from R?\n\nThanks in advance\n-Velma", "votes": null }, { "id": "196666", "postDate": "06/27/2017 22:15:25", "content": "Good question! I'm not familiar with Google's cloud offerings (only accustomed to AWS), but I know I'll need to learn the ropes on GC soon enough. Do you have any interest in collaborating on a \"How To Get Started\" primer for this competition?
", "rawMarkdown": "Good question! I'm not familiar with Google's cloud offerings (only accustomed to AWS), but I know I'll need to learn the ropes on GC soon enough. Do you have any interest in collaborating on a \"How To Get Started\" primer for this competition?", "votes": null }, { "id": "196697", "postDate": "06/28/2017 00:09:16", "content": "You don't have at least a 3TB hdd?? Whaaat? Quit living in the stone age! Just kidding. Yea, 3TB is a little big, huh? I think there's an R Kernel you may find useful but I have only glanced. I'll update you once I get access to the Cloud Bucket.
", "rawMarkdown": "You don't have at least a 3TB hdd?? Whaaat? Quit living in the stone age! Just kidding. Yea, 3TB is a little big, huh? I think there's an R Kernel you may find useful but I have only glanced. I'll update you once I get access to the Cloud Bucket.", "votes": null }, { "id": "196864", "postDate": "06/28/2017 09:17:17", "content": "I would certainly be interested in collaborating on that primer, but I'm also interested in understanding all the options available for tackling this problem. Have you been using AWS for this competition so far? How pricey/convenient has it been for you so far?
", "rawMarkdown": "I would certainly be interested in collaborating on that primer, but I'm also interested in understanding all the options available for tackling this problem. Have you been using AWS for this competition so far? How pricey/convenient has it been for you so far?", "votes": null }, { "id": "197039", "postDate": "06/28/2017 16:55:21", "content": "Mmm, gotcha'. I am not yet at the point where I feel like my model needs raw horsepower, so I've been tooling around locally on a laptop and passing off work to an AWS EC2 m3 spot-instance when I need to be on the go. I don't think I've crossed $5 in spending yet.
\n\nn.b. I'm working with the smallest file format for simplicity and speed, while I try to devise a strategy (e.g. divide image into areas of body then look for threat; alternatively, look for threat then identify area).
\n\nThus, I host a Jupyter (Python) Notebook whenever I can't leave the laptop to chug for a prolonged period. With that tier of instance, I'm entirely reliant on the CPU and training on the whole set of small-format images requires overnight processing. My thought is that I could easily spin up a p2 instance with a GPU attached and really start flying through training when I'm ready to tackle the problem in earnest.
\n\nTheoretically, I could transfer the large-format (3TB) dataset from the GC bucket to S3 or similar, but I expect that would incur a large fee for the transfer alone, never mind the storage over time and computing expense. Similarly, pulling data from GC to an EC2 instance probably involves some transfer costs, so I'm inclined to learn Google's Compute system, if not for the cost savings then for the convenience and novelty alone. I'll look into the pricing differences for comparable high-end hardware (GPU-accelerated) later this evening and perhaps do a write-up in a new topic.
", "rawMarkdown": "Mmm, gotcha'. I am not yet at the point where I feel like my model needs raw horsepower, so I've been tooling around locally on a laptop and passing off work to an [AWS EC2 m3 spot-instance][1] when I need to be on the go. I don't think I've crossed $5 in spending yet.\n\n*n.b. I'm working with the smallest file format for simplicity and speed, while I try to devise a strategy (e.g. divide image into areas of body then look for threat; alternatively, look for threat then identify area).*\n\nThus, I host a Jupyter (Python) Notebook whenever I can't leave the laptop to chug for a prolonged period. With that tier of instance, I'm entirely reliant on the CPU and training on the whole set of small-format images requires overnight processing. My thought is that I could easily spin up a p2 instance with a GPU attached and really start flying through training when I'm ready to tackle the problem in earnest.\n\nTheoretically, I could transfer the large-format (3TB) dataset from the GC bucket to S3 or similar, but I expect that would incur a large fee for the transfer alone, never mind the storage over time and computing expense. Similarly, pulling data from GC to an EC2 instance probably involves some transfer costs, so I'm inclined to learn Google's Compute system, if not for the cost savings then for the convenience and novelty alone. I'll look into the pricing differences for comparable high-end hardware (GPU-accelerated) later this evening and perhaps do a write-up in a new topic.\n\n [1]: https://aws.amazon.com/ec2/instance-types/", "votes": null }, { "id": "197294", "postDate": "06/29/2017 08:04:30", "content": "Thanks for the quick sum-up - I'm intrigued by \"passing off\" computation as I've always used my own CPU/GPU for analytics. I would definitely be interested in reading that write-up should you have time to post one, but either way I'm looking forward to learning some new techniques through this competition!
", "rawMarkdown": "Thanks for the quick sum-up - I'm intrigued by \"passing off\" computation as I've always used my own CPU/GPU for analytics. I would definitely be interested in reading that write-up should you have time to post one, but either way I'm looking forward to learning some new techniques through this competition!", "votes": null }, { "id": "197543", "postDate": "06/29/2017 19:27:04", "content": "Hi Velma - The data is in a Google Cloud Storage bucket. You can find out more about Cloud Storage and see samples for accessing data in many programming languages: https://cloud.google.com/storage/docs.
\n\nHowever, unfortunately there isn't a sample on that site for interacting with Cloud Storage from R.
\n\nHere are two blog posts from Google about R:
\n\n\n\nIf you search for \"Google Cloud Storage R\", there are a couple of third party libraries for accessing Google Cloud Storage buckets from R. These are the top two results:
\n\nLet me know if that helps.
", "rawMarkdown": "Hi Velma - The data is in a Google Cloud Storage bucket. You can find out more about Cloud Storage and see samples for accessing data in many programming languages: https://cloud.google.com/storage/docs. \n\nHowever, unfortunately there isn't a sample on that site for interacting with Cloud Storage from R. \n\nHere are two blog posts from Google about R: \n\n - [Using R with Google BigQuery](https://cloud.google.com/blog/big-data/2017/03/google-cloud-platform-for-data-scientists-using-r-with-google-cloud-sql-for-mysql)\n - [Using R with Google Cloud SQL](https://cloud.google.com/blog/big-data/2017/03/google-cloud-platform-for-data-scientists-using-r-with-google-cloud-sql-for-mysql)\n\nIf you search for \"Google Cloud Storage R\", there are a couple of third party libraries for accessing Google Cloud Storage buckets from R. These are the top two results: \n\n - https://github.com/cloudyr/googleCloudStorageR\n - https://cran.r-project.org/web/packages/googleCloudStorageR/vignettes/googleCloudStorageR.html\n\nLet me know if that helps.", "votes": null } ], "comments": [ { "id": 196666, "author_name": "jangerhofer", "author_url": "", "post_date": "06/27/2017 22:15:25", "content": "Good question! I'm not familiar with Google's cloud offerings (only accustomed to AWS), but I know I'll need to learn the ropes on GC soon enough. Do you have any interest in collaborating on a \"How To Get Started\" primer for this competition?
", "votes": null, "replies": [ { "id": 196864, "author_name": "trapvelma", "author_url": "", "post_date": "06/28/2017 09:17:17", "content": "I would certainly be interested in collaborating on that primer, but I'm also interested in understanding all the options available for tackling this problem. Have you been using AWS for this competition so far? How pricey/convenient has it been for you so far?
", "votes": null, "replies": [] }, { "id": 197039, "author_name": "jangerhofer", "author_url": "", "post_date": "06/28/2017 16:55:21", "content": "Mmm, gotcha'. I am not yet at the point where I feel like my model needs raw horsepower, so I've been tooling around locally on a laptop and passing off work to an AWS EC2 m3 spot-instance when I need to be on the go. I don't think I've crossed $5 in spending yet.
\n\nn.b. I'm working with the smallest file format for simplicity and speed, while I try to devise a strategy (e.g. divide image into areas of body then look for threat; alternatively, look for threat then identify area).
\n\nThus, I host a Jupyter (Python) Notebook whenever I can't leave the laptop to chug for a prolonged period. With that tier of instance, I'm entirely reliant on the CPU and training on the whole set of small-format images requires overnight processing. My thought is that I could easily spin up a p2 instance with a GPU attached and really start flying through training when I'm ready to tackle the problem in earnest.
\n\nTheoretically, I could transfer the large-format (3TB) dataset from the GC bucket to S3 or similar, but I expect that would incur a large fee for the transfer alone, never mind the storage over time and computing expense. Similarly, pulling data from GC to an EC2 instance probably involves some transfer costs, so I'm inclined to learn Google's Compute system, if not for the cost savings then for the convenience and novelty alone. I'll look into the pricing differences for comparable high-end hardware (GPU-accelerated) later this evening and perhaps do a write-up in a new topic.
", "votes": null, "replies": [] }, { "id": 197294, "author_name": "trapvelma", "author_url": "", "post_date": "06/29/2017 08:04:30", "content": "Thanks for the quick sum-up - I'm intrigued by \"passing off\" computation as I've always used my own CPU/GPU for analytics. I would definitely be interested in reading that write-up should you have time to post one, but either way I'm looking forward to learning some new techniques through this competition!
", "votes": null, "replies": [] } ] }, { "id": 196697, "author_name": "millerintllc", "author_url": "", "post_date": "06/28/2017 00:09:16", "content": "You don't have at least a 3TB hdd?? Whaaat? Quit living in the stone age! Just kidding. Yea, 3TB is a little big, huh? I think there's an R Kernel you may find useful but I have only glanced. I'll update you once I get access to the Cloud Bucket.
", "votes": null, "replies": [] }, { "id": 197543, "author_name": "stevegreenberg", "author_url": "", "post_date": "06/29/2017 19:27:04", "content": "Hi Velma - The data is in a Google Cloud Storage bucket. You can find out more about Cloud Storage and see samples for accessing data in many programming languages: https://cloud.google.com/storage/docs.
\n\nHowever, unfortunately there isn't a sample on that site for interacting with Cloud Storage from R.
\n\nHere are two blog posts from Google about R:
\n\n\n\nIf you search for \"Google Cloud Storage R\", there are a couple of third party libraries for accessing Google Cloud Storage buckets from R. These are the top two results:
\n\nLet me know if that helps.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "196499": "Hello I am excited to begin this competition but I do not have the hard drive space for 3 TB so I'll have to do my analysis off the cloud. I am not familiar with working with Cloud Buckets so could anyone give me some advice or point me in the right direction to find how to access this info for analysis from R?\n\nThanks in advance\n-Velma", "196666": "Good question! I'm not familiar with Google's cloud offerings (only accustomed to AWS), but I know I'll need to learn the ropes on GC soon enough. Do you have any interest in collaborating on a \"How To Get Started\" primer for this competition?", "196697": "You don't have at least a 3TB hdd?? Whaaat? Quit living in the stone age! Just kidding. Yea, 3TB is a little big, huh? I think there's an R Kernel you may find useful but I have only glanced. I'll update you once I get access to the Cloud Bucket.", "196864": "I would certainly be interested in collaborating on that primer, but I'm also interested in understanding all the options available for tackling this problem. Have you been using AWS for this competition so far? How pricey/convenient has it been for you so far?", "197039": "Mmm, gotcha'. I am not yet at the point where I feel like my model needs raw horsepower, so I've been tooling around locally on a laptop and passing off work to an [AWS EC2 m3 spot-instance][1] when I need to be on the go. I don't think I've crossed $5 in spending yet.\n\n*n.b. I'm working with the smallest file format for simplicity and speed, while I try to devise a strategy (e.g. divide image into areas of body then look for threat; alternatively, look for threat then identify area).*\n\nThus, I host a Jupyter (Python) Notebook whenever I can't leave the laptop to chug for a prolonged period. With that tier of instance, I'm entirely reliant on the CPU and training on the whole set of small-format images requires overnight processing. My thought is that I could easily spin up a p2 instance with a GPU attached and really start flying through training when I'm ready to tackle the problem in earnest.\n\nTheoretically, I could transfer the large-format (3TB) dataset from the GC bucket to S3 or similar, but I expect that would incur a large fee for the transfer alone, never mind the storage over time and computing expense. Similarly, pulling data from GC to an EC2 instance probably involves some transfer costs, so I'm inclined to learn Google's Compute system, if not for the cost savings then for the convenience and novelty alone. I'll look into the pricing differences for comparable high-end hardware (GPU-accelerated) later this evening and perhaps do a write-up in a new topic.\n\n [1]: https://aws.amazon.com/ec2/instance-types/", "197294": "Thanks for the quick sum-up - I'm intrigued by \"passing off\" computation as I've always used my own CPU/GPU for analytics. I would definitely be interested in reading that write-up should you have time to post one, but either way I'm looking forward to learning some new techniques through this competition!", "197543": "Hi Velma - The data is in a Google Cloud Storage bucket. You can find out more about Cloud Storage and see samples for accessing data in many programming languages: https://cloud.google.com/storage/docs. \n\nHowever, unfortunately there isn't a sample on that site for interacting with Cloud Storage from R. \n\nHere are two blog posts from Google about R: \n\n - [Using R with Google BigQuery](https://cloud.google.com/blog/big-data/2017/03/google-cloud-platform-for-data-scientists-using-r-with-google-cloud-sql-for-mysql)\n - [Using R with Google Cloud SQL](https://cloud.google.com/blog/big-data/2017/03/google-cloud-platform-for-data-scientists-using-r-with-google-cloud-sql-for-mysql)\n\nIf you search for \"Google Cloud Storage R\", there are a couple of third party libraries for accessing Google Cloud Storage buckets from R. These are the top two results: \n\n - https://github.com/cloudyr/googleCloudStorageR\n - https://cran.r-project.org/web/packages/googleCloudStorageR/vignettes/googleCloudStorageR.html\n\nLet me know if that helps." }, "source": "meta" }