Hello all, looking at the size of this data, it is surely gonna overwhelm my personal system. Will google compute engine or AWS be able to manage this huge data. Please suggest a viable way to participate in this competition. Any kind of details would be very helpful.
", "messages": [ { "id": "207485", "postDate": "07/26/2017 18:25:12", "content": "Hello all, looking at the size of this data, it is surely gonna overwhelm my personal system. Will google compute engine or AWS be able to manage this huge data. Please suggest a viable way to participate in this competition. Any kind of details would be very helpful.
", "rawMarkdown": "Hello all, looking at the size of this data, it is surely gonna overwhelm my personal system. Will google compute engine or AWS be able to manage this huge data. Please suggest a viable way to participate in this competition. Any kind of details would be very helpful.", "votes": null }, { "id": "207486", "postDate": "07/26/2017 18:27:45", "content": "Definitely. I use AWS instances (out of habit, not arguing for one over the other) and it can handle this much data. Both have great GPU instances for processing. Just depends on your budget :)
", "rawMarkdown": "Definitely. I use AWS instances (out of habit, not arguing for one over the other) and it can handle this much data. Both have great GPU instances for processing. Just depends on your budget :)", "votes": null }, { "id": "207501", "postDate": "07/26/2017 18:52:49", "content": "I am currently using p2 instance and it costs me $.90 per hour but it has memory of only 60 GB, which cannot even handle the size of compressed tar.gz file :/
", "rawMarkdown": "I am currently using p2 instance and it costs me $.90 per hour but it has memory of only 60 GB, which cannot even handle the size of compressed tar.gz file :/", "votes": null }, { "id": "207506", "postDate": "07/26/2017 19:09:27", "content": "Do you mean 60Gb of disk space or RAM? If you meant disk space then note, that you can enlarge it in control panel.
", "rawMarkdown": "Do you mean 60Gb of disk space or RAM? If you meant disk space then note, that you can enlarge it in control panel.", "votes": null }, { "id": "207514", "postDate": "07/26/2017 19:39:43", "content": "It's actually the RAM, the AWS instances having higher RAMs are too costly!
", "rawMarkdown": "It's actually the RAM, the AWS instances having higher RAMs are too costly!", "votes": null }, { "id": "207535", "postDate": "07/26/2017 20:50:33", "content": "See my setup here: https://www.kaggle.com/c/imagenet-object-detection-challenge/discussion/37068
", "rawMarkdown": "See my setup here: https://www.kaggle.com/c/imagenet-object-detection-challenge/discussion/37068", "votes": null }, { "id": "207536", "postDate": "07/26/2017 20:50:55", "content": "Why do you need so much RAM?
", "rawMarkdown": "Why do you need so much RAM?", "votes": null }, { "id": "208189", "postDate": "07/28/2017 23:25:17", "content": "You don't need so much RAM, you should split your data into batches, which fit into RAM of the GPU, which is 12Gb for K80
", "rawMarkdown": "You don't need so much RAM, you should split your data into batches, which fit into RAM of the GPU, which is 12Gb for K80", "votes": null } ], "comments": [ { "id": 207486, "author_name": "theamazingdrew", "author_url": "", "post_date": "07/26/2017 18:27:45", "content": "Definitely. I use AWS instances (out of habit, not arguing for one over the other) and it can handle this much data. Both have great GPU instances for processing. Just depends on your budget :)
", "votes": null, "replies": [ { "id": 207501, "author_name": "janpreets", "author_url": "", "post_date": "07/26/2017 18:52:49", "content": "I am currently using p2 instance and it costs me $.90 per hour but it has memory of only 60 GB, which cannot even handle the size of compressed tar.gz file :/
", "votes": null, "replies": [] }, { "id": 207506, "author_name": "dimskraft", "author_url": "", "post_date": "07/26/2017 19:09:27", "content": "Do you mean 60Gb of disk space or RAM? If you meant disk space then note, that you can enlarge it in control panel.
", "votes": null, "replies": [] }, { "id": 207514, "author_name": "janpreets", "author_url": "", "post_date": "07/26/2017 19:39:43", "content": "It's actually the RAM, the AWS instances having higher RAMs are too costly!
", "votes": null, "replies": [] }, { "id": 207536, "author_name": "solomonk", "author_url": "", "post_date": "07/26/2017 20:50:55", "content": "Why do you need so much RAM?
", "votes": null, "replies": [] }, { "id": 208189, "author_name": "dimskraft", "author_url": "", "post_date": "07/28/2017 23:25:17", "content": "You don't need so much RAM, you should split your data into batches, which fit into RAM of the GPU, which is 12Gb for K80
", "votes": null, "replies": [] } ] }, { "id": 207535, "author_name": "solomonk", "author_url": "", "post_date": "07/26/2017 20:50:33", "content": "See my setup here: https://www.kaggle.com/c/imagenet-object-detection-challenge/discussion/37068
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "207485": "Hello all, looking at the size of this data, it is surely gonna overwhelm my personal system. Will google compute engine or AWS be able to manage this huge data. Please suggest a viable way to participate in this competition. Any kind of details would be very helpful.", "207486": "Definitely. I use AWS instances (out of habit, not arguing for one over the other) and it can handle this much data. Both have great GPU instances for processing. Just depends on your budget :)", "207501": "I am currently using p2 instance and it costs me $.90 per hour but it has memory of only 60 GB, which cannot even handle the size of compressed tar.gz file :/", "207506": "Do you mean 60Gb of disk space or RAM? If you meant disk space then note, that you can enlarge it in control panel.", "207514": "It's actually the RAM, the AWS instances having higher RAMs are too costly!", "207535": "See my setup here: https://www.kaggle.com/c/imagenet-object-detection-challenge/discussion/37068", "207536": "Why do you need so much RAM?", "208189": "You don't need so much RAM, you should split your data into batches, which fit into RAM of the GPU, which is 12Gb for K80" }, "source": "meta" }