{
  "id": 98851,
  "title": "Tricks for efficiently loading data on Google Cloud instances?",
  "url": "/competitions/recursion-cellular-image-classification/discussion/98851",
  "author_name": "",
  "post_date": "2019-07-07T02:05:10.954212300Z",
  "votes": null,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I'm starting to think I should use something more heavy-duty than Kernels/Colab for this competition. If I'm spinning up pre-emptible VMs on Google Cloud, what's the best way to efficiently get the data onto the machine? Should I just run a download command right after I've provisioned the machina? Or should I make the extra effort to download images as needed and then cache them locally? </p>",
  "messages": [
    {
      "id": "569593",
      "postDate": "07/07/2019 02:05:10",
      "content": "<p>I'm starting to think I should use something more heavy-duty than Kernels/Colab for this competition. If I'm spinning up pre-emptible VMs on Google Cloud, what's the best way to efficiently get the data onto the machine? Should I just run a download command right after I've provisioned the machina? Or should I make the extra effort to download images as needed and then cache them locally? </p>",
      "rawMarkdown": "I'm starting to think I should use something more heavy-duty than Kernels/Colab for this competition. If I'm spinning up pre-emptible VMs on Google Cloud, what's the best way to efficiently get the data onto the machine? Should I just run a download command right after I've provisioned the machina? Or should I make the extra effort to download images as needed and then cache them locally?",
      "votes": null
    },
    {
      "id": "569689",
      "postDate": "07/07/2019 06:58:31",
      "content": "<p>On AWS, you can download data to a disk which would be persisted after machine shutdown, and can be re-attached, looks like something similar is possible on GCP as well: <a href=\"https://cloud.google.com/compute/docs/disks/add-persistent-disk\">https://cloud.google.com/compute/docs/disks/add-persistent-disk</a></p>",
      "rawMarkdown": "On AWS, you can download data to a disk which would be persisted after machine shutdown, and can be re-attached, looks like something similar is possible on GCP as well: https://cloud.google.com/compute/docs/disks/add-persistent-disk",
      "votes": null
    },
    {
      "id": "569854",
      "postDate": "07/07/2019 12:57:38",
      "content": "<p>Thanks! I'm figuring this out as I got, and that really helps.</p>",
      "rawMarkdown": "Thanks! I'm figuring this out as I got, and that really helps.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 569689,
      "author_name": "lopuhin",
      "author_url": "",
      "post_date": "07/07/2019 06:58:31",
      "content": "<p>On AWS, you can download data to a disk which would be persisted after machine shutdown, and can be re-attached, looks like something similar is possible on GCP as well: <a href=\"https://cloud.google.com/compute/docs/disks/add-persistent-disk\">https://cloud.google.com/compute/docs/disks/add-persistent-disk</a></p>",
      "votes": null,
      "replies": [
        {
          "id": 569854,
          "author_name": "mrosett",
          "author_url": "",
          "post_date": "07/07/2019 12:57:38",
          "content": "<p>Thanks! I'm figuring this out as I got, and that really helps.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "569593": "I'm starting to think I should use something more heavy-duty than Kernels/Colab for this competition. If I'm spinning up pre-emptible VMs on Google Cloud, what's the best way to efficiently get the data onto the machine? Should I just run a download command right after I've provisioned the machina? Or should I make the extra effort to download images as needed and then cache them locally?",
    "569689": "On AWS, you can download data to a disk which would be persisted after machine shutdown, and can be re-attached, looks like something similar is possible on GCP as well: https://cloud.google.com/compute/docs/disks/add-persistent-disk",
    "569854": "Thanks! I'm figuring this out as I got, and that really helps."
  },
  "source": "meta"
}