{
  "id": 100201,
  "title": "Setup GCP",
  "url": "/competitions/open-images-2019-object-detection/discussion/100201",
  "author_name": "",
  "post_date": "2019-07-17T06:54:45.112616900Z",
  "votes": 3,
  "comment_count": 7,
  "views": 0,
  "content": "<p>Hi everyone,</p>\n\n<p>I've not used Google Cloud Platform before and the amount of options and menus on the dashboard is quite overwhelming. Please guide or share links to some kickstart tutorials on how to setup specific to kaggle competition code on GCP.\nTo be specific I need to understand:</p>\n\n<ol>\n<li><p>How to store data in GCP such that I don't need to download training and test data everytime I run something. Also I want to store my trained models and results. (Google colab connects to drive but the space is limited and requires setting up everytime I resume after a while).</p></li>\n<li><p>Create multiple scripts/notebooks and run in batch (just like we commit on Kaggle kernels).</p></li>\n<li><p>Monitor usage (uptime and disk) to keep track of credits used on average per training/inference.</p></li>\n</ol>\n\n<p>Thanks.</p>\n\n<p>Regards,\nAcku</p>",
  "messages": [
    {
      "id": "577883",
      "postDate": "07/17/2019 06:54:45",
      "content": "<p>Hi everyone,</p>\n\n<p>I've not used Google Cloud Platform before and the amount of options and menus on the dashboard is quite overwhelming. Please guide or share links to some kickstart tutorials on how to setup specific to kaggle competition code on GCP.\nTo be specific I need to understand:</p>\n\n<ol>\n<li><p>How to store data in GCP such that I don't need to download training and test data everytime I run something. Also I want to store my trained models and results. (Google colab connects to drive but the space is limited and requires setting up everytime I resume after a while).</p></li>\n<li><p>Create multiple scripts/notebooks and run in batch (just like we commit on Kaggle kernels).</p></li>\n<li><p>Monitor usage (uptime and disk) to keep track of credits used on average per training/inference.</p></li>\n</ol>\n\n<p>Thanks.</p>\n\n<p>Regards,\nAcku</p>",
      "rawMarkdown": "Hi everyone,\n\nI've not used Google Cloud Platform before and the amount of options and menus on the dashboard is quite overwhelming. Please guide or share links to some kickstart tutorials on how to setup specific to kaggle competition code on GCP.\nTo be specific I need to understand:\n\n1. How to store data in GCP such that I don't need to download training and test data everytime I run something. Also I want to store my trained models and results. (Google colab connects to drive but the space is limited and requires setting up everytime I resume after a while).\n\n2. Create multiple scripts/notebooks and run in batch (just like we commit on Kaggle kernels).\n\n3. Monitor usage (uptime and disk) to keep track of credits used on average per training/inference.\n\nThanks.\n\nRegards,\nAcku",
      "votes": null
    },
    {
      "id": "577897",
      "postDate": "07/17/2019 07:14:26",
      "content": "<p>Hello <a href=\"/ackusingh\">@ackusingh</a> ,</p>\n\n<p>Can you please check this link , which might address your questions.</p>\n\n<p><a href=\"https://medium.com/google-cloud/how-to-run-deep-learning-models-on-google-cloud-platform-in-6-steps-4950a57acfa5\">https://medium.com/google-cloud/how-to-run-deep-learning-models-on-google-cloud-platform-in-6-steps-4950a57acfa5</a></p>\n\n<p>Do let me know if this is not what you were looking for .</p>",
      "rawMarkdown": "Hello @ackusingh ,\n\nCan you please check this link , which might address your questions.\n\nhttps://medium.com/google-cloud/how-to-run-deep-learning-models-on-google-cloud-platform-in-6-steps-4950a57acfa5\n\nDo let me know if this is not what you were looking for .",
      "votes": null
    },
    {
      "id": "578015",
      "postDate": "07/17/2019 09:31:38",
      "content": "<p>Hi <a href=\"/prmohanty\">@prmohanty</a> ,\nSorry it seems like the link is missing in your reply.</p>",
      "rawMarkdown": "Hi @prmohanty ,\nSorry it seems like the link is missing in your reply.",
      "votes": null
    },
    {
      "id": "578019",
      "postDate": "07/17/2019 09:36:42",
      "content": "<p>Hello <a href=\"/ackusingh\">@ackusingh</a> </p>\n\n<p>My bad , I have updated my previous comments with the missing link ..</p>\n\n<p>PRM</p>",
      "rawMarkdown": "Hello @ackusingh \n\nMy bad , I have updated my previous comments with the missing link ..\n\nPRM",
      "votes": null
    },
    {
      "id": "578051",
      "postDate": "07/17/2019 10:06:34",
      "content": "<p>For the object detection task, you can simply create a <em>high memory</em> machine with around 26GB or RAM and 4core CPU. To this instance, you can simply attach an additional hard-drive with around 700GB of space and can store all your training and testing files there. ( To download these files, you can use <em>awscli</em> as well as <em>kaggle cli</em> ). </p>\n\n<p>Once you are done with primary testing of all your codes, you can alter your instance to add a GPU into your instance and use it for reduced costs only after your codes have been tested. With this, you will be hardly incurr a bill of $400/month and complete all your training.</p>",
      "rawMarkdown": "For the object detection task, you can simply create a _high memory_ machine with around 26GB or RAM and 4core CPU. To this instance, you can simply attach an additional hard-drive with around 700GB of space and can store all your training and testing files there. ( To download these files, you can use _awscli_ as well as _kaggle cli_ ). \n\nOnce you are done with primary testing of all your codes, you can alter your instance to add a GPU into your instance and use it for reduced costs only after your codes have been tested. With this, you will be hardly incurr a bill of $400/month and complete all your training.",
      "votes": null
    },
    {
      "id": "578204",
      "postDate": "07/17/2019 13:16:50",
      "content": "<p><a href=\"/prmohanty\">@prmohanty</a> Thanks a lot, this is very helpful.</p>",
      "rawMarkdown": "prmohanty Thanks a lot, this is very helpful.",
      "votes": null
    },
    {
      "id": "578207",
      "postDate": "07/17/2019 13:18:09",
      "content": "<p><a href=\"/thanatoz\">@thanatoz</a> Thank you very much for the resource and budget estimate. This is a good approach to test everything first and then add GPU instances for full run.\nRegards</p>",
      "rawMarkdown": "thanatoz Thank you very much for the resource and budget estimate. This is a good approach to test everything first and then add GPU instances for full run.\nRegards",
      "votes": null
    },
    {
      "id": "578275",
      "postDate": "07/17/2019 14:41:47",
      "content": "<p>Glad that it was helpful <a href=\"/ackusingh\">@ackusingh</a> </p>",
      "rawMarkdown": "Glad that it was helpful @ackusingh",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 577897,
      "author_name": "prmohanty",
      "author_url": "",
      "post_date": "07/17/2019 07:14:26",
      "content": "<p>Hello <a href=\"/ackusingh\">@ackusingh</a> ,</p>\n\n<p>Can you please check this link , which might address your questions.</p>\n\n<p><a href=\"https://medium.com/google-cloud/how-to-run-deep-learning-models-on-google-cloud-platform-in-6-steps-4950a57acfa5\">https://medium.com/google-cloud/how-to-run-deep-learning-models-on-google-cloud-platform-in-6-steps-4950a57acfa5</a></p>\n\n<p>Do let me know if this is not what you were looking for .</p>",
      "votes": null,
      "replies": [
        {
          "id": 578015,
          "author_name": "ackusingh",
          "author_url": "",
          "post_date": "07/17/2019 09:31:38",
          "content": "<p>Hi <a href=\"/prmohanty\">@prmohanty</a> ,\nSorry it seems like the link is missing in your reply.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 578019,
          "author_name": "prmohanty",
          "author_url": "",
          "post_date": "07/17/2019 09:36:42",
          "content": "<p>Hello <a href=\"/ackusingh\">@ackusingh</a> </p>\n\n<p>My bad , I have updated my previous comments with the missing link ..</p>\n\n<p>PRM</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 578204,
          "author_name": "ackusingh",
          "author_url": "",
          "post_date": "07/17/2019 13:16:50",
          "content": "<p><a href=\"/prmohanty\">@prmohanty</a> Thanks a lot, this is very helpful.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 578275,
          "author_name": "prmohanty",
          "author_url": "",
          "post_date": "07/17/2019 14:41:47",
          "content": "<p>Glad that it was helpful <a href=\"/ackusingh\">@ackusingh</a> </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 578051,
      "author_name": "thanatoz",
      "author_url": "",
      "post_date": "07/17/2019 10:06:34",
      "content": "<p>For the object detection task, you can simply create a <em>high memory</em> machine with around 26GB or RAM and 4core CPU. To this instance, you can simply attach an additional hard-drive with around 700GB of space and can store all your training and testing files there. ( To download these files, you can use <em>awscli</em> as well as <em>kaggle cli</em> ). </p>\n\n<p>Once you are done with primary testing of all your codes, you can alter your instance to add a GPU into your instance and use it for reduced costs only after your codes have been tested. With this, you will be hardly incurr a bill of $400/month and complete all your training.</p>",
      "votes": null,
      "replies": [
        {
          "id": 578207,
          "author_name": "ackusingh",
          "author_url": "",
          "post_date": "07/17/2019 13:18:09",
          "content": "<p><a href=\"/thanatoz\">@thanatoz</a> Thank you very much for the resource and budget estimate. This is a good approach to test everything first and then add GPU instances for full run.\nRegards</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "577883": "Hi everyone,\n\nI've not used Google Cloud Platform before and the amount of options and menus on the dashboard is quite overwhelming. Please guide or share links to some kickstart tutorials on how to setup specific to kaggle competition code on GCP.\nTo be specific I need to understand:\n\n1. How to store data in GCP such that I don't need to download training and test data everytime I run something. Also I want to store my trained models and results. (Google colab connects to drive but the space is limited and requires setting up everytime I resume after a while).\n\n2. Create multiple scripts/notebooks and run in batch (just like we commit on Kaggle kernels).\n\n3. Monitor usage (uptime and disk) to keep track of credits used on average per training/inference.\n\nThanks.\n\nRegards,\nAcku",
    "577897": "Hello @ackusingh ,\n\nCan you please check this link , which might address your questions.\n\nhttps://medium.com/google-cloud/how-to-run-deep-learning-models-on-google-cloud-platform-in-6-steps-4950a57acfa5\n\nDo let me know if this is not what you were looking for .",
    "578015": "Hi @prmohanty ,\nSorry it seems like the link is missing in your reply.",
    "578019": "Hello @ackusingh \n\nMy bad , I have updated my previous comments with the missing link ..\n\nPRM",
    "578051": "For the object detection task, you can simply create a _high memory_ machine with around 26GB or RAM and 4core CPU. To this instance, you can simply attach an additional hard-drive with around 700GB of space and can store all your training and testing files there. ( To download these files, you can use _awscli_ as well as _kaggle cli_ ). \n\nOnce you are done with primary testing of all your codes, you can alter your instance to add a GPU into your instance and use it for reduced costs only after your codes have been tested. With this, you will be hardly incurr a bill of $400/month and complete all your training.",
    "578204": "prmohanty Thanks a lot, this is very helpful.",
    "578207": "thanatoz Thank you very much for the resource and budget estimate. This is a good approach to test everything first and then add GPU instances for full run.\nRegards",
    "578275": "Glad that it was helpful @ackusingh"
  },
  "source": "meta"
}