{
  "id": 66197,
  "title": "Is there a connection free way of training your model on GCP?",
  "url": "/competitions/rsna-pneumonia-detection-challenge/discussion/66197",
  "author_name": "",
  "post_date": "2018-09-19T05:54:28.608015200Z",
  "votes": 2,
  "comment_count": 6,
  "views": 0,
  "content": "<p>Due to frequent power cuts the ssh connection with the GCP instance is lost and the hours of training  is wasted. One option is to save weights every epoch but is there a way in which we can train our model like we do in Kaggle Kernels, they keep running in the cloud even when the PC is switched off?</p>",
  "messages": [
    {
      "id": "389721",
      "postDate": "09/19/2018 05:54:28",
      "content": "<p>Due to frequent power cuts the ssh connection with the GCP instance is lost and the hours of training  is wasted. One option is to save weights every epoch but is there a way in which we can train our model like we do in Kaggle Kernels, they keep running in the cloud even when the PC is switched off?</p>",
      "rawMarkdown": "Due to frequent power cuts the ssh connection with the GCP instance is lost and the hours of training  is wasted. One option is to save weights every epoch but is there a way in which we can train our model like we do in Kaggle Kernels, they keep running in the cloud even when the PC is switched off?",
      "votes": null
    },
    {
      "id": "389724",
      "postDate": "09/19/2018 06:03:08",
      "content": "<p>If you are using shell and not jupyter, tmux works really well</p>",
      "rawMarkdown": "If you are using shell and not jupyter, tmux works really well",
      "votes": null
    },
    {
      "id": "389738",
      "postDate": "09/19/2018 06:30:48",
      "content": "<p>Thanks Moshel, I do use Jupyter but I think if tmux works fine I would switch over to using shell.  But I would prefer something that works with Jupyter. </p>",
      "rawMarkdown": "Thanks Moshel, I do use Jupyter but I think if tmux works fine I would switch over to using shell.  But I would prefer something that works with Jupyter.",
      "votes": null
    },
    {
      "id": "390047",
      "postDate": "09/19/2018 16:13:24",
      "content": "<p>There are two options I know:</p>\n\n<p>Datalab: <a href=\"https://cloud.google.com/datalab/\">https://cloud.google.com/datalab/</a> where you get a Jupyter with custom look and feel running in the cloud as long as you wish.\nMl-engine: <a href=\"https://cloud.google.com/ml-engine/\">https://cloud.google.com/ml-engine/</a> which would be a non-interactive way of working: You submit your code as a python package for execution and it must be a fully automated procedure then. </p>",
      "rawMarkdown": "There are two options I know:\n\n Datalab: https://cloud.google.com/datalab/ where you get a Jupyter with custom look and feel running in the cloud as long as you wish.\nMl-engine: https://cloud.google.com/ml-engine/ which would be a non-interactive way of working: You submit your code as a python package for execution and it must be a fully automated procedure then.",
      "votes": null
    },
    {
      "id": "390070",
      "postDate": "09/19/2018 16:59:13",
      "content": "<p>Thanks Tomasz for your reply. I am currently exploring Datalab(well found out about it a few hours back) and ml-engine sounds like something that would match my requirement better </p>",
      "rawMarkdown": "Thanks Tomasz for your reply. I am currently exploring Datalab(well found out about it a few hours back) and ml-engine sounds like something that would match my requirement better",
      "votes": null
    },
    {
      "id": "390589",
      "postDate": "09/20/2018 13:30:46",
      "content": "<p>have you tried using \"nohup\" ?\nlike: </p>\n\n<p>nohup 2&gt;&amp;1 command &gt; yourlog.log &amp;</p>\n\n<p>to capture the possible error  msgs too...</p>",
      "rawMarkdown": "have you tried using \"nohup\" ?\nlike: \n\nnohup 2&gt;&amp;1 command &gt; yourlog.log &amp;\n\nto capture the possible error  msgs too...",
      "votes": null
    },
    {
      "id": "390643",
      "postDate": "09/20/2018 15:18:27",
      "content": "<p>\"nohup\" is exactly what I wanted. Just tested it and it works perfectly. Thanks Henrique.</p>\n\n<p>P.S. I found more information <a href=\"https://www.digitalocean.com/community/tutorials/how-to-use-bash-s-job-control-to-manage-foreground-and-background-processes\">here</a> and <a href=\"https://unix.stackexchange.com/questions/343569/how-do-i-use-nohup-and-tail-it-in-one-command/344659\">here</a></p>",
      "rawMarkdown": "\"nohup\" is exactly what I wanted. Just tested it and it works perfectly. Thanks Henrique.\n\nP.S. I found more information [here][1] and [here][2]\n\n\n  [1]: https://www.digitalocean.com/community/tutorials/how-to-use-bash-s-job-control-to-manage-foreground-and-background-processes\n  [2]: https://unix.stackexchange.com/questions/343569/how-do-i-use-nohup-and-tail-it-in-one-command/344659",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 389724,
      "author_name": "moshel",
      "author_url": "",
      "post_date": "09/19/2018 06:03:08",
      "content": "<p>If you are using shell and not jupyter, tmux works really well</p>",
      "votes": null,
      "replies": [
        {
          "id": 389738,
          "author_name": "nazimgirach",
          "author_url": "",
          "post_date": "09/19/2018 06:30:48",
          "content": "<p>Thanks Moshel, I do use Jupyter but I think if tmux works fine I would switch over to using shell.  But I would prefer something that works with Jupyter. </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 390589,
          "author_name": "hmendonca",
          "author_url": "",
          "post_date": "09/20/2018 13:30:46",
          "content": "<p>have you tried using \"nohup\" ?\nlike: </p>\n\n<p>nohup 2&gt;&amp;1 command &gt; yourlog.log &amp;</p>\n\n<p>to capture the possible error  msgs too...</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 390643,
          "author_name": "nazimgirach",
          "author_url": "",
          "post_date": "09/20/2018 15:18:27",
          "content": "<p>\"nohup\" is exactly what I wanted. Just tested it and it works perfectly. Thanks Henrique.</p>\n\n<p>P.S. I found more information <a href=\"https://www.digitalocean.com/community/tutorials/how-to-use-bash-s-job-control-to-manage-foreground-and-background-processes\">here</a> and <a href=\"https://unix.stackexchange.com/questions/343569/how-do-i-use-nohup-and-tail-it-in-one-command/344659\">here</a></p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 390047,
      "author_name": "kretes",
      "author_url": "",
      "post_date": "09/19/2018 16:13:24",
      "content": "<p>There are two options I know:</p>\n\n<p>Datalab: <a href=\"https://cloud.google.com/datalab/\">https://cloud.google.com/datalab/</a> where you get a Jupyter with custom look and feel running in the cloud as long as you wish.\nMl-engine: <a href=\"https://cloud.google.com/ml-engine/\">https://cloud.google.com/ml-engine/</a> which would be a non-interactive way of working: You submit your code as a python package for execution and it must be a fully automated procedure then. </p>",
      "votes": null,
      "replies": [
        {
          "id": 390070,
          "author_name": "nazimgirach",
          "author_url": "",
          "post_date": "09/19/2018 16:59:13",
          "content": "<p>Thanks Tomasz for your reply. I am currently exploring Datalab(well found out about it a few hours back) and ml-engine sounds like something that would match my requirement better </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "389721": "Due to frequent power cuts the ssh connection with the GCP instance is lost and the hours of training  is wasted. One option is to save weights every epoch but is there a way in which we can train our model like we do in Kaggle Kernels, they keep running in the cloud even when the PC is switched off?",
    "389724": "If you are using shell and not jupyter, tmux works really well",
    "389738": "Thanks Moshel, I do use Jupyter but I think if tmux works fine I would switch over to using shell.  But I would prefer something that works with Jupyter.",
    "390047": "There are two options I know:\n\n Datalab: https://cloud.google.com/datalab/ where you get a Jupyter with custom look and feel running in the cloud as long as you wish.\nMl-engine: https://cloud.google.com/ml-engine/ which would be a non-interactive way of working: You submit your code as a python package for execution and it must be a fully automated procedure then.",
    "390070": "Thanks Tomasz for your reply. I am currently exploring Datalab(well found out about it a few hours back) and ml-engine sounds like something that would match my requirement better",
    "390589": "have you tried using \"nohup\" ?\nlike: \n\nnohup 2&gt;&amp;1 command &gt; yourlog.log &amp;\n\nto capture the possible error  msgs too...",
    "390643": "\"nohup\" is exactly what I wanted. Just tested it and it works perfectly. Thanks Henrique.\n\nP.S. I found more information [here][1] and [here][2]\n\n\n  [1]: https://www.digitalocean.com/community/tutorials/how-to-use-bash-s-job-control-to-manage-foreground-and-background-processes\n  [2]: https://unix.stackexchange.com/questions/343569/how-do-i-use-nohup-and-tail-it-in-one-command/344659"
  },
  "source": "meta"
}