{
  "id": 63247,
  "title": "GCP instance killing processes",
  "url": "/competitions/google-ai-open-images-object-detection-track/discussion/63247",
  "author_name": "Adithya Seshasayee",
  "post_date": "2018-08-14T00:30:58.327000",
  "votes": 0,
  "comment_count": 8,
  "views": 0,
  "content": "<p>I am trying to  train a model on a GCP cloud instance. I am using the model from <a href=\"https://github.com/fizyr/keras-retinanet\">https://github.com/fizyr/keras-retinanet</a>. I use screen to start the code and run it after logging out. It takes a day to complete the first epoch and then the process outputs : Killed96160. Do instances automatically kill the processes after a long time? If so, how can I get around this?</p>",
  "messages": [
    {
      "id": 371950,
      "postDate": "2018-08-17T22:18:04.480Z",
      "content": "<p>Haha dude happened to me so many times, burned through most of my credit like that. Google should be charging for instance usage rather than the per minute charge. and the latency to read and write data in the bucket is just so sad</p>",
      "rawMarkdown": "Haha dude happened to me so many times, burned through most of my credit like that. Google should be charging for instance usage rather than the per minute charge. and the latency to read and write data in the bucket is just so sad",
      "replies": [
        {
          "id": 372016,
          "postDate": "2018-08-18T04:35:46.860Z",
          "content": "<p>How did you fix this issue?</p>",
          "rawMarkdown": "How did you fix this issue?"
        },
        {
          "id": 372027,
          "postDate": "2018-08-18T05:19:42.900Z",
          "content": "<p>Couldn't fix it. Just restarted the instance each time it failed. Probably there might be a solution out there but after spending hours going through forums and Google docs came to the conclusion its nt worth the time</p>",
          "rawMarkdown": "Couldn't fix it. Just restarted the instance each time it failed. Probably there might be a solution out there but after spending hours going through forums and Google docs came to the conclusion its nt worth the time"
        },
        {
          "id": 372032,
          "postDate": "2018-08-18T05:34:42.227Z",
          "content": "<p>Oh! Just remembered something. It had to do i think with swapfile being too small so operating system process killer kicked in when memory was exhausted</p>",
          "rawMarkdown": "Oh! Just remembered something. It had to do i think with swapfile being too small so operating system process killer kicked in when memory was exhausted"
        },
        {
          "id": 372360,
          "postDate": "2018-08-19T05:04:26.353Z",
          "content": "<p>Oh! I never thought of that. Thanks, I'll check it out.</p>",
          "rawMarkdown": "Oh! I never thought of that. Thanks, I'll check it out."
        }
      ]
    },
    {
      "id": 369930,
      "postDate": "2018-08-14T00:49:14.843Z",
      "content": "<p>Is it preemptive instance? </p>",
      "rawMarkdown": "Is it preemptive instance? ",
      "replies": [
        {
          "id": 370108,
          "postDate": "2018-08-14T09:26:40.257Z",
          "content": "<p>Thanks for replying. I don't think so. Anyway, the process stops and the instance keeps running.</p>",
          "rawMarkdown": "Thanks for replying. I don't think so. Anyway, the process stops and the instance keeps running."
        },
        {
          "id": 371949,
          "postDate": "2018-08-17T22:17:39.917Z",
          "rawMarkdown": "",
          "isDeleted": true
        }
      ]
    },
    {
      "id": 369913,
      "postDate": "2018-08-14T00:30:58.327Z",
      "content": "<p>I am trying to  train a model on a GCP cloud instance. I am using the model from <a href=\"https://github.com/fizyr/keras-retinanet\">https://github.com/fizyr/keras-retinanet</a>. I use screen to start the code and run it after logging out. It takes a day to complete the first epoch and then the process outputs : Killed96160. Do instances automatically kill the processes after a long time? If so, how can I get around this?</p>",
      "rawMarkdown": "I am trying to  train a model on a GCP cloud instance. I am using the model from https://github.com/fizyr/keras-retinanet. I use screen to start the code and run it after logging out. It takes a day to complete the first epoch and then the process outputs : Killed96160. Do instances automatically kill the processes after a long time? If so, how can I get around this?"
    }
  ],
  "comments": [
    {
      "id": 371950,
      "author_name": "Mukesh",
      "author_url": "",
      "post_date": "2018-08-17T22:18:04.480000",
      "content": "<p>Haha dude happened to me so many times, burned through most of my credit like that. Google should be charging for instance usage rather than the per minute charge. and the latency to read and write data in the bucket is just so sad</p>",
      "votes": 0,
      "replies": [
        {
          "id": 372016,
          "author_name": "Adithya Seshasayee",
          "author_url": "",
          "post_date": "2018-08-18T04:35:46.860000",
          "content": "<p>How did you fix this issue?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 372027,
          "author_name": "Mukesh",
          "author_url": "",
          "post_date": "2018-08-18T05:19:42.900000",
          "content": "<p>Couldn't fix it. Just restarted the instance each time it failed. Probably there might be a solution out there but after spending hours going through forums and Google docs came to the conclusion its nt worth the time</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 372032,
          "author_name": "Moshel",
          "author_url": "",
          "post_date": "2018-08-18T05:34:42.227000",
          "content": "<p>Oh! Just remembered something. It had to do i think with swapfile being too small so operating system process killer kicked in when memory was exhausted</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 372360,
          "author_name": "Adithya Seshasayee",
          "author_url": "",
          "post_date": "2018-08-19T05:04:26.353000",
          "content": "<p>Oh! I never thought of that. Thanks, I'll check it out.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 369930,
      "author_name": "Moshel",
      "author_url": "",
      "post_date": "2018-08-14T00:49:14.843000",
      "content": "<p>Is it preemptive instance? </p>",
      "votes": 0,
      "replies": [
        {
          "id": 370108,
          "author_name": "Adithya Seshasayee",
          "author_url": "",
          "post_date": "2018-08-14T09:26:40.257000",
          "content": "<p>Thanks for replying. I don't think so. Anyway, the process stops and the instance keeps running.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 371949,
          "author_name": "",
          "author_url": "",
          "post_date": "2018-08-17T22:17:39.917000",
          "content": "",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "371950": "Haha dude happened to me so many times, burned through most of my credit like that. Google should be charging for instance usage rather than the per minute charge. and the latency to read and write data in the bucket is just so sad",
    "369930": "Is it preemptive instance? ",
    "369913": "I am trying to  train a model on a GCP cloud instance. I am using the model from https://github.com/fizyr/keras-retinanet. I use screen to start the code and run it after logging out. It takes a day to complete the first epoch and then the process outputs : Killed96160. Do instances automatically kill the processes after a long time? If so, how can I get around this?"
  }
}