{
  "id": 120419,
  "title": "GPU quota bug, and be careful everyone.",
  "url": "/competitions/tensorflow2-question-answering/discussion/120419",
  "author_name": "",
  "post_date": "2019-12-06T00:59:25.522212800Z",
  "votes": 6,
  "comment_count": 4,
  "views": 0,
  "content": "<p>I'm here to report a bug, and warn others to do not make the same mistake.</p>\n\n<p>I have recently created 3 kernels to try training BERT on Kaggle kernels, the thing is, one of them runs for more than two days and the others didn't even finish. But Kaggle GPUs were supposed to run a max of 9 hours (correct me if I'm wrong), the result was this:</p>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1182060%2F1b262562f7bae70176335d96fbcaf46f%2FScreenshot%20from%202019-12-05%2021-55-36.png?generation=1575593769514142&amp;alt=media\" alt=\"\"></p>\n\n<p>Unfortunately, I've deleted them before noticing the bug otherwise I could share.</p>\n\n<p>I did many modifications to the original code, so I'm not sure what was the problem, but might be related to checkpoint saving, or memory issues.</p>",
  "messages": [
    {
      "id": "688710",
      "postDate": "12/06/2019 00:59:25",
      "content": "<p>I'm here to report a bug, and warn others to do not make the same mistake.</p>\n\n<p>I have recently created 3 kernels to try training BERT on Kaggle kernels, the thing is, one of them runs for more than two days and the others didn't even finish. But Kaggle GPUs were supposed to run a max of 9 hours (correct me if I'm wrong), the result was this:</p>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1182060%2F1b262562f7bae70176335d96fbcaf46f%2FScreenshot%20from%202019-12-05%2021-55-36.png?generation=1575593769514142&amp;alt=media\" alt=\"\"></p>\n\n<p>Unfortunately, I've deleted them before noticing the bug otherwise I could share.</p>\n\n<p>I did many modifications to the original code, so I'm not sure what was the problem, but might be related to checkpoint saving, or memory issues.</p>",
      "rawMarkdown": "I'm here to report a bug, and warn others to do not make the same mistake.\n\nI have recently created 3 kernels to try training BERT on Kaggle kernels, the thing is, one of them runs for more than two days and the others didn't even finish. But Kaggle GPUs were supposed to run a max of 9 hours (correct me if I'm wrong), the result was this:\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1182060%2F1b262562f7bae70176335d96fbcaf46f%2FScreenshot%20from%202019-12-05%2021-55-36.png?generation=1575593769514142&amp;alt=media)\n\nUnfortunately, I've deleted them before noticing the bug otherwise I could share.\n\nI did many modifications to the original code, so I'm not sure what was the problem, but might be related to checkpoint saving, or memory issues.",
      "votes": null
    },
    {
      "id": "689035",
      "postDate": "12/06/2019 11:28:57",
      "content": "<p><a href=\"/dimitreoliveira\">@dimitreoliveira</a> this has happened with me too in few contests. This happens when your committed kernel runs out of memory and restarts in commit mode. But for some reason kaggle doesn't kill that kernel so it keeps running forever until you stop it manually</p>",
      "rawMarkdown": "dimitreoliveira this has happened with me too in few contests. This happens when your committed kernel runs out of memory and restarts in commit mode. But for some reason kaggle doesn't kill that kernel so it keeps running forever until you stop it manually",
      "votes": null
    },
    {
      "id": "689050",
      "postDate": "12/06/2019 11:50:55",
      "content": "<p>Thanks <a href=\"/axel81\">@axel81</a> , this makes sense, so we need to be extra careful on this competition because with BERT is easy to run out of memory</p>",
      "rawMarkdown": "Thanks @axel81 , this makes sense, so we need to be extra careful on this competition because with BERT is easy to run out of memory",
      "votes": null
    },
    {
      "id": "689173",
      "postDate": "12/06/2019 15:18:37",
      "content": "<p>Yeah right</p>",
      "rawMarkdown": "Yeah right",
      "votes": null
    },
    {
      "id": "689251",
      "postDate": "12/06/2019 17:47:03",
      "content": "<p>We're working on a fix to prevent the time limit overrun issue. It should be a rare issue in general, but likely happens as you suggest when our timeout gets killed or missed due to resource exhaustion.</p>",
      "rawMarkdown": "We're working on a fix to prevent the time limit overrun issue. It should be a rare issue in general, but likely happens as you suggest when our timeout gets killed or missed due to resource exhaustion.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 689035,
      "author_name": "axel81",
      "author_url": "",
      "post_date": "12/06/2019 11:28:57",
      "content": "<p><a href=\"/dimitreoliveira\">@dimitreoliveira</a> this has happened with me too in few contests. This happens when your committed kernel runs out of memory and restarts in commit mode. But for some reason kaggle doesn't kill that kernel so it keeps running forever until you stop it manually</p>",
      "votes": null,
      "replies": [
        {
          "id": 689050,
          "author_name": "dimitreoliveira",
          "author_url": "",
          "post_date": "12/06/2019 11:50:55",
          "content": "<p>Thanks <a href=\"/axel81\">@axel81</a> , this makes sense, so we need to be extra careful on this competition because with BERT is easy to run out of memory</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 689173,
          "author_name": "axel81",
          "author_url": "",
          "post_date": "12/06/2019 15:18:37",
          "content": "<p>Yeah right</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 689251,
          "author_name": "herbison",
          "author_url": "",
          "post_date": "12/06/2019 17:47:03",
          "content": "<p>We're working on a fix to prevent the time limit overrun issue. It should be a rare issue in general, but likely happens as you suggest when our timeout gets killed or missed due to resource exhaustion.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "688710": "I'm here to report a bug, and warn others to do not make the same mistake.\n\nI have recently created 3 kernels to try training BERT on Kaggle kernels, the thing is, one of them runs for more than two days and the others didn't even finish. But Kaggle GPUs were supposed to run a max of 9 hours (correct me if I'm wrong), the result was this:\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1182060%2F1b262562f7bae70176335d96fbcaf46f%2FScreenshot%20from%202019-12-05%2021-55-36.png?generation=1575593769514142&amp;alt=media)\n\nUnfortunately, I've deleted them before noticing the bug otherwise I could share.\n\nI did many modifications to the original code, so I'm not sure what was the problem, but might be related to checkpoint saving, or memory issues.",
    "689035": "dimitreoliveira this has happened with me too in few contests. This happens when your committed kernel runs out of memory and restarts in commit mode. But for some reason kaggle doesn't kill that kernel so it keeps running forever until you stop it manually",
    "689050": "Thanks @axel81 , this makes sense, so we need to be extra careful on this competition because with BERT is easy to run out of memory",
    "689173": "Yeah right",
    "689251": "We're working on a fix to prevent the time limit overrun issue. It should be a rare issue in general, but likely happens as you suggest when our timeout gets killed or missed due to resource exhaustion."
  },
  "source": "meta"
}