{
  "id": 132867,
  "title": "Does anyone encounter TPU 15 idle time and notebooks stop?",
  "url": "/competitions/flower-classification-with-tpus/discussion/132867",
  "author_name": "",
  "post_date": "2020-02-28T10:06:53.835274800Z",
  "votes": 2,
  "comment_count": 15,
  "views": 0,
  "content": "<p>when I use TPU to train model, even copy and run successful nootbook,  the nootbook always stopped and display \"TPU has 15 minutes idle time\" .\n^_^!</p>",
  "messages": [
    {
      "id": "758902",
      "postDate": "02/28/2020 10:06:53",
      "content": "<p>when I use TPU to train model, even copy and run successful nootbook,  the nootbook always stopped and display \"TPU has 15 minutes idle time\" .\n^_^!</p>",
      "rawMarkdown": "when I use TPU to train model, even copy and run successful nootbook,  the nootbook always stopped and display \"TPU has 15 minutes idle time\" .\n^_^!",
      "votes": null
    },
    {
      "id": "758949",
      "postDate": "02/28/2020 11:35:13",
      "content": "<p>i got this problem ,too.\n^_^!</p>",
      "rawMarkdown": "i got this problem ,too.\n^_^!",
      "votes": null
    },
    {
      "id": "759327",
      "postDate": "02/28/2020 22:30:02",
      "content": "<p>I confirm the idle timeout on TPU notebooks is 15 min. TPUs are a valuable ressource and we do not want them to sit idle while other users might be needing them.</p>\n\n<p>Is there a specific issue you are encountering around the timeout user experience ?</p>",
      "rawMarkdown": "I confirm the idle timeout on TPU notebooks is 15 min. TPUs are a valuable ressource and we do not want them to sit idle while other users might be needing them.\n\nIs there a specific issue you are encountering around the timeout user experience ?",
      "votes": null
    },
    {
      "id": "759472",
      "postDate": "02/29/2020 04:41:32",
      "content": "<p>This is my code:\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3926097%2F1ba1111cfa8ccdf756d4206438ec85d3%2Fscreenshot-www.kaggle.com-2020.02.29-12_56_02.png?generation=1582952222623785&amp;alt=media\" alt=\"\"></p>\n\n<p>Is there a problem? \nI think my model did  run on TPU. Why the TPU still sit idle?</p>",
      "rawMarkdown": "This is my code:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3926097%2F1ba1111cfa8ccdf756d4206438ec85d3%2Fscreenshot-www.kaggle.com-2020.02.29-12_56_02.png?generation=1582952222623785&amp;alt=media)\n\nIs there a problem? \nI think my model did  run on TPU. Why the TPU still sit idle?",
      "votes": null
    },
    {
      "id": "759628",
      "postDate": "02/29/2020 09:03:43",
      "content": "<p>good work.</p>",
      "rawMarkdown": "good work.",
      "votes": null
    },
    {
      "id": "760243",
      "postDate": "03/01/2020 02:53:15",
      "content": "<p>Thanks to your reply, I think the time used by load dataset and configure environment, expecially the first epoch, is too long.Then I change the notework, let works that not use TPU disperse in the whole kernel. So running succcessful.</p>",
      "rawMarkdown": "Thanks to your reply, I think the time used by load dataset and configure environment, expecially the first epoch, is too long.Then I change the notework, let works that not use TPU disperse in the whole kernel. So running succcessful.",
      "votes": null
    },
    {
      "id": "761711",
      "postDate": "03/02/2020 21:20:51",
      "content": "<p>Before the first epoch can start, you model is compiled through XLA. That is how TPUs work. But this compilation time should not count as \"idle\" time and your notebook should not timeout.</p>\n\n<p>Can you confirm what is happening exactly ? Does your notebook timeout in the middle of executing model.fit() ? If so, can you share the notebook (or a version of it that reproduces the issue) ?</p>",
      "rawMarkdown": "Before the first epoch can start, you model is compiled through XLA. That is how TPUs work. But this compilation time should not count as \"idle\" time and your notebook should not timeout.\n\nCan you confirm what is happening exactly ? Does your notebook timeout in the middle of executing model.fit() ? If so, can you share the notebook (or a version of it that reproduces the issue) ?",
      "votes": null
    },
    {
      "id": "762316",
      "postDate": "03/03/2020 12:23:52",
      "content": "<p>Thanks to your reply! I've made it successfully!</p>",
      "rawMarkdown": "Thanks to your reply! I've made it successfully!",
      "votes": null
    },
    {
      "id": "807535",
      "postDate": "04/14/2020 17:50:16",
      "content": "<p>I met the same issue! How did you fix that?</p>",
      "rawMarkdown": "I met the same issue! How did you fix that?",
      "votes": null
    },
    {
      "id": "824086",
      "postDate": "04/28/2020 06:31:37",
      "content": "<p>I am facing this problem too. \nIn an interactive session, while fitting, the notebook disappeared with that red banner saying \"TPU 15 idle time ... are you here\" message</p>\n\n<p>So I decided to commit and run all. Commitment is apparently still going on while I am writing this message but the red banner asking me if I am still here has appeared again and my notebook has disappeared.</p>\n\n<p>Sorry, I am discovering kaggle.</p>",
      "rawMarkdown": "I am facing this problem too. \nIn an interactive session, while fitting, the notebook disappeared with that red banner saying \"TPU 15 idle time ... are you here\" message\n\nSo I decided to commit and run all. Commitment is apparently still going on while I am writing this message but the red banner asking me if I am still here has appeared again and my notebook has disappeared.\n\nSorry, I am discovering kaggle.",
      "votes": null
    },
    {
      "id": "825135",
      "postDate": "04/28/2020 20:00:25",
      "content": "<p>The notebook probably finished and sat idle for 15 min.</p>",
      "rawMarkdown": "The notebook probably finished and sat idle for 15 min.",
      "votes": null
    },
    {
      "id": "826805",
      "postDate": "04/29/2020 21:48:14",
      "content": "<p>Hi Martin,</p>\n\n<p>It has just happened again. The notebook went idle when fitting. I was just in front of it.</p>\n\n<p>Is there a way to recover the execution state? otherwise, again, a couple of TPU hours wasted.</p>",
      "rawMarkdown": "Hi Martin,\n\nIt has just happened again. The notebook went idle when fitting. I was just in front of it.\n\nIs there a way to recover the execution state? otherwise, again, a couple of TPU hours wasted.",
      "votes": null
    },
    {
      "id": "826840",
      "postDate": "04/29/2020 22:35:35",
      "content": "<p>Let me bump this to the attention of Kaggle devs</p>",
      "rawMarkdown": "Let me bump this to the attention of Kaggle devs",
      "votes": null
    },
    {
      "id": "826856",
      "postDate": "04/29/2020 22:48:30",
      "content": "<p>Thank you, It happened in an interactive session. </p>\n\n<p>Now currently I am trying to commit. The notebook went idle again but apparently committing is still going on i.e. I am still seeing the small window on the left saying that my notebook version is being created.</p>\n\n<p>Is using the kaggle kernel the only way to work with TPU? I tried to use colab but apparently I got a downgraded TPU version causing OOM problems</p>",
      "rawMarkdown": "Thank you, It happened in an interactive session. \n\nNow currently I am trying to commit. The notebook went idle again but apparently committing is still going on i.e. I am still seeing the small window on the left saying that my notebook version is being created.\n\nIs using the kaggle kernel the only way to work with TPU? I tried to use colab but apparently I got a downgraded TPU version causing OOM problems",
      "votes": null
    },
    {
      "id": "826892",
      "postDate": "04/29/2020 23:29:49",
      "content": "<p>You can also use TPUs and even TPU pods on Google Cloud Platform, even from a Notebook. I have a tutorial here: <a href=\"https://bit.ly/keras-tpu-tf21\">bit.ly/keras-tpu-tf21 </a>(ignore the \"tf nightly\" mentions in there since TF2.1 is out now)</p>",
      "rawMarkdown": "You can also use TPUs and even TPU pods on Google Cloud Platform, even from a Notebook. I have a tutorial here: [bit.ly/keras-tpu-tf21 ](https://bit.ly/keras-tpu-tf21)(ignore the \"tf nightly\" mentions in there since TF2.1 is out now)",
      "votes": null
    },
    {
      "id": "826916",
      "postDate": "04/30/2020 00:12:53",
      "content": "<p>Thanks for raising the issue. We are looking into this.</p>",
      "rawMarkdown": "Thanks for raising the issue. We are looking into this.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 758949,
      "author_name": "dynmiwang",
      "author_url": "",
      "post_date": "02/28/2020 11:35:13",
      "content": "<p>i got this problem ,too.\n^_^!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 759327,
      "author_name": "mgorner",
      "author_url": "",
      "post_date": "02/28/2020 22:30:02",
      "content": "<p>I confirm the idle timeout on TPU notebooks is 15 min. TPUs are a valuable ressource and we do not want them to sit idle while other users might be needing them.</p>\n\n<p>Is there a specific issue you are encountering around the timeout user experience ?</p>",
      "votes": null,
      "replies": [
        {
          "id": 759472,
          "author_name": "dynmiwang",
          "author_url": "",
          "post_date": "02/29/2020 04:41:32",
          "content": "<p>This is my code:\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3926097%2F1ba1111cfa8ccdf756d4206438ec85d3%2Fscreenshot-www.kaggle.com-2020.02.29-12_56_02.png?generation=1582952222623785&amp;alt=media\" alt=\"\"></p>\n\n<p>Is there a problem? \nI think my model did  run on TPU. Why the TPU still sit idle?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 760243,
          "author_name": "hddchelsea",
          "author_url": "",
          "post_date": "03/01/2020 02:53:15",
          "content": "<p>Thanks to your reply, I think the time used by load dataset and configure environment, expecially the first epoch, is too long.Then I change the notework, let works that not use TPU disperse in the whole kernel. So running succcessful.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 761711,
          "author_name": "mgorner",
          "author_url": "",
          "post_date": "03/02/2020 21:20:51",
          "content": "<p>Before the first epoch can start, you model is compiled through XLA. That is how TPUs work. But this compilation time should not count as \"idle\" time and your notebook should not timeout.</p>\n\n<p>Can you confirm what is happening exactly ? Does your notebook timeout in the middle of executing model.fit() ? If so, can you share the notebook (or a version of it that reproduces the issue) ?</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 759628,
      "author_name": "harshithasivakumar",
      "author_url": "",
      "post_date": "02/29/2020 09:03:43",
      "content": "<p>good work.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 762316,
      "author_name": "dynmiwang",
      "author_url": "",
      "post_date": "03/03/2020 12:23:52",
      "content": "<p>Thanks to your reply! I've made it successfully!</p>",
      "votes": null,
      "replies": [
        {
          "id": 807535,
          "author_name": "liuzhian",
          "author_url": "",
          "post_date": "04/14/2020 17:50:16",
          "content": "<p>I met the same issue! How did you fix that?</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 824086,
      "author_name": "albangabillon",
      "author_url": "",
      "post_date": "04/28/2020 06:31:37",
      "content": "<p>I am facing this problem too. \nIn an interactive session, while fitting, the notebook disappeared with that red banner saying \"TPU 15 idle time ... are you here\" message</p>\n\n<p>So I decided to commit and run all. Commitment is apparently still going on while I am writing this message but the red banner asking me if I am still here has appeared again and my notebook has disappeared.</p>\n\n<p>Sorry, I am discovering kaggle.</p>",
      "votes": null,
      "replies": [
        {
          "id": 825135,
          "author_name": "mgorner",
          "author_url": "",
          "post_date": "04/28/2020 20:00:25",
          "content": "<p>The notebook probably finished and sat idle for 15 min.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 826805,
          "author_name": "albangabillon",
          "author_url": "",
          "post_date": "04/29/2020 21:48:14",
          "content": "<p>Hi Martin,</p>\n\n<p>It has just happened again. The notebook went idle when fitting. I was just in front of it.</p>\n\n<p>Is there a way to recover the execution state? otherwise, again, a couple of TPU hours wasted.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 826840,
          "author_name": "mgorner",
          "author_url": "",
          "post_date": "04/29/2020 22:35:35",
          "content": "<p>Let me bump this to the attention of Kaggle devs</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 826856,
          "author_name": "albangabillon",
          "author_url": "",
          "post_date": "04/29/2020 22:48:30",
          "content": "<p>Thank you, It happened in an interactive session. </p>\n\n<p>Now currently I am trying to commit. The notebook went idle again but apparently committing is still going on i.e. I am still seeing the small window on the left saying that my notebook version is being created.</p>\n\n<p>Is using the kaggle kernel the only way to work with TPU? I tried to use colab but apparently I got a downgraded TPU version causing OOM problems</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 826892,
          "author_name": "mgorner",
          "author_url": "",
          "post_date": "04/29/2020 23:29:49",
          "content": "<p>You can also use TPUs and even TPU pods on Google Cloud Platform, even from a Notebook. I have a tutorial here: <a href=\"https://bit.ly/keras-tpu-tf21\">bit.ly/keras-tpu-tf21 </a>(ignore the \"tf nightly\" mentions in there since TF2.1 is out now)</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 826916,
          "author_name": "ifigotin",
          "author_url": "",
          "post_date": "04/30/2020 00:12:53",
          "content": "<p>Thanks for raising the issue. We are looking into this.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "758902": "when I use TPU to train model, even copy and run successful nootbook,  the nootbook always stopped and display \"TPU has 15 minutes idle time\" .\n^_^!",
    "758949": "i got this problem ,too.\n^_^!",
    "759327": "I confirm the idle timeout on TPU notebooks is 15 min. TPUs are a valuable ressource and we do not want them to sit idle while other users might be needing them.\n\nIs there a specific issue you are encountering around the timeout user experience ?",
    "759472": "This is my code:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3926097%2F1ba1111cfa8ccdf756d4206438ec85d3%2Fscreenshot-www.kaggle.com-2020.02.29-12_56_02.png?generation=1582952222623785&amp;alt=media)\n\nIs there a problem? \nI think my model did  run on TPU. Why the TPU still sit idle?",
    "759628": "good work.",
    "760243": "Thanks to your reply, I think the time used by load dataset and configure environment, expecially the first epoch, is too long.Then I change the notework, let works that not use TPU disperse in the whole kernel. So running succcessful.",
    "761711": "Before the first epoch can start, you model is compiled through XLA. That is how TPUs work. But this compilation time should not count as \"idle\" time and your notebook should not timeout.\n\nCan you confirm what is happening exactly ? Does your notebook timeout in the middle of executing model.fit() ? If so, can you share the notebook (or a version of it that reproduces the issue) ?",
    "762316": "Thanks to your reply! I've made it successfully!",
    "807535": "I met the same issue! How did you fix that?",
    "824086": "I am facing this problem too. \nIn an interactive session, while fitting, the notebook disappeared with that red banner saying \"TPU 15 idle time ... are you here\" message\n\nSo I decided to commit and run all. Commitment is apparently still going on while I am writing this message but the red banner asking me if I am still here has appeared again and my notebook has disappeared.\n\nSorry, I am discovering kaggle.",
    "825135": "The notebook probably finished and sat idle for 15 min.",
    "826805": "Hi Martin,\n\nIt has just happened again. The notebook went idle when fitting. I was just in front of it.\n\nIs there a way to recover the execution state? otherwise, again, a couple of TPU hours wasted.",
    "826840": "Let me bump this to the attention of Kaggle devs",
    "826856": "Thank you, It happened in an interactive session. \n\nNow currently I am trying to commit. The notebook went idle again but apparently committing is still going on i.e. I am still seeing the small window on the left saying that my notebook version is being created.\n\nIs using the kaggle kernel the only way to work with TPU? I tried to use colab but apparently I got a downgraded TPU version causing OOM problems",
    "826892": "You can also use TPUs and even TPU pods on Google Cloud Platform, even from a Notebook. I have a tutorial here: [bit.ly/keras-tpu-tf21 ](https://bit.ly/keras-tpu-tf21)(ignore the \"tf nightly\" mentions in there since TF2.1 is out now)",
    "826916": "Thanks for raising the issue. We are looking into this."
  },
  "source": "meta"
}