{
  "id": 204765,
  "title": "Two bugs in 2 weeks іn this competition:-)",
  "url": "/competitions/rfcx-species-audio-detection/discussion/204765",
  "author_name": "",
  "post_date": "2020-12-16T17:29:44.712644100Z",
  "votes": 1,
  "comment_count": 5,
  "views": 0,
  "content": "<p>Hi to all! In the last two weeks I have seen 2 bugs: </p>\n<ol>\n<li>When using tpu when the model uses a large amount of RAM the notebook runs for 9 hours and terminates after the end of time. Instead of stopping through the use of all RAM. </li>\n<li>My notebooks are not loading right now. <br>\nIt's just that I have such problems or you have also encountered them. Good luck to all!</li>\n</ol>",
  "messages": [
    {
      "id": "1115944",
      "postDate": "12/16/2020 17:29:44",
      "content": "<p>Hi to all! In the last two weeks I have seen 2 bugs: </p>\n<ol>\n<li>When using tpu when the model uses a large amount of RAM the notebook runs for 9 hours and terminates after the end of time. Instead of stopping through the use of all RAM. </li>\n<li>My notebooks are not loading right now. <br>\nIt's just that I have such problems or you have also encountered them. Good luck to all!</li>\n</ol>",
      "rawMarkdown": "Hi to all! In the last two weeks I have seen 2 bugs: \n1. When using tpu when the model uses a large amount of RAM the notebook runs for 9 hours and terminates after the end of time. Instead of stopping through the use of all RAM. \n2. My notebooks are not loading right now. \nIt's just that I have such problems or you have also encountered them. Good luck to all!",
      "votes": null
    },
    {
      "id": "1115956",
      "postDate": "12/16/2020 17:37:20",
      "content": "<p>My notebooks are not loading right now either.</p>",
      "rawMarkdown": "My notebooks are not loading right now either.",
      "votes": null
    },
    {
      "id": "1116224",
      "postDate": "12/17/2020 01:08:00",
      "content": "<p>Generally good to check and/or post to the \"Product Feedback\" when experiencing issues: <a href=\"https://kaggle.com/product-feedback/204767\" target=\"_blank\">https://kaggle.com/product-feedback/204767</a></p>\n<p>There's a few posts there on this (since been resolved).</p>",
      "rawMarkdown": "Generally good to check and/or post to the \"Product Feedback\" when experiencing issues: https://kaggle.com/product-feedback/204767\n\nThere's a few posts there on this (since been resolved).",
      "votes": null
    },
    {
      "id": "1116228",
      "postDate": "12/17/2020 01:15:37",
      "content": "<p>The TPU issue is separate, effectively running out of memory leads to undefined behavior.</p>\n<p>Save Version Notebooks stop when one of two conditions are met:</p>\n<ul>\n<li>They run for longer than the timeout we specify (usually 9hrs).</li>\n<li>The notebook process exits.</li>\n</ul>\n<p>There's no nice way for us to detect that a notebook is 'stuck out of RAM' vs. using all available RAM and making progress, and we don't want to stop a 'working notebook' early.</p>\n<p>I suggest adding a background thread which you can code to decide to exit early when you desire. Depending on the libraries involved, you could theoretically use this to exit 'gracefully' even, and try to save your model/results/progress instead of losing everything abruptly when you reach the 9hr limit.</p>",
      "rawMarkdown": "The TPU issue is separate, effectively running out of memory leads to undefined behavior.\n\nSave Version Notebooks stop when one of two conditions are met:\n* They run for longer than the timeout we specify (usually 9hrs).\n* The notebook process exits.\n\nThere's no nice way for us to detect that a notebook is 'stuck out of RAM' vs. using all available RAM and making progress, and we don't want to stop a 'working notebook' early.\n\nI suggest adding a background thread which you can code to decide to exit early when you desire. Depending on the libraries involved, you could theoretically use this to exit 'gracefully' even, and try to save your model/results/progress instead of losing everything abruptly when you reach the 9hr limit.",
      "votes": null
    },
    {
      "id": "1116544",
      "postDate": "12/17/2020 08:54:34",
      "content": "<p>I think TPU limit is 3 hr per session not 9 , according to <a href=\"https://www.kaggle.com/docs/tpu\" target=\"_blank\">https://www.kaggle.com/docs/tpu</a></p>",
      "rawMarkdown": "I think TPU limit is 3 hr per session not 9 , according to https://www.kaggle.com/docs/tpu",
      "votes": null
    },
    {
      "id": "1139883",
      "postDate": "01/05/2021 17:17:55",
      "content": "<p>Thanks for pointing that out, I updated it. We bumped the runtime to 9hrs for all sessions a little while ago.</p>",
      "rawMarkdown": "Thanks for pointing that out, I updated it. We bumped the runtime to 9hrs for all sessions a little while ago.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1115956,
      "author_name": "cjturner",
      "author_url": "",
      "post_date": "12/16/2020 17:37:20",
      "content": "<p>My notebooks are not loading right now either.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1116224,
      "author_name": "herbison",
      "author_url": "",
      "post_date": "12/17/2020 01:08:00",
      "content": "<p>Generally good to check and/or post to the \"Product Feedback\" when experiencing issues: <a href=\"https://kaggle.com/product-feedback/204767\" target=\"_blank\">https://kaggle.com/product-feedback/204767</a></p>\n<p>There's a few posts there on this (since been resolved).</p>",
      "votes": null,
      "replies": [
        {
          "id": 1116228,
          "author_name": "herbison",
          "author_url": "",
          "post_date": "12/17/2020 01:15:37",
          "content": "<p>The TPU issue is separate, effectively running out of memory leads to undefined behavior.</p>\n<p>Save Version Notebooks stop when one of two conditions are met:</p>\n<ul>\n<li>They run for longer than the timeout we specify (usually 9hrs).</li>\n<li>The notebook process exits.</li>\n</ul>\n<p>There's no nice way for us to detect that a notebook is 'stuck out of RAM' vs. using all available RAM and making progress, and we don't want to stop a 'working notebook' early.</p>\n<p>I suggest adding a background thread which you can code to decide to exit early when you desire. Depending on the libraries involved, you could theoretically use this to exit 'gracefully' even, and try to save your model/results/progress instead of losing everything abruptly when you reach the 9hr limit.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1116544,
          "author_name": "nyleve",
          "author_url": "",
          "post_date": "12/17/2020 08:54:34",
          "content": "<p>I think TPU limit is 3 hr per session not 9 , according to <a href=\"https://www.kaggle.com/docs/tpu\" target=\"_blank\">https://www.kaggle.com/docs/tpu</a></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1139883,
          "author_name": "herbison",
          "author_url": "",
          "post_date": "01/05/2021 17:17:55",
          "content": "<p>Thanks for pointing that out, I updated it. We bumped the runtime to 9hrs for all sessions a little while ago.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1115944": "Hi to all! In the last two weeks I have seen 2 bugs: \n1. When using tpu when the model uses a large amount of RAM the notebook runs for 9 hours and terminates after the end of time. Instead of stopping through the use of all RAM. \n2. My notebooks are not loading right now. \nIt's just that I have such problems or you have also encountered them. Good luck to all!",
    "1115956": "My notebooks are not loading right now either.",
    "1116224": "Generally good to check and/or post to the \"Product Feedback\" when experiencing issues: https://kaggle.com/product-feedback/204767\n\nThere's a few posts there on this (since been resolved).",
    "1116228": "The TPU issue is separate, effectively running out of memory leads to undefined behavior.\n\nSave Version Notebooks stop when one of two conditions are met:\n* They run for longer than the timeout we specify (usually 9hrs).\n* The notebook process exits.\n\nThere's no nice way for us to detect that a notebook is 'stuck out of RAM' vs. using all available RAM and making progress, and we don't want to stop a 'working notebook' early.\n\nI suggest adding a background thread which you can code to decide to exit early when you desire. Depending on the libraries involved, you could theoretically use this to exit 'gracefully' even, and try to save your model/results/progress instead of losing everything abruptly when you reach the 9hr limit.",
    "1116544": "I think TPU limit is 3 hr per session not 9 , according to https://www.kaggle.com/docs/tpu",
    "1139883": "Thanks for pointing that out, I updated it. We bumped the runtime to 9hrs for all sessions a little while ago."
  },
  "source": "meta"
}