{
  "id": 174757,
  "title": "TPU training, seems to hang, no progress",
  "url": "/competitions/siim-isic-melanoma-classification/discussion/174757",
  "author_name": "",
  "post_date": "2020-08-15T07:28:24.308997100Z",
  "votes": null,
  "comment_count": 7,
  "views": 0,
  "content": "<p>Nearly time, <br>\nMy notebook seems to hang, no progress at the first epoch (look my attachment file).</p>\n<p>Waste an hour for this. You should check this before turn off your tab, hope this topic save your resource limit</p>\n<p>I had used the last environment,<br>\nBoth the colab get the issue too. </p>\n<p>I don't think that the large memory is problem. BC it still work in last week<br>\nDo anyone tell me why and how to fix ?</p>\n<p>Thanks.</p>",
  "messages": [
    {
      "id": "971110",
      "postDate": "08/15/2020 07:28:24",
      "content": "<p>Nearly time, <br>\nMy notebook seems to hang, no progress at the first epoch (look my attachment file).</p>\n<p>Waste an hour for this. You should check this before turn off your tab, hope this topic save your resource limit</p>\n<p>I had used the last environment,<br>\nBoth the colab get the issue too. </p>\n<p>I don't think that the large memory is problem. BC it still work in last week<br>\nDo anyone tell me why and how to fix ?</p>\n<p>Thanks.</p>",
      "rawMarkdown": "Nearly time, \nMy notebook seems to hang, no progress at the first epoch (look my attachment file).\n\nWaste an hour for this. You should check this before turn off your tab, hope this topic save your resource limit\n\nI had used the last environment,\nBoth the colab get the issue too. \n\nI don't think that the large memory is problem. BC it still work in last week\nDo anyone tell me why and how to fix ?\n\nThanks.",
      "votes": null
    },
    {
      "id": "971159",
      "postDate": "08/15/2020 08:19:28",
      "content": "<p>I experienced similar things and i solved it by slightly decrease batch size (72 to 64 in my case). It's based on my intuition, so it might not work on your end.</p>",
      "rawMarkdown": "I experienced similar things and i solved it by slightly decrease batch size (72 to 64 in my case). It's based on my intuition, so it might not work on your end.",
      "votes": null
    },
    {
      "id": "971181",
      "postDate": "08/15/2020 08:59:03",
      "content": "<p>on kaggle notebook, i decrease batch size 16 to 12 (image size 384) . And over 1st epoch after 25 minutes. too slow compared to before time</p>",
      "rawMarkdown": "on kaggle notebook, i decrease batch size 16 to 12 (image size 384) . And over 1st epoch after 25 minutes. too slow compared to before time",
      "votes": null
    },
    {
      "id": "971192",
      "postDate": "08/15/2020 09:19:30",
      "content": "<p>It may be due to the browser or internet issue. Kaggle recommends Chrome as their preferred browser.</p>",
      "rawMarkdown": "It may be due to the browser or internet issue. Kaggle recommends Chrome as their preferred browser.",
      "votes": null
    },
    {
      "id": "971208",
      "postDate": "08/15/2020 09:32:32",
      "content": "<p>Thank your reply, <br>\nI use Chrome </p>",
      "rawMarkdown": "Thank your reply, \nI use Chrome",
      "votes": null
    },
    {
      "id": "971425",
      "postDate": "08/15/2020 14:11:10",
      "content": "<p>Check your notebook's TF version and make sure it is TF2.2. If it is not, make sure your enviroment is set to \"always use latest\". (For a few days this week Kaggle upgraded to TF2.3 which caused many problems with TPU. But then they downgraded, so currently TF2.2 is the Kaggle latest).</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1723677%2F38e92cf743cbb494f5abcc3f1b240381%2Ftf.png?generation=1597500571867037&amp;alt=media\" alt=\"\"></p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1723677%2Fd9ac2e00017ff7ced90f68af8a9dd230%2Ftf2.png?generation=1597500613319783&amp;alt=media\" alt=\"\"></p>",
      "rawMarkdown": "Check your notebook's TF version and make sure it is TF2.2. If it is not, make sure your enviroment is set to \"always use latest\". (For a few days this week Kaggle upgraded to TF2.3 which caused many problems with TPU. But then they downgraded, so currently TF2.2 is the Kaggle latest).\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1723677%2F38e92cf743cbb494f5abcc3f1b240381%2Ftf.png?generation=1597500571867037&alt=media)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1723677%2Fd9ac2e00017ff7ced90f68af8a9dd230%2Ftf2.png?generation=1597500613319783&alt=media)",
      "votes": null
    },
    {
      "id": "971841",
      "postDate": "08/16/2020 01:46:05",
      "content": "<p>I had tried downgrade but get issue on colab. Let try on kaggle. With TPU, It always use latest enviroment.<br>\nI think bc they upgrade TF 2.3 too. But my LB score decrease 0.04 too :(</p>",
      "rawMarkdown": "I had tried downgrade but get issue on colab. Let try on kaggle. With TPU, It always use latest enviroment.\nI think bc they upgrade TF 2.3 too. But my LB score decrease 0.04 too :(",
      "votes": null
    },
    {
      "id": "971873",
      "postDate": "08/16/2020 02:59:07",
      "content": "<p>EfficientNet TPU is not working correctly in TF2.3. Kaggle is currently TF2.2, so things are working well at Kaggle now.</p>",
      "rawMarkdown": "EfficientNet TPU is not working correctly in TF2.3. Kaggle is currently TF2.2, so things are working well at Kaggle now.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 971159,
      "author_name": "ilosvigil",
      "author_url": "",
      "post_date": "08/15/2020 08:19:28",
      "content": "<p>I experienced similar things and i solved it by slightly decrease batch size (72 to 64 in my case). It's based on my intuition, so it might not work on your end.</p>",
      "votes": null,
      "replies": [
        {
          "id": 971181,
          "author_name": "truonghoang",
          "author_url": "",
          "post_date": "08/15/2020 08:59:03",
          "content": "<p>on kaggle notebook, i decrease batch size 16 to 12 (image size 384) . And over 1st epoch after 25 minutes. too slow compared to before time</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 971192,
      "author_name": "dtprksh50",
      "author_url": "",
      "post_date": "08/15/2020 09:19:30",
      "content": "<p>It may be due to the browser or internet issue. Kaggle recommends Chrome as their preferred browser.</p>",
      "votes": null,
      "replies": [
        {
          "id": 971208,
          "author_name": "truonghoang",
          "author_url": "",
          "post_date": "08/15/2020 09:32:32",
          "content": "<p>Thank your reply, <br>\nI use Chrome </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 971425,
      "author_name": "cdeotte",
      "author_url": "",
      "post_date": "08/15/2020 14:11:10",
      "content": "<p>Check your notebook's TF version and make sure it is TF2.2. If it is not, make sure your enviroment is set to \"always use latest\". (For a few days this week Kaggle upgraded to TF2.3 which caused many problems with TPU. But then they downgraded, so currently TF2.2 is the Kaggle latest).</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1723677%2F38e92cf743cbb494f5abcc3f1b240381%2Ftf.png?generation=1597500571867037&amp;alt=media\" alt=\"\"></p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1723677%2Fd9ac2e00017ff7ced90f68af8a9dd230%2Ftf2.png?generation=1597500613319783&amp;alt=media\" alt=\"\"></p>",
      "votes": null,
      "replies": [
        {
          "id": 971841,
          "author_name": "truonghoang",
          "author_url": "",
          "post_date": "08/16/2020 01:46:05",
          "content": "<p>I had tried downgrade but get issue on colab. Let try on kaggle. With TPU, It always use latest enviroment.<br>\nI think bc they upgrade TF 2.3 too. But my LB score decrease 0.04 too :(</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 971873,
          "author_name": "cdeotte",
          "author_url": "",
          "post_date": "08/16/2020 02:59:07",
          "content": "<p>EfficientNet TPU is not working correctly in TF2.3. Kaggle is currently TF2.2, so things are working well at Kaggle now.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "971110": "Nearly time, \nMy notebook seems to hang, no progress at the first epoch (look my attachment file).\n\nWaste an hour for this. You should check this before turn off your tab, hope this topic save your resource limit\n\nI had used the last environment,\nBoth the colab get the issue too. \n\nI don't think that the large memory is problem. BC it still work in last week\nDo anyone tell me why and how to fix ?\n\nThanks.",
    "971159": "I experienced similar things and i solved it by slightly decrease batch size (72 to 64 in my case). It's based on my intuition, so it might not work on your end.",
    "971181": "on kaggle notebook, i decrease batch size 16 to 12 (image size 384) . And over 1st epoch after 25 minutes. too slow compared to before time",
    "971192": "It may be due to the browser or internet issue. Kaggle recommends Chrome as their preferred browser.",
    "971208": "Thank your reply, \nI use Chrome",
    "971425": "Check your notebook's TF version and make sure it is TF2.2. If it is not, make sure your enviroment is set to \"always use latest\". (For a few days this week Kaggle upgraded to TF2.3 which caused many problems with TPU. But then they downgraded, so currently TF2.2 is the Kaggle latest).\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1723677%2F38e92cf743cbb494f5abcc3f1b240381%2Ftf.png?generation=1597500571867037&alt=media)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1723677%2Fd9ac2e00017ff7ced90f68af8a9dd230%2Ftf2.png?generation=1597500613319783&alt=media)",
    "971841": "I had tried downgrade but get issue on colab. Let try on kaggle. With TPU, It always use latest enviroment.\nI think bc they upgrade TF 2.3 too. But my LB score decrease 0.04 too :(",
    "971873": "EfficientNet TPU is not working correctly in TF2.3. Kaggle is currently TF2.2, so things are working well at Kaggle now."
  },
  "source": "meta"
}