{
  "id": 214411,
  "title": "NaN while training with TPUs",
  "url": "/competitions/cassava-leaf-disease-classification/discussion/214411",
  "author_name": "",
  "post_date": "2021-01-26T14:55:49.815904300Z",
  "votes": 2,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Hello all,<br>\nDid anybody ever encounter a NaN while using pytorch XLA to train a model on TPU which works perfectly well on a GPU?<br>\nThis is my first time to deal with a TPU, that too in pytorch. <br>\nYou can find my kernel <a href=\"https://www.kaggle.com/saimanojakondi/efficientnetb5?scriptVersionId=52775614\" target=\"_blank\">here</a>.<br>\nI tried mimicking <a href=\"https://www.kaggle.com/mobassir\" target=\"_blank\">@mobassir</a> 's notebook!<br>\nThanks in Advance.</p>",
  "messages": [
    {
      "id": "1170955",
      "postDate": "01/26/2021 14:55:49",
      "content": "<p>Hello all,<br>\nDid anybody ever encounter a NaN while using pytorch XLA to train a model on TPU which works perfectly well on a GPU?<br>\nThis is my first time to deal with a TPU, that too in pytorch. <br>\nYou can find my kernel <a href=\"https://www.kaggle.com/saimanojakondi/efficientnetb5?scriptVersionId=52775614\" target=\"_blank\">here</a>.<br>\nI tried mimicking <a href=\"https://www.kaggle.com/mobassir\" target=\"_blank\">@mobassir</a> 's notebook!<br>\nThanks in Advance.</p>",
      "rawMarkdown": "Hello all,\nDid anybody ever encounter a NaN while using pytorch XLA to train a model on TPU which works perfectly well on a GPU?\nThis is my first time to deal with a TPU, that too in pytorch. \nYou can find my kernel [here](https://www.kaggle.com/saimanojakondi/efficientnetb5?scriptVersionId=52775614).\nI tried mimicking @mobassir 's notebook!\nThanks in Advance.",
      "votes": null
    },
    {
      "id": "1171249",
      "postDate": "01/26/2021 18:14:23",
      "content": "<p><code>os.environ[\"XLA_USE_BF16\"] = \"1\"</code></p>\n<p>Comment this line and try again. Let me know if it helps.</p>",
      "rawMarkdown": "`os.environ[\"XLA_USE_BF16\"] = \"1\"`\n\nComment this line and try again. Let me know if it helps.",
      "votes": null
    },
    {
      "id": "1171309",
      "postDate": "01/26/2021 19:09:16",
      "content": "<p>wow!! thanks man!!<br>\nYou saved me!! and by the way what does that line mean?</p>",
      "rawMarkdown": "wow!! thanks man!!\nYou saved me!! and by the way what does that line mean?",
      "votes": null
    },
    {
      "id": "1172612",
      "postDate": "01/27/2021 13:10:52",
      "content": "<p>It enables the mixed precision training on TPU. </p>",
      "rawMarkdown": "It enables the mixed precision training on TPU.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1171249,
      "author_name": "joshi98kishan",
      "author_url": "",
      "post_date": "01/26/2021 18:14:23",
      "content": "<p><code>os.environ[\"XLA_USE_BF16\"] = \"1\"</code></p>\n<p>Comment this line and try again. Let me know if it helps.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1171309,
          "author_name": "saimanojakondi",
          "author_url": "",
          "post_date": "01/26/2021 19:09:16",
          "content": "<p>wow!! thanks man!!<br>\nYou saved me!! and by the way what does that line mean?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1172612,
          "author_name": "joshi98kishan",
          "author_url": "",
          "post_date": "01/27/2021 13:10:52",
          "content": "<p>It enables the mixed precision training on TPU. </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1170955": "Hello all,\nDid anybody ever encounter a NaN while using pytorch XLA to train a model on TPU which works perfectly well on a GPU?\nThis is my first time to deal with a TPU, that too in pytorch. \nYou can find my kernel [here](https://www.kaggle.com/saimanojakondi/efficientnetb5?scriptVersionId=52775614).\nI tried mimicking @mobassir 's notebook!\nThanks in Advance.",
    "1171249": "`os.environ[\"XLA_USE_BF16\"] = \"1\"`\n\nComment this line and try again. Let me know if it helps.",
    "1171309": "wow!! thanks man!!\nYou saved me!! and by the way what does that line mean?",
    "1172612": "It enables the mixed precision training on TPU."
  },
  "source": "meta"
}