{
  "id": 157050,
  "title": "Focal loss nan values",
  "url": "/competitions/siim-isic-melanoma-classification/discussion/157050",
  "author_name": "",
  "post_date": "2020-06-09T05:22:49.411717400Z",
  "votes": -2,
  "comment_count": 3,
  "views": 0,
  "content": "<p>while training the focal loss is becoming nan, I found that if gamma is less than 1 then focal loss may take large values eventually becomes nan but I am using gamma as 2 and alpha 0.25( because  it works best for image classification according to many articles). I have tried adding K.epsilon still same issue.Can  anyone suggest how to fine tune alpha ? .  Few details of model I am using \nEffecientnetB7 , metric -&gt; AUC , weights-&gt; noisy students , adam optimizer and reducing learning rate on plateu</p>",
  "messages": [
    {
      "id": "878993",
      "postDate": "06/09/2020 05:22:49",
      "content": "<p>while training the focal loss is becoming nan, I found that if gamma is less than 1 then focal loss may take large values eventually becomes nan but I am using gamma as 2 and alpha 0.25( because  it works best for image classification according to many articles). I have tried adding K.epsilon still same issue.Can  anyone suggest how to fine tune alpha ? .  Few details of model I am using \nEffecientnetB7 , metric -&gt; AUC , weights-&gt; noisy students , adam optimizer and reducing learning rate on plateu</p>",
      "rawMarkdown": "while training the focal loss is becoming nan, I found that if gamma is less than 1 then focal loss may take large values eventually becomes nan but I am using gamma as 2 and alpha 0.25( because  it works best for image classification according to many articles). I have tried adding K.epsilon still same issue.Can  anyone suggest how to fine tune alpha ? .  Few details of model I am using \nEffecientnetB7 , metric -&gt; AUC , weights-&gt; noisy students , adam optimizer and reducing learning rate on plateu",
      "votes": null
    },
    {
      "id": "879002",
      "postDate": "06/09/2020 05:30:09",
      "content": "<p>Try to use clipnorm.</p>",
      "rawMarkdown": "Try to use clipnorm.",
      "votes": null
    },
    {
      "id": "879006",
      "postDate": "06/09/2020 05:33:31",
      "content": "<p>I had the same issue while training using Focal Loss on a TPU. It didn't always happen though and I discovered from testing, and an answer from another competition, that it was a batch size issue.  The code below, using drop_remainder, solved it.</p>\n\n<p><code>dataset = dataset.batch(BATCH_SIZE, drop_remainder=True)</code></p>\n\n<p>Also, the focal loss code I am using is supposed to fix NaNs and INFs but it doesn't deal with this issue.</p>",
      "rawMarkdown": "I had the same issue while training using Focal Loss on a TPU. It didn't always happen though and I discovered from testing, and an answer from another competition, that it was a batch size issue.  The code below, using drop_remainder, solved it.\n\n`dataset = dataset.batch(BATCH_SIZE, drop_remainder=True)`\n\nAlso, the focal loss code I am using is supposed to fix NaNs and INFs but it doesn't deal with this issue.",
      "votes": null
    },
    {
      "id": "879138",
      "postDate": "06/09/2020 08:20:20",
      "content": "<p>thanks , using clipnorm helped</p>",
      "rawMarkdown": "thanks , using clipnorm helped",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 879002,
      "author_name": "benboren",
      "author_url": "",
      "post_date": "06/09/2020 05:30:09",
      "content": "<p>Try to use clipnorm.</p>",
      "votes": null,
      "replies": [
        {
          "id": 879138,
          "author_name": "priyt00",
          "author_url": "",
          "post_date": "06/09/2020 08:20:20",
          "content": "<p>thanks , using clipnorm helped</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 879006,
      "author_name": "mutantspore",
      "author_url": "",
      "post_date": "06/09/2020 05:33:31",
      "content": "<p>I had the same issue while training using Focal Loss on a TPU. It didn't always happen though and I discovered from testing, and an answer from another competition, that it was a batch size issue.  The code below, using drop_remainder, solved it.</p>\n\n<p><code>dataset = dataset.batch(BATCH_SIZE, drop_remainder=True)</code></p>\n\n<p>Also, the focal loss code I am using is supposed to fix NaNs and INFs but it doesn't deal with this issue.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "878993": "while training the focal loss is becoming nan, I found that if gamma is less than 1 then focal loss may take large values eventually becomes nan but I am using gamma as 2 and alpha 0.25( because  it works best for image classification according to many articles). I have tried adding K.epsilon still same issue.Can  anyone suggest how to fine tune alpha ? .  Few details of model I am using \nEffecientnetB7 , metric -&gt; AUC , weights-&gt; noisy students , adam optimizer and reducing learning rate on plateu",
    "879002": "Try to use clipnorm.",
    "879006": "I had the same issue while training using Focal Loss on a TPU. It didn't always happen though and I discovered from testing, and an answer from another competition, that it was a batch size issue.  The code below, using drop_remainder, solved it.\n\n`dataset = dataset.batch(BATCH_SIZE, drop_remainder=True)`\n\nAlso, the focal loss code I am using is supposed to fix NaNs and INFs but it doesn't deal with this issue.",
    "879138": "thanks , using clipnorm helped"
  },
  "source": "meta"
}