{
  "id": 169897,
  "title": "mixed precision with TPU tf.distrubute?",
  "url": "/competitions/siim-isic-melanoma-classification/discussion/169897",
  "author_name": "",
  "post_date": "2020-07-25T15:32:14.552816300Z",
  "votes": 3,
  "comment_count": 6,
  "views": 0,
  "content": "<p>I'm getting the following error while I set</p>\n\n<p><code>tf.keras.mixed_precision.experimental.set_policy('mixed_float16')</code></p>\n\n<p>ValueError: Mixed precision is not supported with the tf.distribute.Strategy: TPUStrategy. Either stop using mixed precision by removing the use of the \"mixed_float16\" policy or use a different Strategy, e.g. a MirroredStrategy.</p>\n\n<p>Question: has anyone been able to setup mixed precision for TPU's? If so please share. I'm currently using tensorflow.</p>",
  "messages": [
    {
      "id": "945129",
      "postDate": "07/25/2020 15:32:14",
      "content": "<p>I'm getting the following error while I set</p>\n\n<p><code>tf.keras.mixed_precision.experimental.set_policy('mixed_float16')</code></p>\n\n<p>ValueError: Mixed precision is not supported with the tf.distribute.Strategy: TPUStrategy. Either stop using mixed precision by removing the use of the \"mixed_float16\" policy or use a different Strategy, e.g. a MirroredStrategy.</p>\n\n<p>Question: has anyone been able to setup mixed precision for TPU's? If so please share. I'm currently using tensorflow.</p>",
      "rawMarkdown": "I'm getting the following error while I set\n\n`tf.keras.mixed_precision.experimental.set_policy('mixed_float16')`\n\nValueError: Mixed precision is not supported with the tf.distribute.Strategy: TPUStrategy. Either stop using mixed precision by removing the use of the \"mixed_float16\" policy or use a different Strategy, e.g. a MirroredStrategy.\n\nQuestion: has anyone been able to setup mixed precision for TPU's? If so please share. I'm currently using tensorflow.",
      "votes": null
    },
    {
      "id": "945146",
      "postDate": "07/25/2020 15:41:00",
      "content": "<p>See Chris's <a href=\"https://www.kaggle.com/cdeotte/cutmix-and-mixup-on-gpu-tpu\">CutMix and MixUp kernel</a> using mixed_float16</p>",
      "rawMarkdown": "See Chris's [CutMix and MixUp kernel](https://www.kaggle.com/cdeotte/cutmix-and-mixup-on-gpu-tpu) using mixed_float16",
      "votes": null
    },
    {
      "id": "945440",
      "postDate": "07/25/2020 20:01:44",
      "content": "<p>If you're using TPU you must use <code>tf.keras.mixed_precision.experimental.Policy('mixed_bfloat16')</code>. This makes the TPU go faster and allows for larger TPU batch sizes.</p>\n\n<p>Note this says <code>bfloat16</code> whereas your error says <code>float16</code>. If you are using GPU (not TPU) then you would do <code>mixed_float16</code> and then your GPU will go faster and allow larger batch sizes. This will only accelerate GPUs with tensor cores so it does not speed up Kaggle's P100 GPU</p>",
      "rawMarkdown": "If you're using TPU you must use `tf.keras.mixed_precision.experimental.Policy('mixed_bfloat16')`. This makes the TPU go faster and allows for larger TPU batch sizes.\n\nNote this says `bfloat16` whereas your error says `float16`. If you are using GPU (not TPU) then you would do `mixed_float16` and then your GPU will go faster and allow larger batch sizes. This will only accelerate GPUs with tensor cores so it does not speed up Kaggle's P100 GPU",
      "votes": null
    },
    {
      "id": "965612",
      "postDate": "08/10/2020 18:22:58",
      "content": "<p>Hi <a href=\"/cdeotte\">@cdeotte</a> i had set the mixprecision with policy = mixed_precision.Policy('mixed_bfloat16')\n            keras.mixed_precision.experimental.set_policy(policy)\nbut it gives an error TypeError: Input 'y' of 'Mul' Op has type float32 that does not match type bfloat16 of argument 'x'.   i trace this error, it coms from BCE loss, and my BCE loss use  this code :\nif LOSS_TYPE.upper() == 'BCE':\n            loss = keras.losses.BinaryCrossentropy(**LOSS_PARAMS)\nelif LOSS_TYPE.upper() == 'FOCAL':\n            loss = losses.BinaryFocalLoss(**LOSS_PARAMS)</p>\n\n<p>Do you know how to fix?</p>",
      "rawMarkdown": "Hi @cdeotte i had set the mixprecision with policy = mixed_precision.Policy('mixed_bfloat16')\n            keras.mixed_precision.experimental.set_policy(policy)\nbut it gives an error TypeError: Input 'y' of 'Mul' Op has type float32 that does not match type bfloat16 of argument 'x'.   i trace this error, it coms from BCE loss, and my BCE loss use  this code :\nif LOSS_TYPE.upper() == 'BCE':\n            loss = keras.losses.BinaryCrossentropy(**LOSS_PARAMS)\nelif LOSS_TYPE.upper() == 'FOCAL':\n            loss = losses.BinaryFocalLoss(**LOSS_PARAMS)\n\nDo you know how to fix?",
      "votes": null
    },
    {
      "id": "965627",
      "postDate": "08/10/2020 18:26:47",
      "content": "<p>weird, i have never seen that nor had a problem using mixed precision with TPU. Try using default binary crossentropy without your additional settings, i.e. <code>loss = keras.losses.BinaryCrossentropy()</code>, and make sure you're using BCE and not Focal loss for debugging purposes.</p>",
      "rawMarkdown": "weird, i have never seen that nor had a problem using mixed precision with TPU. Try using default binary crossentropy without your additional settings, i.e. `loss = keras.losses.BinaryCrossentropy()`, and make sure you're using BCE and not Focal loss for debugging purposes.",
      "votes": null
    },
    {
      "id": "965630",
      "postDate": "08/10/2020 18:28:17",
      "content": "<p>Actually it may be your model's last layer, you need to cast the output to <code>float32</code> like this</p>\n\n<pre><code>x = tf.keras.layers.Dense(1,activation='sigmoid',dtype='float32')(x)\n</code></pre>",
      "rawMarkdown": "Actually it may be your model's last layer, you need to cast the output to `float32` like this\n\n    x = tf.keras.layers.Dense(1,activation='sigmoid',dtype='float32')(x)",
      "votes": null
    },
    {
      "id": "965944",
      "postDate": "08/11/2020 02:03:11",
      "content": "<p>Thank you <a href=\"/cdeotte\">@cdeotte</a> I fixed the problem as you said.👍 😃 </p>",
      "rawMarkdown": "Thank you @cdeotte I fixed the problem as you said.👍 😃",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 945146,
      "author_name": "sirishks",
      "author_url": "",
      "post_date": "07/25/2020 15:41:00",
      "content": "<p>See Chris's <a href=\"https://www.kaggle.com/cdeotte/cutmix-and-mixup-on-gpu-tpu\">CutMix and MixUp kernel</a> using mixed_float16</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 945440,
      "author_name": "cdeotte",
      "author_url": "",
      "post_date": "07/25/2020 20:01:44",
      "content": "<p>If you're using TPU you must use <code>tf.keras.mixed_precision.experimental.Policy('mixed_bfloat16')</code>. This makes the TPU go faster and allows for larger TPU batch sizes.</p>\n\n<p>Note this says <code>bfloat16</code> whereas your error says <code>float16</code>. If you are using GPU (not TPU) then you would do <code>mixed_float16</code> and then your GPU will go faster and allow larger batch sizes. This will only accelerate GPUs with tensor cores so it does not speed up Kaggle's P100 GPU</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 965612,
      "author_name": "cswwp347724",
      "author_url": "",
      "post_date": "08/10/2020 18:22:58",
      "content": "<p>Hi <a href=\"/cdeotte\">@cdeotte</a> i had set the mixprecision with policy = mixed_precision.Policy('mixed_bfloat16')\n            keras.mixed_precision.experimental.set_policy(policy)\nbut it gives an error TypeError: Input 'y' of 'Mul' Op has type float32 that does not match type bfloat16 of argument 'x'.   i trace this error, it coms from BCE loss, and my BCE loss use  this code :\nif LOSS_TYPE.upper() == 'BCE':\n            loss = keras.losses.BinaryCrossentropy(**LOSS_PARAMS)\nelif LOSS_TYPE.upper() == 'FOCAL':\n            loss = losses.BinaryFocalLoss(**LOSS_PARAMS)</p>\n\n<p>Do you know how to fix?</p>",
      "votes": null,
      "replies": [
        {
          "id": 965627,
          "author_name": "cdeotte",
          "author_url": "",
          "post_date": "08/10/2020 18:26:47",
          "content": "<p>weird, i have never seen that nor had a problem using mixed precision with TPU. Try using default binary crossentropy without your additional settings, i.e. <code>loss = keras.losses.BinaryCrossentropy()</code>, and make sure you're using BCE and not Focal loss for debugging purposes.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 965630,
          "author_name": "cdeotte",
          "author_url": "",
          "post_date": "08/10/2020 18:28:17",
          "content": "<p>Actually it may be your model's last layer, you need to cast the output to <code>float32</code> like this</p>\n\n<pre><code>x = tf.keras.layers.Dense(1,activation='sigmoid',dtype='float32')(x)\n</code></pre>",
          "votes": null,
          "replies": []
        },
        {
          "id": 965944,
          "author_name": "cswwp347724",
          "author_url": "",
          "post_date": "08/11/2020 02:03:11",
          "content": "<p>Thank you <a href=\"/cdeotte\">@cdeotte</a> I fixed the problem as you said.👍 😃 </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "945129": "I'm getting the following error while I set\n\n`tf.keras.mixed_precision.experimental.set_policy('mixed_float16')`\n\nValueError: Mixed precision is not supported with the tf.distribute.Strategy: TPUStrategy. Either stop using mixed precision by removing the use of the \"mixed_float16\" policy or use a different Strategy, e.g. a MirroredStrategy.\n\nQuestion: has anyone been able to setup mixed precision for TPU's? If so please share. I'm currently using tensorflow.",
    "945146": "See Chris's [CutMix and MixUp kernel](https://www.kaggle.com/cdeotte/cutmix-and-mixup-on-gpu-tpu) using mixed_float16",
    "945440": "If you're using TPU you must use `tf.keras.mixed_precision.experimental.Policy('mixed_bfloat16')`. This makes the TPU go faster and allows for larger TPU batch sizes.\n\nNote this says `bfloat16` whereas your error says `float16`. If you are using GPU (not TPU) then you would do `mixed_float16` and then your GPU will go faster and allow larger batch sizes. This will only accelerate GPUs with tensor cores so it does not speed up Kaggle's P100 GPU",
    "965612": "Hi @cdeotte i had set the mixprecision with policy = mixed_precision.Policy('mixed_bfloat16')\n            keras.mixed_precision.experimental.set_policy(policy)\nbut it gives an error TypeError: Input 'y' of 'Mul' Op has type float32 that does not match type bfloat16 of argument 'x'.   i trace this error, it coms from BCE loss, and my BCE loss use  this code :\nif LOSS_TYPE.upper() == 'BCE':\n            loss = keras.losses.BinaryCrossentropy(**LOSS_PARAMS)\nelif LOSS_TYPE.upper() == 'FOCAL':\n            loss = losses.BinaryFocalLoss(**LOSS_PARAMS)\n\nDo you know how to fix?",
    "965627": "weird, i have never seen that nor had a problem using mixed precision with TPU. Try using default binary crossentropy without your additional settings, i.e. `loss = keras.losses.BinaryCrossentropy()`, and make sure you're using BCE and not Focal loss for debugging purposes.",
    "965630": "Actually it may be your model's last layer, you need to cast the output to `float32` like this\n\n    x = tf.keras.layers.Dense(1,activation='sigmoid',dtype='float32')(x)",
    "965944": "Thank you @cdeotte I fixed the problem as you said.👍 😃"
  },
  "source": "meta"
}