{
  "id": 203374,
  "title": "Tip to speed up training 2x in Keras",
  "url": "/competitions/cassava-leaf-disease-classification/discussion/203374",
  "author_name": "",
  "post_date": "2020-12-15T01:00:45.563335500Z",
  "votes": 10,
  "comment_count": 8,
  "views": 0,
  "content": "<p>Enable mixed precision, read more about it here: <a href=\"https://www.tensorflow.org/guide/mixed_precision\" target=\"_blank\">https://www.tensorflow.org/guide/mixed_precision</a></p>\n<p>Include the following lines in your code:</p>\n<p>from tensorflow.keras.mixed_precision import experimental as mixed_precision<br>\npolicy = tf.keras.mixed_precision.experimental.Policy('mixed_float16')<br>\nmixed_precision.set_policy(policy)</p>\n<p>And include dtype=\"float32\" in your activation layer if you're using softmax.</p>\n<p>I only know how to do this for Keras, I'm unfamiliar with Pytorch, sorry! Hope this helps.</p>",
  "messages": [
    {
      "id": "1112881",
      "postDate": "12/15/2020 01:00:45",
      "content": "<p>Enable mixed precision, read more about it here: <a href=\"https://www.tensorflow.org/guide/mixed_precision\" target=\"_blank\">https://www.tensorflow.org/guide/mixed_precision</a></p>\n<p>Include the following lines in your code:</p>\n<p>from tensorflow.keras.mixed_precision import experimental as mixed_precision<br>\npolicy = tf.keras.mixed_precision.experimental.Policy('mixed_float16')<br>\nmixed_precision.set_policy(policy)</p>\n<p>And include dtype=\"float32\" in your activation layer if you're using softmax.</p>\n<p>I only know how to do this for Keras, I'm unfamiliar with Pytorch, sorry! Hope this helps.</p>",
      "rawMarkdown": "Enable mixed precision, read more about it here: https://www.tensorflow.org/guide/mixed_precision\n\nInclude the following lines in your code:\n\nfrom tensorflow.keras.mixed_precision import experimental as mixed_precision\npolicy = tf.keras.mixed_precision.experimental.Policy('mixed_float16')\nmixed_precision.set_policy(policy)\n\nAnd include dtype=\"float32\" in your activation layer if you're using softmax.\n\nI only know how to do this for Keras, I'm unfamiliar with Pytorch, sorry! Hope this helps.",
      "votes": null
    },
    {
      "id": "1113125",
      "postDate": "12/15/2020 07:49:19",
      "content": "<p>I was wondering if people have tried <code>channel_first</code> instead of the default <code>channel_last</code>? From tensorflow documentations, GPU should use <code>channel_first</code> for speed up but I couldn't make it work for efficientnet.</p>",
      "rawMarkdown": "I was wondering if people have tried `channel_first` instead of the default `channel_last`? From tensorflow documentations, GPU should use `channel_first` for speed up but I couldn't make it work for efficientnet.",
      "votes": null
    },
    {
      "id": "1113377",
      "postDate": "12/15/2020 12:05:41",
      "content": "<p>Will surely give this a try and let you know!</p>",
      "rawMarkdown": "Will surely give this a try and let you know!",
      "votes": null
    },
    {
      "id": "1113452",
      "postDate": "12/15/2020 12:53:47",
      "content": "<p>To alse use XLA:</p>\n<pre><code># @https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96\n\nMIXED_PRECISION = False\nXLA_ACCELERATE = False\n\nif MIXED_PRECISION:\n    from tensorflow.keras.mixed_precision import experimental as mixed_precision\n    if tpu: policy = tf.keras.mixed_precision.experimental.Policy('mixed_bfloat16')\n    else: policy = tf.keras.mixed_precision.experimental.Policy('mixed_float16')\n    mixed_precision.set_policy(policy)\n    print('Mixed precision enabled')\n\nif XLA_ACCELERATE:\n    tf.config.optimizer.set_jit(True)\n    print('Accelerated Linear Algebra enabled')\n</code></pre>",
      "rawMarkdown": "To alse use XLA:\n\n```\n# @https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96\n\nMIXED_PRECISION = False\nXLA_ACCELERATE = False\n\nif MIXED_PRECISION:\n    from tensorflow.keras.mixed_precision import experimental as mixed_precision\n    if tpu: policy = tf.keras.mixed_precision.experimental.Policy('mixed_bfloat16')\n    else: policy = tf.keras.mixed_precision.experimental.Policy('mixed_float16')\n    mixed_precision.set_policy(policy)\n    print('Mixed precision enabled')\n\nif XLA_ACCELERATE:\n    tf.config.optimizer.set_jit(True)\n    print('Accelerated Linear Algebra enabled')\n```",
      "votes": null
    },
    {
      "id": "1119562",
      "postDate": "12/20/2020 07:19:49",
      "content": "<p>I am not sure if I did this right, but it did not seem to improve my training time…</p>",
      "rawMarkdown": "I am not sure if I did this right, but it did not seem to improve my training time...",
      "votes": null
    },
    {
      "id": "1119806",
      "postDate": "12/20/2020 11:30:46",
      "content": "<p>I think a lot of other things are also needed to done, like you need to cast the image to float16 using tf.cast while reading the image and then I am also not sure that the weights you load for efficientnet or any other arch they also need to be fp16.</p>",
      "rawMarkdown": "I think a lot of other things are also needed to done, like you need to cast the image to float16 using tf.cast while reading the image and then I am also not sure that the weights you load for efficientnet or any other arch they also need to be fp16.",
      "votes": null
    },
    {
      "id": "1122013",
      "postDate": "12/22/2020 06:11:23",
      "content": "<p>The speed improvement comes from being able to double your batch size.  </p>\n<p>Almost always I can multiply my current batch by 2.  </p>",
      "rawMarkdown": "The speed improvement comes from being able to double your batch size.  \n\nAlmost always I can multiply my current batch by 2.",
      "votes": null
    },
    {
      "id": "1122023",
      "postDate": "12/22/2020 06:19:13",
      "content": "<p>When you save the model shouldn’t the size of model be also reduced to half? I tried in my case the size is the same?</p>",
      "rawMarkdown": "When you save the model shouldn’t the size of model be also reduced to half? I tried in my case the size is the same?",
      "votes": null
    },
    {
      "id": "1122036",
      "postDate": "12/22/2020 06:29:43",
      "content": "<p>Hmmm. Not sure about saved model size as I never worry about that factor.  </p>",
      "rawMarkdown": "Hmmm. Not sure about saved model size as I never worry about that factor.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1113125,
      "author_name": "louis925",
      "author_url": "",
      "post_date": "12/15/2020 07:49:19",
      "content": "<p>I was wondering if people have tried <code>channel_first</code> instead of the default <code>channel_last</code>? From tensorflow documentations, GPU should use <code>channel_first</code> for speed up but I couldn't make it work for efficientnet.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1113377,
      "author_name": "harveenchadha",
      "author_url": "",
      "post_date": "12/15/2020 12:05:41",
      "content": "<p>Will surely give this a try and let you know!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1113452,
      "author_name": "mightyrains",
      "author_url": "",
      "post_date": "12/15/2020 12:53:47",
      "content": "<p>To alse use XLA:</p>\n<pre><code># @https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96\n\nMIXED_PRECISION = False\nXLA_ACCELERATE = False\n\nif MIXED_PRECISION:\n    from tensorflow.keras.mixed_precision import experimental as mixed_precision\n    if tpu: policy = tf.keras.mixed_precision.experimental.Policy('mixed_bfloat16')\n    else: policy = tf.keras.mixed_precision.experimental.Policy('mixed_float16')\n    mixed_precision.set_policy(policy)\n    print('Mixed precision enabled')\n\nif XLA_ACCELERATE:\n    tf.config.optimizer.set_jit(True)\n    print('Accelerated Linear Algebra enabled')\n</code></pre>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1119562,
      "author_name": "viswajithkn",
      "author_url": "",
      "post_date": "12/20/2020 07:19:49",
      "content": "<p>I am not sure if I did this right, but it did not seem to improve my training time…</p>",
      "votes": null,
      "replies": [
        {
          "id": 1122013,
          "author_name": "pcjimmmy",
          "author_url": "",
          "post_date": "12/22/2020 06:11:23",
          "content": "<p>The speed improvement comes from being able to double your batch size.  </p>\n<p>Almost always I can multiply my current batch by 2.  </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1122023,
          "author_name": "harveenchadha",
          "author_url": "",
          "post_date": "12/22/2020 06:19:13",
          "content": "<p>When you save the model shouldn’t the size of model be also reduced to half? I tried in my case the size is the same?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1122036,
          "author_name": "pcjimmmy",
          "author_url": "",
          "post_date": "12/22/2020 06:29:43",
          "content": "<p>Hmmm. Not sure about saved model size as I never worry about that factor.  </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1119806,
      "author_name": "harveenchadha",
      "author_url": "",
      "post_date": "12/20/2020 11:30:46",
      "content": "<p>I think a lot of other things are also needed to done, like you need to cast the image to float16 using tf.cast while reading the image and then I am also not sure that the weights you load for efficientnet or any other arch they also need to be fp16.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1112881": "Enable mixed precision, read more about it here: https://www.tensorflow.org/guide/mixed_precision\n\nInclude the following lines in your code:\n\nfrom tensorflow.keras.mixed_precision import experimental as mixed_precision\npolicy = tf.keras.mixed_precision.experimental.Policy('mixed_float16')\nmixed_precision.set_policy(policy)\n\nAnd include dtype=\"float32\" in your activation layer if you're using softmax.\n\nI only know how to do this for Keras, I'm unfamiliar with Pytorch, sorry! Hope this helps.",
    "1113125": "I was wondering if people have tried `channel_first` instead of the default `channel_last`? From tensorflow documentations, GPU should use `channel_first` for speed up but I couldn't make it work for efficientnet.",
    "1113377": "Will surely give this a try and let you know!",
    "1113452": "To alse use XLA:\n\n```\n# @https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96\n\nMIXED_PRECISION = False\nXLA_ACCELERATE = False\n\nif MIXED_PRECISION:\n    from tensorflow.keras.mixed_precision import experimental as mixed_precision\n    if tpu: policy = tf.keras.mixed_precision.experimental.Policy('mixed_bfloat16')\n    else: policy = tf.keras.mixed_precision.experimental.Policy('mixed_float16')\n    mixed_precision.set_policy(policy)\n    print('Mixed precision enabled')\n\nif XLA_ACCELERATE:\n    tf.config.optimizer.set_jit(True)\n    print('Accelerated Linear Algebra enabled')\n```",
    "1119562": "I am not sure if I did this right, but it did not seem to improve my training time...",
    "1119806": "I think a lot of other things are also needed to done, like you need to cast the image to float16 using tf.cast while reading the image and then I am also not sure that the weights you load for efficientnet or any other arch they also need to be fp16.",
    "1122013": "The speed improvement comes from being able to double your batch size.  \n\nAlmost always I can multiply my current batch by 2.",
    "1122023": "When you save the model shouldn’t the size of model be also reduced to half? I tried in my case the size is the same?",
    "1122036": "Hmmm. Not sure about saved model size as I never worry about that factor."
  },
  "source": "meta"
}