{
  "id": 138956,
  "title": "Not able to train BERT using TPU",
  "url": "/competitions/jigsaw-multilingual-toxic-comment-classification/discussion/138956",
  "author_name": "",
  "post_date": "2020-03-26T22:07:05.299971400Z",
  "votes": 5,
  "comment_count": 4,
  "views": 0,
  "content": "<p>I am trying to train BERT (!wget -q <a href=\"https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-12_H-768_A-12.zip\">https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-12_H-768_A-12.zip</a>) using transfer learning method but my BERT model is not detecting TPU memoory. While training my model i can see TPU usage is 0%</p>\n\n<p>```\nprint(\"Tensorflow version \" + tf.<strong>version</strong>)\nAUTO = tf.data.experimental.AUTOTUNE</p>\n\n<p>try:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()  # TPU detection. No parameters necessary if TPU_NAME environment variable is set. On Kaggle this is always the case.\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None</p>\n\n<p>if tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy() # default distribution strategy in Tensorflow. Works on CPU and single GPU.</p>\n\n<p>print(\"REPLICAS: \", strategy.num_replicas_in_sync)</p>\n\n<p>GCS_DS_PATH = KaggleDatasets().get_gcs_path()</p>\n\n<p>def build_model():\n    from keras.layers.normalization import BatchNormalization\n    model = load_trained_model_from_checkpoint(\n        config_path,\n        checkpoint_path,\n        training=True,\n        trainable=True,\n        seq_len=SEQ_LEN,\n    )</p>\n\n<pre><code>inputs = model.inputs[:2]\ndense = model.layers[-3].output\ndense1 = keras.layers.Dense(100,activation='relu')(dense)\nNorm1 = BatchNormalization()(dense1)\ndense2 = keras.layers.Dense(50,activation='relu')(Norm1)\nNorm2 = BatchNormalization()(dense2)\noutputs = keras.layers.Dense(1, activation='sigmoid', kernel_initializer=keras.initializers.TruncatedNormal(stddev=0.03),\n                             name = 'real_output')(dense)\n\n\n\nmodel = keras.models.Model(inputs, outputs)\n\nreturn model\n</code></pre>\n\n<p>with strategy.scope():\n    model = build_model()\nmodel.summary()\ndecay_steps, warmup_steps = calc_train_steps(Y_train.shape[0],batch_size=BATCH_SIZE,epochs=EPOCHS,)\nmodel.compile(AdamWarmup(decay_steps=decay_steps, warmup_steps=warmup_steps, lr=LR),loss='binary_crossentropy',metrics=['accuracy'])</p>\n\n<p>BERT = model.fit(\n        X_train,\n        Y_train,\n        epochs=2,\n        batch_size=BATCH_SIZE,\n        validation_data=(X_valid,Y_valid),\n        callbacks=[reduce_lr]\n    )\n```</p>",
  "messages": [
    {
      "id": "787563",
      "postDate": "03/26/2020 22:07:05",
      "content": "<p>I am trying to train BERT (!wget -q <a href=\"https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-12_H-768_A-12.zip\">https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-12_H-768_A-12.zip</a>) using transfer learning method but my BERT model is not detecting TPU memoory. While training my model i can see TPU usage is 0%</p>\n\n<p>```\nprint(\"Tensorflow version \" + tf.<strong>version</strong>)\nAUTO = tf.data.experimental.AUTOTUNE</p>\n\n<p>try:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()  # TPU detection. No parameters necessary if TPU_NAME environment variable is set. On Kaggle this is always the case.\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None</p>\n\n<p>if tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy() # default distribution strategy in Tensorflow. Works on CPU and single GPU.</p>\n\n<p>print(\"REPLICAS: \", strategy.num_replicas_in_sync)</p>\n\n<p>GCS_DS_PATH = KaggleDatasets().get_gcs_path()</p>\n\n<p>def build_model():\n    from keras.layers.normalization import BatchNormalization\n    model = load_trained_model_from_checkpoint(\n        config_path,\n        checkpoint_path,\n        training=True,\n        trainable=True,\n        seq_len=SEQ_LEN,\n    )</p>\n\n<pre><code>inputs = model.inputs[:2]\ndense = model.layers[-3].output\ndense1 = keras.layers.Dense(100,activation='relu')(dense)\nNorm1 = BatchNormalization()(dense1)\ndense2 = keras.layers.Dense(50,activation='relu')(Norm1)\nNorm2 = BatchNormalization()(dense2)\noutputs = keras.layers.Dense(1, activation='sigmoid', kernel_initializer=keras.initializers.TruncatedNormal(stddev=0.03),\n                             name = 'real_output')(dense)\n\n\n\nmodel = keras.models.Model(inputs, outputs)\n\nreturn model\n</code></pre>\n\n<p>with strategy.scope():\n    model = build_model()\nmodel.summary()\ndecay_steps, warmup_steps = calc_train_steps(Y_train.shape[0],batch_size=BATCH_SIZE,epochs=EPOCHS,)\nmodel.compile(AdamWarmup(decay_steps=decay_steps, warmup_steps=warmup_steps, lr=LR),loss='binary_crossentropy',metrics=['accuracy'])</p>\n\n<p>BERT = model.fit(\n        X_train,\n        Y_train,\n        epochs=2,\n        batch_size=BATCH_SIZE,\n        validation_data=(X_valid,Y_valid),\n        callbacks=[reduce_lr]\n    )\n```</p>",
      "rawMarkdown": "I am trying to train BERT (!wget -q https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-12_H-768_A-12.zip) using transfer learning method but my BERT model is not detecting TPU memoory. While training my model i can see TPU usage is 0%\n\n```\nprint(\"Tensorflow version \" + tf.__version__)\nAUTO = tf.data.experimental.AUTOTUNE\n\n\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()  # TPU detection. No parameters necessary if TPU_NAME environment variable is set. On Kaggle this is always the case.\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy() # default distribution strategy in Tensorflow. Works on CPU and single GPU.\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)\n\nGCS_DS_PATH = KaggleDatasets().get_gcs_path()\n\n\ndef build_model():\n    from keras.layers.normalization import BatchNormalization\n    model = load_trained_model_from_checkpoint(\n        config_path,\n        checkpoint_path,\n        training=True,\n        trainable=True,\n        seq_len=SEQ_LEN,\n    )\n\n    inputs = model.inputs[:2]\n    dense = model.layers[-3].output\n    dense1 = keras.layers.Dense(100,activation='relu')(dense)\n    Norm1 = BatchNormalization()(dense1)\n    dense2 = keras.layers.Dense(50,activation='relu')(Norm1)\n    Norm2 = BatchNormalization()(dense2)\n    outputs = keras.layers.Dense(1, activation='sigmoid', kernel_initializer=keras.initializers.TruncatedNormal(stddev=0.03),\n                                 name = 'real_output')(dense)\n\n    \n\n    model = keras.models.Model(inputs, outputs)\n       \n    return model\n\nwith strategy.scope():\n    model = build_model()\nmodel.summary()\ndecay_steps, warmup_steps = calc_train_steps(Y_train.shape[0],batch_size=BATCH_SIZE,epochs=EPOCHS,)\nmodel.compile(AdamWarmup(decay_steps=decay_steps, warmup_steps=warmup_steps, lr=LR),loss='binary_crossentropy',metrics=['accuracy'])\n\nBERT = model.fit(\n        X_train,\n        Y_train,\n        epochs=2,\n        batch_size=BATCH_SIZE,\n        validation_data=(X_valid,Y_valid),\n        callbacks=[reduce_lr]\n    )\n```",
      "votes": null
    },
    {
      "id": "787697",
      "postDate": "03/27/2020 02:07:47",
      "content": "<p>Try these two notebooks that use BERT:</p>\n\n<p><a href=\"https://www.kaggle.com/kivlichangoogle/jigsaw-multilingual-getting-started\">Getting started notebook</a></p>\n\n<p><a href=\"https://www.kaggle.com/xhlulu/jigsaw-tpu-distilbert-with-huggingface-and-keras\">DistilBERT from HuggingFace</a></p>",
      "rawMarkdown": "Try these two notebooks that use BERT:\n\n[Getting started notebook](https://www.kaggle.com/kivlichangoogle/jigsaw-multilingual-getting-started)\n\n[DistilBERT from HuggingFace](https://www.kaggle.com/xhlulu/jigsaw-tpu-distilbert-with-huggingface-and-keras)",
      "votes": null
    },
    {
      "id": "788475",
      "postDate": "03/27/2020 17:38:38",
      "content": "<p>Should <code>model.compile</code> be under <code>with strategy.scope()</code>? </p>",
      "rawMarkdown": "Should `model.compile` be under `with strategy.scope()`?",
      "votes": null
    },
    {
      "id": "788499",
      "postDate": "03/27/2020 18:01:32",
      "content": "<p>For now yes but it's a bug. You metrics will be a bit off if you call it outside of the scope.</p>",
      "rawMarkdown": "For now yes but it's a bug. You metrics will be a bit off if you call it outside of the scope.",
      "votes": null
    },
    {
      "id": "789639",
      "postDate": "03/28/2020 20:50:37",
      "content": "<p>I tried. But same issue</p>",
      "rawMarkdown": "I tried. But same issue",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 787697,
      "author_name": "mgorner",
      "author_url": "",
      "post_date": "03/27/2020 02:07:47",
      "content": "<p>Try these two notebooks that use BERT:</p>\n\n<p><a href=\"https://www.kaggle.com/kivlichangoogle/jigsaw-multilingual-getting-started\">Getting started notebook</a></p>\n\n<p><a href=\"https://www.kaggle.com/xhlulu/jigsaw-tpu-distilbert-with-huggingface-and-keras\">DistilBERT from HuggingFace</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 788475,
      "author_name": "ifigotin",
      "author_url": "",
      "post_date": "03/27/2020 17:38:38",
      "content": "<p>Should <code>model.compile</code> be under <code>with strategy.scope()</code>? </p>",
      "votes": null,
      "replies": [
        {
          "id": 788499,
          "author_name": "mgorner",
          "author_url": "",
          "post_date": "03/27/2020 18:01:32",
          "content": "<p>For now yes but it's a bug. You metrics will be a bit off if you call it outside of the scope.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 789639,
          "author_name": "ashoksrinivas",
          "author_url": "",
          "post_date": "03/28/2020 20:50:37",
          "content": "<p>I tried. But same issue</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "787563": "I am trying to train BERT (!wget -q https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-12_H-768_A-12.zip) using transfer learning method but my BERT model is not detecting TPU memoory. While training my model i can see TPU usage is 0%\n\n```\nprint(\"Tensorflow version \" + tf.__version__)\nAUTO = tf.data.experimental.AUTOTUNE\n\n\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()  # TPU detection. No parameters necessary if TPU_NAME environment variable is set. On Kaggle this is always the case.\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy() # default distribution strategy in Tensorflow. Works on CPU and single GPU.\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)\n\nGCS_DS_PATH = KaggleDatasets().get_gcs_path()\n\n\ndef build_model():\n    from keras.layers.normalization import BatchNormalization\n    model = load_trained_model_from_checkpoint(\n        config_path,\n        checkpoint_path,\n        training=True,\n        trainable=True,\n        seq_len=SEQ_LEN,\n    )\n\n    inputs = model.inputs[:2]\n    dense = model.layers[-3].output\n    dense1 = keras.layers.Dense(100,activation='relu')(dense)\n    Norm1 = BatchNormalization()(dense1)\n    dense2 = keras.layers.Dense(50,activation='relu')(Norm1)\n    Norm2 = BatchNormalization()(dense2)\n    outputs = keras.layers.Dense(1, activation='sigmoid', kernel_initializer=keras.initializers.TruncatedNormal(stddev=0.03),\n                                 name = 'real_output')(dense)\n\n    \n\n    model = keras.models.Model(inputs, outputs)\n       \n    return model\n\nwith strategy.scope():\n    model = build_model()\nmodel.summary()\ndecay_steps, warmup_steps = calc_train_steps(Y_train.shape[0],batch_size=BATCH_SIZE,epochs=EPOCHS,)\nmodel.compile(AdamWarmup(decay_steps=decay_steps, warmup_steps=warmup_steps, lr=LR),loss='binary_crossentropy',metrics=['accuracy'])\n\nBERT = model.fit(\n        X_train,\n        Y_train,\n        epochs=2,\n        batch_size=BATCH_SIZE,\n        validation_data=(X_valid,Y_valid),\n        callbacks=[reduce_lr]\n    )\n```",
    "787697": "Try these two notebooks that use BERT:\n\n[Getting started notebook](https://www.kaggle.com/kivlichangoogle/jigsaw-multilingual-getting-started)\n\n[DistilBERT from HuggingFace](https://www.kaggle.com/xhlulu/jigsaw-tpu-distilbert-with-huggingface-and-keras)",
    "788475": "Should `model.compile` be under `with strategy.scope()`?",
    "788499": "For now yes but it's a bug. You metrics will be a bit off if you call it outside of the scope.",
    "789639": "I tried. But same issue"
  },
  "source": "meta"
}