{
  "id": 145334,
  "title": "Transformers' schedulers with Keras",
  "url": "/competitions/jigsaw-multilingual-toxic-comment-classification/discussion/145334",
  "author_name": "",
  "post_date": "2020-04-22T18:15:19.523345200Z",
  "votes": 6,
  "comment_count": 9,
  "views": 0,
  "content": "<p>Hello everyone,</p>\n\n<p>I have used almost exclusively Keras to build my models for this competition. I'd like to use the schedulers and optimizers given by the transformers library. Right now, I can't find a way to do it. </p>\n\n<p>Are AdamW or get_linear_schedule_with_warmup only available with PyTorch? </p>\n\n<p>Any help is appreciated ;)</p>\n\n<p>Many thanks in advance</p>",
  "messages": [
    {
      "id": "816940",
      "postDate": "04/22/2020 18:15:19",
      "content": "<p>Hello everyone,</p>\n\n<p>I have used almost exclusively Keras to build my models for this competition. I'd like to use the schedulers and optimizers given by the transformers library. Right now, I can't find a way to do it. </p>\n\n<p>Are AdamW or get_linear_schedule_with_warmup only available with PyTorch? </p>\n\n<p>Any help is appreciated ;)</p>\n\n<p>Many thanks in advance</p>",
      "rawMarkdown": "Hello everyone,\n\nI have used almost exclusively Keras to build my models for this competition. I'd like to use the schedulers and optimizers given by the transformers library. Right now, I can't find a way to do it. \n\nAre AdamW or get_linear_schedule_with_warmup only available with PyTorch? \n\nAny help is appreciated ;)\n\nMany thanks in advance",
      "votes": null
    },
    {
      "id": "817173",
      "postDate": "04/22/2020 22:57:04",
      "content": "<p>Not a direct answer but maybe this can help:</p>\n\n<p>AdamW is available in Tensorflow Addons <a href=\"https://www.tensorflow.org/addons/api_docs/python/tfa/optimizers/AdamW\">here</a>. (I'm not sure TFA is installed on Kaggle yet though, pip installing it on top of the Kaggle container used to be problematic)</p>\n\n<p>The source code of an LR schedule with warmup can be found in the <a href=\"https://www.kaggle.com/mgornergoogle/five-flowers-with-keras-and-xception-on-tpu\">TPU getting started notebook</a>.</p>",
      "rawMarkdown": "Not a direct answer but maybe this can help:\n\nAdamW is available in Tensorflow Addons [here](https://www.tensorflow.org/addons/api_docs/python/tfa/optimizers/AdamW). (I'm not sure TFA is installed on Kaggle yet though, pip installing it on top of the Kaggle container used to be problematic)\n\nThe source code of an LR schedule with warmup can be found in the [TPU getting started notebook](https://www.kaggle.com/mgornergoogle/five-flowers-with-keras-and-xception-on-tpu).",
      "votes": null
    },
    {
      "id": "817815",
      "postDate": "04/23/2020 12:46:54",
      "content": "<p>Thanks a lot !</p>",
      "rawMarkdown": "Thanks a lot !",
      "votes": null
    },
    {
      "id": "817884",
      "postDate": "04/23/2020 13:53:38",
      "content": "<p>The <a href=\"https://github.com/tensorflow/models/blob/master/official/nlp/optimization.py\">official optimizer</a> works with TF2.1. You can copy the file and use the <code>create_optimizer</code> method. <code>num_warmup_steps</code> is usually <code>int(0.1 * num_train_steps)</code>. It also implements many details like gradient clipping and regularization.</p>",
      "rawMarkdown": "The [official optimizer](https://github.com/tensorflow/models/blob/master/official/nlp/optimization.py) works with TF2.1. You can copy the file and use the `create_optimizer` method. `num_warmup_steps` is usually `int(0.1 * num_train_steps)`. It also implements many details like gradient clipping and regularization.",
      "votes": null
    },
    {
      "id": "818322",
      "postDate": "04/23/2020 19:41:23",
      "content": "<p><a href=\"/seesee\">@seesee</a> When I update to TF2.2.0, something went wrong with AdamWeightDecay. Have you experienced same thing?(The reason why asked this is you wrote it works with TF2.1)</p>",
      "rawMarkdown": "seesee When I update to TF2.2.0, something went wrong with AdamWeightDecay. Have you experienced same thing?(The reason why asked this is you wrote it works with TF2.1)",
      "votes": null
    },
    {
      "id": "818356",
      "postDate": "04/23/2020 20:01:13",
      "content": "<p>Yes, I recall that the solver that I posted didn't work with <code>tensorflow:2.2.0rc3</code>. I didn't investigate the error further and decided to stick to 2.1 until the full 2.2 release.</p>\n\n<p>Edit: The fix is <a href=\"https://github.com/tensorflow/addons/pull/1566\">here</a>.</p>",
      "rawMarkdown": "Yes, I recall that the solver that I posted didn't work with `tensorflow:2.2.0rc3`. I didn't investigate the error further and decided to stick to 2.1 until the full 2.2 release.\n\nEdit: The fix is [here](https://github.com/tensorflow/addons/pull/1566).",
      "votes": null
    },
    {
      "id": "820558",
      "postDate": "04/25/2020 14:33:20",
      "content": "<p><a href=\"/seesee\">@seesee</a> Thanks, yeah, that's better choice. I'll wait too:)</p>",
      "rawMarkdown": "seesee Thanks, yeah, that's better choice. I'll wait too:)",
      "votes": null
    },
    {
      "id": "833262",
      "postDate": "05/04/2020 17:37:27",
      "content": "<p>Thanks for your answer See--!</p>",
      "rawMarkdown": "Thanks for your answer See--!",
      "votes": null
    },
    {
      "id": "833263",
      "postDate": "05/04/2020 17:38:44",
      "content": "<p>For your information, I solved my issue by using AdamW from Tensorflow_addons library combined with the create_optimizer method mentioned by <a href=\"/seesee\">@seesee</a> </p>",
      "rawMarkdown": "For your information, I solved my issue by using AdamW from Tensorflow_addons library combined with the create_optimizer method mentioned by @seesee",
      "votes": null
    },
    {
      "id": "834746",
      "postDate": "05/05/2020 18:41:24",
      "content": "<p>Can you post your code ? I though Tensorflow addons could not be installed in a Kaggle Notebook ?</p>",
      "rawMarkdown": "Can you post your code ? I though Tensorflow addons could not be installed in a Kaggle Notebook ?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 817173,
      "author_name": "mgorner",
      "author_url": "",
      "post_date": "04/22/2020 22:57:04",
      "content": "<p>Not a direct answer but maybe this can help:</p>\n\n<p>AdamW is available in Tensorflow Addons <a href=\"https://www.tensorflow.org/addons/api_docs/python/tfa/optimizers/AdamW\">here</a>. (I'm not sure TFA is installed on Kaggle yet though, pip installing it on top of the Kaggle container used to be problematic)</p>\n\n<p>The source code of an LR schedule with warmup can be found in the <a href=\"https://www.kaggle.com/mgornergoogle/five-flowers-with-keras-and-xception-on-tpu\">TPU getting started notebook</a>.</p>",
      "votes": null,
      "replies": [
        {
          "id": 817815,
          "author_name": "rftexas",
          "author_url": "",
          "post_date": "04/23/2020 12:46:54",
          "content": "<p>Thanks a lot !</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 817884,
      "author_name": "seesee",
      "author_url": "",
      "post_date": "04/23/2020 13:53:38",
      "content": "<p>The <a href=\"https://github.com/tensorflow/models/blob/master/official/nlp/optimization.py\">official optimizer</a> works with TF2.1. You can copy the file and use the <code>create_optimizer</code> method. <code>num_warmup_steps</code> is usually <code>int(0.1 * num_train_steps)</code>. It also implements many details like gradient clipping and regularization.</p>",
      "votes": null,
      "replies": [
        {
          "id": 818322,
          "author_name": "bamps53",
          "author_url": "",
          "post_date": "04/23/2020 19:41:23",
          "content": "<p><a href=\"/seesee\">@seesee</a> When I update to TF2.2.0, something went wrong with AdamWeightDecay. Have you experienced same thing?(The reason why asked this is you wrote it works with TF2.1)</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 818356,
          "author_name": "seesee",
          "author_url": "",
          "post_date": "04/23/2020 20:01:13",
          "content": "<p>Yes, I recall that the solver that I posted didn't work with <code>tensorflow:2.2.0rc3</code>. I didn't investigate the error further and decided to stick to 2.1 until the full 2.2 release.</p>\n\n<p>Edit: The fix is <a href=\"https://github.com/tensorflow/addons/pull/1566\">here</a>.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 820558,
          "author_name": "bamps53",
          "author_url": "",
          "post_date": "04/25/2020 14:33:20",
          "content": "<p><a href=\"/seesee\">@seesee</a> Thanks, yeah, that's better choice. I'll wait too:)</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 833262,
          "author_name": "rftexas",
          "author_url": "",
          "post_date": "05/04/2020 17:37:27",
          "content": "<p>Thanks for your answer See--!</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 833263,
      "author_name": "rftexas",
      "author_url": "",
      "post_date": "05/04/2020 17:38:44",
      "content": "<p>For your information, I solved my issue by using AdamW from Tensorflow_addons library combined with the create_optimizer method mentioned by <a href=\"/seesee\">@seesee</a> </p>",
      "votes": null,
      "replies": [
        {
          "id": 834746,
          "author_name": "mgorner",
          "author_url": "",
          "post_date": "05/05/2020 18:41:24",
          "content": "<p>Can you post your code ? I though Tensorflow addons could not be installed in a Kaggle Notebook ?</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "816940": "Hello everyone,\n\nI have used almost exclusively Keras to build my models for this competition. I'd like to use the schedulers and optimizers given by the transformers library. Right now, I can't find a way to do it. \n\nAre AdamW or get_linear_schedule_with_warmup only available with PyTorch? \n\nAny help is appreciated ;)\n\nMany thanks in advance",
    "817173": "Not a direct answer but maybe this can help:\n\nAdamW is available in Tensorflow Addons [here](https://www.tensorflow.org/addons/api_docs/python/tfa/optimizers/AdamW). (I'm not sure TFA is installed on Kaggle yet though, pip installing it on top of the Kaggle container used to be problematic)\n\nThe source code of an LR schedule with warmup can be found in the [TPU getting started notebook](https://www.kaggle.com/mgornergoogle/five-flowers-with-keras-and-xception-on-tpu).",
    "817815": "Thanks a lot !",
    "817884": "The [official optimizer](https://github.com/tensorflow/models/blob/master/official/nlp/optimization.py) works with TF2.1. You can copy the file and use the `create_optimizer` method. `num_warmup_steps` is usually `int(0.1 * num_train_steps)`. It also implements many details like gradient clipping and regularization.",
    "818322": "seesee When I update to TF2.2.0, something went wrong with AdamWeightDecay. Have you experienced same thing?(The reason why asked this is you wrote it works with TF2.1)",
    "818356": "Yes, I recall that the solver that I posted didn't work with `tensorflow:2.2.0rc3`. I didn't investigate the error further and decided to stick to 2.1 until the full 2.2 release.\n\nEdit: The fix is [here](https://github.com/tensorflow/addons/pull/1566).",
    "820558": "seesee Thanks, yeah, that's better choice. I'll wait too:)",
    "833262": "Thanks for your answer See--!",
    "833263": "For your information, I solved my issue by using AdamW from Tensorflow_addons library combined with the create_optimizer method mentioned by @seesee",
    "834746": "Can you post your code ? I though Tensorflow addons could not be installed in a Kaggle Notebook ?"
  },
  "source": "meta"
}