{
  "id": 149132,
  "title": "Saw tooth pattern of train loss, using custom train loop",
  "url": "/competitions/flower-classification-with-tpus/discussion/149132",
  "author_name": "",
  "post_date": "2020-05-07T01:04:44.479535Z",
  "votes": null,
  "comment_count": 1,
  "views": 0,
  "content": "<p>Hi all,\nI have this notebook: <a href=\"https://www.kaggle.com/zungmann/densenet-with-100-flowers-on-tpu\">https://www.kaggle.com/zungmann/densenet-with-100-flowers-on-tpu</a>, forked and updated from various sources\nhaving issue with the saw tooth like pattern of train loss when using custom training loop. It is happening every 5 epochs, the loss and training time looks doubled. I don't have this pattern when using model.fit.\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1881959%2F3081c18c3bcc150bc8d27afebddabafb%2Ftrain_loss.png?generation=1588911896030073&amp;alt=media\" alt=\"\"></p>\n\n<p>Appreciated any help on this.</p>\n\n<p>Thanks</p>",
  "messages": [
    {
      "id": "836415",
      "postDate": "05/07/2020 01:04:44",
      "content": "<p>Hi all,\nI have this notebook: <a href=\"https://www.kaggle.com/zungmann/densenet-with-100-flowers-on-tpu\">https://www.kaggle.com/zungmann/densenet-with-100-flowers-on-tpu</a>, forked and updated from various sources\nhaving issue with the saw tooth like pattern of train loss when using custom training loop. It is happening every 5 epochs, the loss and training time looks doubled. I don't have this pattern when using model.fit.\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1881959%2F3081c18c3bcc150bc8d27afebddabafb%2Ftrain_loss.png?generation=1588911896030073&amp;alt=media\" alt=\"\"></p>\n\n<p>Appreciated any help on this.</p>\n\n<p>Thanks</p>",
      "rawMarkdown": "Hi all,\nI have this notebook: https://www.kaggle.com/zungmann/densenet-with-100-flowers-on-tpu, forked and updated from various sources\nhaving issue with the saw tooth like pattern of train loss when using custom training loop. It is happening every 5 epochs, the loss and training time looks doubled. I don't have this pattern when using model.fit.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1881959%2F3081c18c3bcc150bc8d27afebddabafb%2Ftrain_loss.png?generation=1588911896030073&amp;alt=media)\n\nAppreciated any help on this.\n\nThanks",
      "votes": null
    },
    {
      "id": "838877",
      "postDate": "05/08/2020 21:25:37",
      "content": "<p>the custom training loop code in tf 2.1 seems to be unstable. Maybe this can be the reason. try to install tf 2.2 and recheck pls.</p>",
      "rawMarkdown": "the custom training loop code in tf 2.1 seems to be unstable. Maybe this can be the reason. try to install tf 2.2 and recheck pls.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 838877,
      "author_name": "romanweilguny",
      "author_url": "",
      "post_date": "05/08/2020 21:25:37",
      "content": "<p>the custom training loop code in tf 2.1 seems to be unstable. Maybe this can be the reason. try to install tf 2.2 and recheck pls.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "836415": "Hi all,\nI have this notebook: https://www.kaggle.com/zungmann/densenet-with-100-flowers-on-tpu, forked and updated from various sources\nhaving issue with the saw tooth like pattern of train loss when using custom training loop. It is happening every 5 epochs, the loss and training time looks doubled. I don't have this pattern when using model.fit.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1881959%2F3081c18c3bcc150bc8d27afebddabafb%2Ftrain_loss.png?generation=1588911896030073&amp;alt=media)\n\nAppreciated any help on this.\n\nThanks",
    "838877": "the custom training loop code in tf 2.1 seems to be unstable. Maybe this can be the reason. try to install tf 2.2 and recheck pls."
  },
  "source": "meta"
}