{
  "id": 93714,
  "title": "Batch normalization in custom head slows down convergence",
  "url": "/competitions/imet-2019-fgvc6/discussion/93714",
  "author_name": "",
  "post_date": "2019-05-29T12:28:58.600635500Z",
  "votes": 1,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I'm using pretrained se_resnext50_32x4d with custom head. However, when I considered adding batch normalization layers into the custom head, I found that bn actually slowed down my model convergence instead of accelerating training. I'm wondering why this would happend  as I couldn't find out an explainable explanation for this. My code is shown in below figure.</p>\n\n<p><img src=\"https://i.imgur.com/HeYXDVF.jpg\" alt=\"Batch Normalization in Custom Head\"></p>",
  "messages": [
    {
      "id": "539035",
      "postDate": "05/29/2019 12:28:58",
      "content": "<p>I'm using pretrained se_resnext50_32x4d with custom head. However, when I considered adding batch normalization layers into the custom head, I found that bn actually slowed down my model convergence instead of accelerating training. I'm wondering why this would happend  as I couldn't find out an explainable explanation for this. My code is shown in below figure.</p>\n\n<p><img src=\"https://i.imgur.com/HeYXDVF.jpg\" alt=\"Batch Normalization in Custom Head\"></p>",
      "rawMarkdown": "I'm using pretrained se_resnext50_32x4d with custom head. However, when I considered adding batch normalization layers into the custom head, I found that bn actually slowed down my model convergence instead of accelerating training. I'm wondering why this would happend  as I couldn't find out an explainable explanation for this. My code is shown in below figure.\n\n![Batch Normalization in Custom Head](https://i.imgur.com/HeYXDVF.jpg)",
      "votes": null
    },
    {
      "id": "539164",
      "postDate": "05/29/2019 16:09:32",
      "content": "<p>May be your batch size is too large. BN will obviously slows down convergence if you use big batchsize .</p>",
      "rawMarkdown": "May be your batch size is too large. BN will obviously slows down convergence if you use big batchsize .",
      "votes": null
    },
    {
      "id": "539204",
      "postDate": "05/29/2019 17:22:17",
      "content": "<p>Thanks. But do you think bs 128 is large enough to lead to this problem?</p>",
      "rawMarkdown": "Thanks. But do you think bs 128 is large enough to lead to this problem?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 539164,
      "author_name": "seefun",
      "author_url": "",
      "post_date": "05/29/2019 16:09:32",
      "content": "<p>May be your batch size is too large. BN will obviously slows down convergence if you use big batchsize .</p>",
      "votes": null,
      "replies": [
        {
          "id": 539204,
          "author_name": "syoya1997",
          "author_url": "",
          "post_date": "05/29/2019 17:22:17",
          "content": "<p>Thanks. But do you think bs 128 is large enough to lead to this problem?</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "539035": "I'm using pretrained se_resnext50_32x4d with custom head. However, when I considered adding batch normalization layers into the custom head, I found that bn actually slowed down my model convergence instead of accelerating training. I'm wondering why this would happend  as I couldn't find out an explainable explanation for this. My code is shown in below figure.\n\n![Batch Normalization in Custom Head](https://i.imgur.com/HeYXDVF.jpg)",
    "539164": "May be your batch size is too large. BN will obviously slows down convergence if you use big batchsize .",
    "539204": "Thanks. But do you think bs 128 is large enough to lead to this problem?"
  },
  "source": "meta"
}