{
  "id": 38910,
  "title": "Why it's impossible to use batch=1 with BatchNorm?",
  "url": "/competitions/carvana-image-masking-challenge/discussion/38910",
  "author_name": "",
  "post_date": "2017-09-02T16:20:03.393962200Z",
  "votes": null,
  "comment_count": 3,
  "views": 0,
  "content": "<p>As I understood from paper BatchNorm uses running average for bias and deviation, so it could be calculated even when feeding samples one by one.\nWhat's wrong?</p>",
  "messages": [
    {
      "id": "218156",
      "postDate": "09/02/2017 16:20:03",
      "content": "<p>As I understood from paper BatchNorm uses running average for bias and deviation, so it could be calculated even when feeding samples one by one.\nWhat's wrong?</p>",
      "rawMarkdown": "As I understood from paper BatchNorm uses running average for bias and deviation, so it could be calculated even when feeding samples one by one.\nWhat's wrong?",
      "votes": null
    },
    {
      "id": "218226",
      "postDate": "09/03/2017 02:09:27",
      "content": "<p>It is not impossible. You can do it and it will work (I did it with BatchNorm and Batch Renorm)</p>\n\n<p>But results are lower than with higher batches.</p>\n\n<p>As you said it runs average. So average of 1 sample is... the same thing, it is not very helping. </p>\n\n<p>At least that how I understand it.</p>",
      "rawMarkdown": "It is not impossible. You can do it and it will work (I did it with BatchNorm and Batch Renorm)\n\nBut results are lower than with higher batches.\n\nAs you said it runs average. So average of 1 sample is... the same thing, it is not very helping. \n\nAt least that how I understand it.",
      "votes": null
    },
    {
      "id": "218533",
      "postDate": "09/04/2017 16:00:54",
      "content": "<p>BatchNorm layer with a batch size of 1 is a no-op.</p>",
      "rawMarkdown": "BatchNorm layer with a batch size of 1 is a no-op.",
      "votes": null
    },
    {
      "id": "221117",
      "postDate": "09/14/2017 06:45:53",
      "content": "<p>And it seems that Batch Renorm performs better if you have small batch size , like batch size=2. Right?</p>",
      "rawMarkdown": "And it seems that Batch Renorm performs better if you have small batch size , like batch size=2. Right?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 218226,
      "author_name": "cjansen",
      "author_url": "",
      "post_date": "09/03/2017 02:09:27",
      "content": "<p>It is not impossible. You can do it and it will work (I did it with BatchNorm and Batch Renorm)</p>\n\n<p>But results are lower than with higher batches.</p>\n\n<p>As you said it runs average. So average of 1 sample is... the same thing, it is not very helping. </p>\n\n<p>At least that how I understand it.</p>",
      "votes": null,
      "replies": [
        {
          "id": 221117,
          "author_name": "zhangsongwei",
          "author_url": "",
          "post_date": "09/14/2017 06:45:53",
          "content": "<p>And it seems that Batch Renorm performs better if you have small batch size , like batch size=2. Right?</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 218533,
      "author_name": "jackkwok",
      "author_url": "",
      "post_date": "09/04/2017 16:00:54",
      "content": "<p>BatchNorm layer with a batch size of 1 is a no-op.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "218156": "As I understood from paper BatchNorm uses running average for bias and deviation, so it could be calculated even when feeding samples one by one.\nWhat's wrong?",
    "218226": "It is not impossible. You can do it and it will work (I did it with BatchNorm and Batch Renorm)\n\nBut results are lower than with higher batches.\n\nAs you said it runs average. So average of 1 sample is... the same thing, it is not very helping. \n\nAt least that how I understand it.",
    "218533": "BatchNorm layer with a batch size of 1 is a no-op.",
    "221117": "And it seems that Batch Renorm performs better if you have small batch size , like batch size=2. Right?"
  },
  "source": "meta"
}