The batch normalization learns the mean and variance of data in a mini-batch size.
\n(I am not sure in PyTorch and TensorFlow)
\nIn my experiment, I think the mix-up method is unsafe to the training batch normalization . The performance increases temporarily (4-5 epoch in transfer learning) and then decreases again.
It seems the neural networks that depend on the convolution with batch normalization are reduced performance to predict the original data. (The mean of the mix-up is constant, but I am not sure if the standard deviation is also constant.)
\n(I think the ResNet was strong in the mix-up method or rotation augmentation.)
I think the batch normalization function may not be trained after getting some training.
\n(And since the needle image is a very small feature, I think that the learning rate should be started high and then sharply lowered. I think the intermediate range of learning rates limits the convergence of the model. Instead, it may be better to increase the batch size gradually.)
", "messages": [ { "id": 1347865, "postDate": "2021-06-13T14:23:26.037Z", "content": "The batch normalization learns the mean and variance of data in a mini-batch size.
\n(I am not sure in PyTorch and TensorFlow)
\nIn my experiment, I think the mix-up method is unsafe to the training batch normalization . The performance increases temporarily (4-5 epoch in transfer learning) and then decreases again.
It seems the neural networks that depend on the convolution with batch normalization are reduced performance to predict the original data. (The mean of the mix-up is constant, but I am not sure if the standard deviation is also constant.)
\n(I think the ResNet was strong in the mix-up method or rotation augmentation.)
I think the batch normalization function may not be trained after getting some training.
\n(And since the needle image is a very small feature, I think that the learning rate should be started high and then sharply lowered. I think the intermediate range of learning rates limits the convergence of the model. Instead, it may be better to increase the batch size gradually.)
", "rawMarkdown": "The batch normalization learns the mean and variance of data in a mini-batch size. \n(I am not sure in PyTorch and TensorFlow)\nIn my experiment, I think the mix-up method is unsafe to the training batch normalization . The performance increases temporarily (4-5 epoch in transfer learning) and then decreases again.\n\nIt seems the neural networks that depend on the convolution with batch normalization are reduced performance to predict the original data. (The mean of the mix-up is constant, but I am not sure if the standard deviation is also constant.)\n(I think the ResNet was strong in the mix-up method or rotation augmentation.)\n\nI think the batch normalization function may not be trained after getting some training. \n\n(And since the needle image is a very small feature, I think that the learning rate should be started high and then sharply lowered. I think the intermediate range of learning rates limits the convergence of the model. Instead, it may be better to increase the batch size gradually.)\n", "votes": 2 }, { "id": 1348849, "postDate": "2021-06-14T10:23:33Z", "content": "\n\n(The mean of the mix-up is constant, but I am not sure if the standard deviation is also constant.)
\n
As mentioned here, to regularize std for the input features you can simply square the lamba coeff (sum of normal distribution properties).
", "rawMarkdown": "> (The mean of the mix-up is constant, but I am not sure if the standard deviation is also constant.) \n\nAs mentioned [here](https://www.kaggle.com/c/seti-breakthrough-listen/discussion/245251), to regularize std for the input features you can simply square the lamba coeff (sum of normal distribution properties)." }, { "id": 1348283, "postDate": "2021-06-13T23:28:43.847Z", "content": "in my case,
\nin batch mixup (order shuffle & mixup) is better than two batch mixup (val auc & time both)
\nbut I use two batch mixup
The performance increases temporarily (4-5 epoch in transfer learning) and then decreases again.
\nI think it is related to Deep Double Descent, it might related to dataset noise
try to read paper this is one of my favorite paper
", "rawMarkdown": "in my case,\nin batch mixup (order shuffle & mixup) is better than two batch mixup (val auc & time both)\nbut I use two batch mixup\n\n`The performance increases temporarily (4-5 epoch in transfer learning) and then decreases again.`\nI think it is related to `Deep Double Descent`, it might related to dataset noise\n\ntry to read [paper](https://arxiv.org/pdf/1912.02292.pdf) this is one of my favorite paper \n " }, { "id": 1348217, "postDate": "2021-06-13T20:57:28.170Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1348849, "author_name": "Riccardo", "author_url": "", "post_date": "2021-06-14T10:23:33", "content": "\n\n(The mean of the mix-up is constant, but I am not sure if the standard deviation is also constant.)
\n
As mentioned here, to regularize std for the input features you can simply square the lamba coeff (sum of normal distribution properties).
", "votes": 0, "replies": [] }, { "id": 1348283, "author_name": "assign", "author_url": "", "post_date": "2021-06-13T23:28:43.847000", "content": "in my case,
\nin batch mixup (order shuffle & mixup) is better than two batch mixup (val auc & time both)
\nbut I use two batch mixup
The performance increases temporarily (4-5 epoch in transfer learning) and then decreases again.
\nI think it is related to Deep Double Descent, it might related to dataset noise
try to read paper this is one of my favorite paper
", "votes": 0, "replies": [] }, { "id": 1348217, "author_name": "", "author_url": "", "post_date": "2021-06-13T20:57:28.170000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1347865": "The batch normalization learns the mean and variance of data in a mini-batch size. \n(I am not sure in PyTorch and TensorFlow)\nIn my experiment, I think the mix-up method is unsafe to the training batch normalization . The performance increases temporarily (4-5 epoch in transfer learning) and then decreases again.\n\nIt seems the neural networks that depend on the convolution with batch normalization are reduced performance to predict the original data. (The mean of the mix-up is constant, but I am not sure if the standard deviation is also constant.)\n(I think the ResNet was strong in the mix-up method or rotation augmentation.)\n\nI think the batch normalization function may not be trained after getting some training. \n\n(And since the needle image is a very small feature, I think that the learning rate should be started high and then sharply lowered. I think the intermediate range of learning rates limits the convergence of the model. Instead, it may be better to increase the batch size gradually.)\n", "1348849": "> (The mean of the mix-up is constant, but I am not sure if the standard deviation is also constant.) \n\nAs mentioned [here](https://www.kaggle.com/c/seti-breakthrough-listen/discussion/245251), to regularize std for the input features you can simply square the lamba coeff (sum of normal distribution properties).", "1348283": "in my case,\nin batch mixup (order shuffle & mixup) is better than two batch mixup (val auc & time both)\nbut I use two batch mixup\n\n`The performance increases temporarily (4-5 epoch in transfer learning) and then decreases again.`\nI think it is related to `Deep Double Descent`, it might related to dataset noise\n\ntry to read [paper](https://arxiv.org/pdf/1912.02292.pdf) this is one of my favorite paper \n ", "1348217": "" } }