A common trick to improve performance of a CNN is to apply channel-wise (i.e. for the red, green, and blue channel) centering and uniform scaling to the training images (similar to what batch-norm does for every batch).
\nThe formula is simply: img = (img - mean) / std
I have computed the channel-wise mean and std of the training set in this kernel. These values were computed on the full, non-resized training set. The images were transformed to a [0, 1] channel range by division (img / 255). If you resize your images or remove duplicates these values will slightly change but that should be insignificant.
Here is an example for a PyTorch transform block:
\ndata_transform = transforms.Compose([\n ..., # your transformations go here\n transforms.ToTensor(),\n transforms.Normalize(mean=[0.48718716, 0.62651698, 0.40807052], \n std=[0.17111757, 0.14961184, 0.17837469])\n])\nA common trick to improve performance of a CNN is to apply channel-wise (i.e. for the red, green, and blue channel) centering and uniform scaling to the training images (similar to what batch-norm does for every batch).
\nThe formula is simply: img = (img - mean) / std
I have computed the channel-wise mean and std of the training set in this kernel. These values were computed on the full, non-resized training set. The images were transformed to a [0, 1] channel range by division (img / 255). If you resize your images or remove duplicates these values will slightly change but that should be insignificant.
Here is an example for a PyTorch transform block:
\ndata_transform = transforms.Compose([\n ..., # your transformations go here\n transforms.ToTensor(),\n transforms.Normalize(mean=[0.48718716, 0.62651698, 0.40807052], \n std=[0.17111757, 0.14961184, 0.17837469])\n])\nIf you are going to train model from scratch, i.e., with randomly initialized weights as supposed to per-trained model on some other data, then you do not need to standardize data. This will happen anyways when you train the model since it will apply batch normalization. All what needs to be done is scaling pixel values to 0-1 range by dividing all image tensors by 255.
", "rawMarkdown": "If you are going to train model from scratch, i.e., with randomly initialized weights as supposed to per-trained model on some other data, then you do not need to standardize data. This will happen anyways when you train the model since it will apply batch normalization. All what needs to be done is scaling pixel values to 0-1 range by dividing all image tensors by 255.", "votes": null }, { "id": "1267445", "postDate": "04/08/2021 14:17:41", "content": "I’ve seen a number of implementations that try to squeeze out as much accuracy as possible when training from scratch (eg resnet18 on cifar). And they often of mot always use normalization. I understand your reasoning but perhaps there’s another effect.
", "rawMarkdown": "I’ve seen a number of implementations that try to squeeze out as much accuracy as possible when training from scratch (eg resnet18 on cifar). And they often of mot always use normalization. I understand your reasoning but perhaps there’s another effect.", "votes": null } ], "comments": [ { "id": 1266273, "author_name": "datasciencegeek", "author_url": "", "post_date": "04/07/2021 16:01:37", "content": "If you are going to train model from scratch, i.e., with randomly initialized weights as supposed to per-trained model on some other data, then you do not need to standardize data. This will happen anyways when you train the model since it will apply batch normalization. All what needs to be done is scaling pixel values to 0-1 range by dividing all image tensors by 255.
", "votes": null, "replies": [ { "id": 1267445, "author_name": "paulgavrikov", "author_url": "", "post_date": "04/08/2021 14:17:41", "content": "I’ve seen a number of implementations that try to squeeze out as much accuracy as possible when training from scratch (eg resnet18 on cifar). And they often of mot always use normalization. I understand your reasoning but perhaps there’s another effect.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1265831": "A common trick to improve performance of a CNN is to apply channel-wise (i.e. for the red, green, and blue channel) centering and uniform scaling to the training images (similar to what batch-norm does for every batch).\n\nThe formula is simply: `img = (img - mean) / std`\n\nI have computed the channel-wise mean and std of the training set [in this kernel.](https://www.kaggle.com/paulgavrikov/normalization-values-for-pp2021) These values were computed on the full, non-resized training set. The images were transformed to a [0, 1] channel range by division (`img / 255`). If you resize your images or remove duplicates these values will slightly change but that should be insignificant.\n\nHere is an example for a PyTorch transform block:\n\n```\ndata_transform = transforms.Compose([\n ..., # your transformations go here\n transforms.ToTensor(),\n transforms.Normalize(mean=[0.48718716, 0.62651698, 0.40807052], \n std=[0.17111757, 0.14961184, 0.17837469])\n])\n```", "1266273": "If you are going to train model from scratch, i.e., with randomly initialized weights as supposed to per-trained model on some other data, then you do not need to standardize data. This will happen anyways when you train the model since it will apply batch normalization. All what needs to be done is scaling pixel values to 0-1 range by dividing all image tensors by 255.", "1267445": "I’ve seen a number of implementations that try to squeeze out as much accuracy as possible when training from scratch (eg resnet18 on cifar). And they often of mot always use normalization. I understand your reasoning but perhaps there’s another effect." }, "source": "meta" }