It's always convenient to re-use other people's code and saves you a lot of time, but sometimes there are mistakes.
\nFor instance, in this great kernel, normalization is made weirdly (see function mono_to_color).
The Resnest was pretrained on ImageNet, therefore the good practice is to scale spectrograms in [0, 1] and then normalize them using ImageNet stats (such as made in the official repo).
\n(edit : you can also normalize with the spectrogram stats, the idea is to have 0 mean~0 and 1 std)
\nThis snippet should do the job :
\nMEAN = np.array([0.485, 0.456, 0.406])\nSTD = np.array([0.229, 0.224, 0.225])\n\ndef mono_to_color(X, eps=1e-6):\n X = np.stack([X, X, X], axis=-1)\n\n # Normalize to [0, 255]\n _min, _max = X.min(), X.max()\n\n if (_max - _min) > eps:\n V = np.clip(X, _min, _max)\n V = 255 * (V - _min) / (_max - _min)\n V = V.astype(np.uint8)\n else:\n # Just zero\n V = np.zeros_like(X, dtype=np.uint8)\n\n return V\n\ndef normalize(image, mean, std):\n image = (image / 255.0).astype(np.float32)\n image = (image - mean) / std\n return np.moveaxis(image, 2, 0)\n\nimage = mono_to_color(melspec)\nimage = normalize(image, mean=MEAN, std=STD)\nIt's always convenient to re-use other people's code and saves you a lot of time, but sometimes there are mistakes.
\nFor instance, in this great kernel, normalization is made weirdly (see function mono_to_color).
The Resnest was pretrained on ImageNet, therefore the good practice is to scale spectrograms in [0, 1] and then normalize them using ImageNet stats (such as made in the official repo).
\n(edit : you can also normalize with the spectrogram stats, the idea is to have 0 mean~0 and 1 std)
\nThis snippet should do the job :
\nMEAN = np.array([0.485, 0.456, 0.406])\nSTD = np.array([0.229, 0.224, 0.225])\n\ndef mono_to_color(X, eps=1e-6):\n X = np.stack([X, X, X], axis=-1)\n\n # Normalize to [0, 255]\n _min, _max = X.min(), X.max()\n\n if (_max - _min) > eps:\n V = np.clip(X, _min, _max)\n V = 255 * (V - _min) / (_max - _min)\n V = V.astype(np.uint8)\n else:\n # Just zero\n V = np.zeros_like(X, dtype=np.uint8)\n\n return V\n\ndef normalize(image, mean, std):\n image = (image / 255.0).astype(np.float32)\n image = (image - mean) / std\n return np.moveaxis(image, 2, 0)\n\nimage = mono_to_color(melspec)\nimage = normalize(image, mean=MEAN, std=STD)\nAlso when training even on ImageNet pretrained models, you can miss normalization to ImageNet stats if you continue training the backbone model
", "rawMarkdown": "Also when training even on ImageNet pretrained models, you can miss normalization to ImageNet stats if you continue training the backbone model", "votes": 1, "replies": [ { "id": 986406, "postDate": "2020-08-26T13:07:30.050Z", "content": "As @cpmpml said, models still expect data with 0 mean and std 1 so normalization usually helps a bit, at least at the beginning.
", "rawMarkdown": "As @cpmpml said, models still expect data with 0 mean and std 1 so normalization usually helps a bit, at least at the beginning.", "votes": 1 } ] }, { "id": 986271, "postDate": "2020-08-26T10:32:27.383Z", "content": "I am not so sure about the normalization part. The MEAN and STD values are the mean and std of imagenet images. Your spectrogram images most certainly have different mean and std.
", "rawMarkdown": "I am not so sure about the normalization part. The MEAN and STD values are the mean and std of imagenet images. Your spectrogram images most certainly have different mean and std.", "votes": 1, "replies": [ { "id": 986404, "postDate": "2020-08-26T13:05:54.160Z", "content": "Indeed, it can be better to normalize using the stats of our dataset.
", "rawMarkdown": "Indeed, it can be better to normalize using the stats of our dataset. ", "votes": 1 }, { "id": 986565, "postDate": "2020-08-26T15:41:11.290Z", "content": "I think it is the best practice to normalize with imagenet's stat but not a MUST.
\nIf use the magenet model and inference directly then we should normalize with the imagenet stat. However in this comp we will continue to train many epochs and as long as we apply the same normalization to the test set, it should be alright?
of coz, no harm to follow the best practice😀
", "rawMarkdown": "I think it is the best practice to normalize with imagenet's stat but not a MUST.\nIf use the magenet model and inference directly then we should normalize with the imagenet stat. However in this comp we will continue to train many epochs and as long as we apply the same normalization to the test set, it should be alright?\n\nof coz, no harm to follow the best practice😀\n\n", "votes": 1 }, { "id": 986617, "postDate": "2020-08-26T16:28:14.063Z", "content": "@fiyeroleung Performances should not be affected much indeed :)
", "rawMarkdown": "@fiyeroleung Performances should not be affected much indeed :) " }, { "id": 986660, "postDate": "2020-08-26T17:06:31.173Z", "content": "@theoviel btw i really like yr profile pic😆😆
", "rawMarkdown": "@theoviel btw i really like yr profile pic😆😆", "votes": 1 } ] }, { "id": 986286, "postDate": "2020-08-26T10:48:43.663Z", "content": "Why can't we train on files from 0 to 1 without normalization?
", "rawMarkdown": "Why can't we train on files from 0 to 1 without normalization?", "votes": 2, "replies": [ { "id": 986332, "postDate": "2020-08-26T11:43:30.287Z", "content": "The pretrained model expects 0 mean and std 1. If you depart from that then your model will have to learn how to deal with your images mean and variance. It can be detrimental…. or not. The only way to know is to try.
", "rawMarkdown": "The pretrained model expects 0 mean and std 1. If you depart from that then your model will have to learn how to deal with your images mean and variance. It can be detrimental.... or not. The only way to know is to try.", "votes": 2 } ] }, { "id": 1742122, "postDate": "2022-04-01T13:39:23.517Z", "content": "Do we need this?
\nV = np.clip(X, _min, _max)
maybe only if we pass a global min and max value to this function?
\nAlso does mono to color really help in your experience? It takes 3X more memory.
", "rawMarkdown": "Do we need this?\n` V = np.clip(X, _min, _max)`\n\nmaybe only if we pass a global min and max value to this function?\n\nAlso does mono to color really help in your experience? It takes 3X more memory. " }, { "id": 997166, "postDate": "2020-09-03T19:46:39.290Z", "content": "I tried to make this change but now I'm running into out of memory errors. Are you running this on kaggle kernels or something better?
\nmaybe I need to be careful copy/pasting from comments warning about copy/pasting
", "rawMarkdown": "I tried to make this change but now I'm running into out of memory errors. Are you running this on kaggle kernels or something better?\n\nmaybe I need to be careful copy/pasting from comments warning about copy/pasting", "replies": [ { "id": 997178, "postDate": "2020-09-03T19:54:03.903Z", "content": "\n\nmaybe I need to be careful copy/pasting from comments warning about copy/pasting
\n
True.
\nThis should not affect memory, I don't really know what's going on here
", "rawMarkdown": "> maybe I need to be careful copy/pasting from comments warning about copy/pasting\n\nTrue.\n\n\nThis should not affect memory, I don't really know what's going on here", "votes": 1 }, { "id": 997252, "postDate": "2020-09-03T21:11:40.267Z", "content": "Thanks for the quick reply. Turns out I had np.moveaxis(image, 2, 0) twice.
", "rawMarkdown": "Thanks for the quick reply. Turns out I had np.moveaxis(image, 2, 0) twice. " } ] }, { "id": 989670, "postDate": "2020-08-29T02:57:42.913Z", "content": "Suppose we normalize using the training dataset's stats, but then wouldn't the stats of the test data be different?
", "rawMarkdown": "Suppose we normalize using the training dataset's stats, but then wouldn't the stats of the *test* data be different?", "replies": [ { "id": 3148457, "postDate": "2025-03-13T06:30:08.227Z", "content": "Actually we also need to calculate test data stats for prediction, the model needs normalized input for both train and test set. and it won't affect training process without test stats
", "rawMarkdown": "Actually we also need to calculate test data stats for prediction, the model needs normalized input for both train and test set. and it won't affect training process without test stats" } ] }, { "id": 989380, "postDate": "2020-08-28T18:44:05.063Z", "content": "Well, there are actually four pieces here: mono-to-color, nan-and-infinity via clip, range-normalization, and gaussian normalization. I have no idea why would you do both gaussian and range normalization. The way its written range normalization followed by gaussian normalization produces the same result as gaussian normalization.
", "rawMarkdown": "Well, there are actually four pieces here: mono-to-color, nan-and-infinity via clip, range-normalization, and gaussian normalization. I have no idea why would you do both gaussian and range normalization. The way its written range normalization followed by gaussian normalization produces the same result as gaussian normalization. ", "replies": [ { "id": 989397, "postDate": "2020-08-28T18:58:14.713Z", "content": "Not really because I don't do gaussian normalization with the stats of the spectrogram but rather with the stats of the imagenet dataset
", "rawMarkdown": "Not really because I don't do gaussian normalization with the stats of the spectrogram but rather with the stats of the imagenet dataset" }, { "id": 989423, "postDate": "2020-08-28T19:15:11.430Z", "content": "this part is range-normalization:
\n V = 255 * (V - _min) / (_max - _min)
\nthis part is gaussian normalization
\n image = (image - mean) / std
\ndoing both is same as doing latter.
It is not exactly identical because of astype(np.uint8) as the decimal part is dropped but its nearly identical.
\nYet, it does makes sense to do this if you saved image as cache pngs but then don't do mono-to-color first.
", "rawMarkdown": "this part is range-normalization:\n V = 255 * (V - _min) / (_max - _min)\nthis part is gaussian normalization\n image = (image - mean) / std\ndoing both is same as doing latter.\n\nIt is not exactly identical because of astype(np.uint8) as the decimal part is dropped but its nearly identical.\n \nYet, it does makes sense to do this if you saved image as cache pngs but then don't do mono-to-color first." }, { "id": 989450, "postDate": "2020-08-28T19:43:38.063Z", "content": "mean != img.mean() ^^
", "rawMarkdown": "mean != img.mean() ^^ " }, { "id": 989763, "postDate": "2020-08-29T05:21:11.113Z", "content": "I … am the original author of mono to color function, feeling sad to see roughly roughly copy-pasted here and there (crying).
\nI think I need to name better to stop this kind of problem, … in the future. :P
Also when training even on ImageNet pretrained models, you can miss normalization to ImageNet stats if you continue training the backbone model
", "votes": 1, "replies": [ { "id": 986406, "author_name": "Theo Viel", "author_url": "", "post_date": "2020-08-26T13:07:30.050000", "content": "As @cpmpml said, models still expect data with 0 mean and std 1 so normalization usually helps a bit, at least at the beginning.
", "votes": 1, "replies": [] } ] }, { "id": 986271, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-26T10:32:27.383000", "content": "I am not so sure about the normalization part. The MEAN and STD values are the mean and std of imagenet images. Your spectrogram images most certainly have different mean and std.
", "votes": 1, "replies": [ { "id": 986404, "author_name": "Theo Viel", "author_url": "", "post_date": "2020-08-26T13:05:54.160000", "content": "Indeed, it can be better to normalize using the stats of our dataset.
", "votes": 1, "replies": [] }, { "id": 986565, "author_name": "Salaryman", "author_url": "", "post_date": "2020-08-26T15:41:11.290000", "content": "I think it is the best practice to normalize with imagenet's stat but not a MUST.
\nIf use the magenet model and inference directly then we should normalize with the imagenet stat. However in this comp we will continue to train many epochs and as long as we apply the same normalization to the test set, it should be alright?
of coz, no harm to follow the best practice😀
", "votes": 1, "replies": [] }, { "id": 986617, "author_name": "Theo Viel", "author_url": "", "post_date": "2020-08-26T16:28:14.063000", "content": "@fiyeroleung Performances should not be affected much indeed :)
", "votes": 0, "replies": [] }, { "id": 986660, "author_name": "Salaryman", "author_url": "", "post_date": "2020-08-26T17:06:31.173000", "content": "@theoviel btw i really like yr profile pic😆😆
", "votes": 1, "replies": [] } ] }, { "id": 986286, "author_name": "Kramarenko Vladislav", "author_url": "", "post_date": "2020-08-26T10:48:43.663000", "content": "Why can't we train on files from 0 to 1 without normalization?
", "votes": 2, "replies": [ { "id": 986332, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-26T11:43:30.287000", "content": "The pretrained model expects 0 mean and std 1. If you depart from that then your model will have to learn how to deal with your images mean and variance. It can be detrimental…. or not. The only way to know is to try.
", "votes": 2, "replies": [] } ] }, { "id": 1742122, "author_name": "Allohvk", "author_url": "", "post_date": "2022-04-01T13:39:23.517000", "content": "Do we need this?
\nV = np.clip(X, _min, _max)
maybe only if we pass a global min and max value to this function?
\nAlso does mono to color really help in your experience? It takes 3X more memory.
", "votes": 0, "replies": [] }, { "id": 997166, "author_name": "Eric Freeman", "author_url": "", "post_date": "2020-09-03T19:46:39.290000", "content": "I tried to make this change but now I'm running into out of memory errors. Are you running this on kaggle kernels or something better?
\nmaybe I need to be careful copy/pasting from comments warning about copy/pasting
", "votes": 0, "replies": [ { "id": 997178, "author_name": "Theo Viel", "author_url": "", "post_date": "2020-09-03T19:54:03.903000", "content": "\n\nmaybe I need to be careful copy/pasting from comments warning about copy/pasting
\n
True.
\nThis should not affect memory, I don't really know what's going on here
", "votes": 1, "replies": [] }, { "id": 997252, "author_name": "Eric Freeman", "author_url": "", "post_date": "2020-09-03T21:11:40.267000", "content": "Thanks for the quick reply. Turns out I had np.moveaxis(image, 2, 0) twice.
", "votes": 0, "replies": [] } ] }, { "id": 989670, "author_name": "Mighty Rains", "author_url": "", "post_date": "2020-08-29T02:57:42.913000", "content": "Suppose we normalize using the training dataset's stats, but then wouldn't the stats of the test data be different?
", "votes": 0, "replies": [ { "id": 3148457, "author_name": "HangVapour", "author_url": "", "post_date": "2025-03-13T06:30:08.227000", "content": "Actually we also need to calculate test data stats for prediction, the model needs normalized input for both train and test set. and it won't affect training process without test stats
", "votes": 0, "replies": [] } ] }, { "id": 989380, "author_name": "leodav", "author_url": "", "post_date": "2020-08-28T18:44:05.063000", "content": "Well, there are actually four pieces here: mono-to-color, nan-and-infinity via clip, range-normalization, and gaussian normalization. I have no idea why would you do both gaussian and range normalization. The way its written range normalization followed by gaussian normalization produces the same result as gaussian normalization.
", "votes": 0, "replies": [ { "id": 989397, "author_name": "Theo Viel", "author_url": "", "post_date": "2020-08-28T18:58:14.713000", "content": "Not really because I don't do gaussian normalization with the stats of the spectrogram but rather with the stats of the imagenet dataset
", "votes": 0, "replies": [] }, { "id": 989423, "author_name": "leodav", "author_url": "", "post_date": "2020-08-28T19:15:11.430000", "content": "this part is range-normalization:
\n V = 255 * (V - _min) / (_max - _min)
\nthis part is gaussian normalization
\n image = (image - mean) / std
\ndoing both is same as doing latter.
It is not exactly identical because of astype(np.uint8) as the decimal part is dropped but its nearly identical.
\nYet, it does makes sense to do this if you saved image as cache pngs but then don't do mono-to-color first.
", "votes": 0, "replies": [] }, { "id": 989450, "author_name": "Theo Viel", "author_url": "", "post_date": "2020-08-28T19:43:38.063000", "content": "mean != img.mean() ^^
", "votes": 0, "replies": [] }, { "id": 989763, "author_name": "daisukelab", "author_url": "", "post_date": "2020-08-29T05:21:11.113000", "content": "I … am the original author of mono to color function, feeling sad to see roughly roughly copy-pasted here and there (crying).
\nI think I need to name better to stop this kind of problem, … in the future. :P