Greetings to everyone! While studying the forum, I've came across a few times with advice to recalculate mean and std for normalizing procedure, rather than using \"default\" ImageNet ones. I tried to do that, but my numbers differ from ImageNet one significantly and lead to bad score. I am sharing the code that I used to do that. Could anyone share the methods or code that was used to calculate these numbers? If you find any mistakes in my calculations, I would be grateful, if you write me about them:
\ndef get_mean_std_opencv(train_df):\n sum_mean = 0.0\n sum_std = 0.0\n file_name_list = train_df[\"image_id\"].values\n for file_name in file_name_list:\n # print(file_name)\n file_path = f\"{CFG.TRAIN_PATH}/{file_name}\"\n image = cv2.imread(file_path)\n image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n resized_image = cv2.resize(image, (CFG.size, CFG.size))\n mean, std = cv2.meanStdDev(resized_image)\n sum_mean += mean\n sum_std += std\n avg_mean = sum_mean / len(file_name_list)\n avg_std = sum_std / len(file_name_list)\n return avg_mean, avg_std\n\n\ndef get_mean_std(loader):\n # var[X] = E[X**2] - E[X]**2\n mean = 0.0\n std = 0.0\n nb_samples = 0.0\n for data, _ in tqdm(loader):\n data = torch.Tensor.float(data)\n batch_samples = data.size(0)\n data = data.view(batch_samples, data.size(1), -1)\n mean += data.mean(2).sum(0)\n std += data.std(2).sum(0)\n nb_samples += batch_samples\n\n mean /= nb_samples\n std /= nb_samples\n return mean, std\nGreetings to everyone! While studying the forum, I've came across a few times with advice to recalculate mean and std for normalizing procedure, rather than using \"default\" ImageNet ones. I tried to do that, but my numbers differ from ImageNet one significantly and lead to bad score. I am sharing the code that I used to do that. Could anyone share the methods or code that was used to calculate these numbers? If you find any mistakes in my calculations, I would be grateful, if you write me about them:
\ndef get_mean_std_opencv(train_df):\n sum_mean = 0.0\n sum_std = 0.0\n file_name_list = train_df[\"image_id\"].values\n for file_name in file_name_list:\n # print(file_name)\n file_path = f\"{CFG.TRAIN_PATH}/{file_name}\"\n image = cv2.imread(file_path)\n image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n resized_image = cv2.resize(image, (CFG.size, CFG.size))\n mean, std = cv2.meanStdDev(resized_image)\n sum_mean += mean\n sum_std += std\n avg_mean = sum_mean / len(file_name_list)\n avg_std = sum_std / len(file_name_list)\n return avg_mean, avg_std\n\n\ndef get_mean_std(loader):\n # var[X] = E[X**2] - E[X]**2\n mean = 0.0\n std = 0.0\n nb_samples = 0.0\n for data, _ in tqdm(loader):\n data = torch.Tensor.float(data)\n batch_samples = data.size(0)\n data = data.view(batch_samples, data.size(1), -1)\n mean += data.mean(2).sum(0)\n std += data.std(2).sum(0)\n nb_samples += batch_samples\n\n mean /= nb_samples\n std /= nb_samples\n return mean, std\nYou can have a look at the code in this video here
", "rawMarkdown": "You can have a look at the code in this video [here](https://www.youtube.com/watch?v=y6IEcEBRZks)", "votes": null }, { "id": "1153176", "postDate": "01/14/2021 16:52:27", "content": "Did you divide values by 255?
", "rawMarkdown": "Did you divide values by 255?", "votes": null }, { "id": "1154766", "postDate": "01/15/2021 21:54:15", "content": "I published a notebook on this topic:
\nhttps://www.kaggle.com/aliabdin1/calculate-mean-std-of-images
", "rawMarkdown": "I published a notebook on this topic:\n\nhttps://www.kaggle.com/aliabdin1/calculate-mean-std-of-images", "votes": null }, { "id": "1155779", "postDate": "01/16/2021 17:07:20", "content": "Ok, guys, I think I get it now, I just forgot to delete by 255, as @woshifym mentioned, and @aliabdin1 showed in his notebook. Thanks a lot!
", "rawMarkdown": "Ok, guys, I think I get it now, I just forgot to delete by 255, as @woshifym mentioned, and @aliabdin1 showed in his notebook. Thanks a lot!", "votes": null } ], "comments": [ { "id": 1153115, "author_name": "debarshichanda", "author_url": "", "post_date": "01/14/2021 16:13:05", "content": "You can have a look at the code in this video here
", "votes": null, "replies": [] }, { "id": 1153176, "author_name": "woshifym", "author_url": "", "post_date": "01/14/2021 16:52:27", "content": "Did you divide values by 255?
", "votes": null, "replies": [] }, { "id": 1154766, "author_name": "aliabdin1", "author_url": "", "post_date": "01/15/2021 21:54:15", "content": "I published a notebook on this topic:
\nhttps://www.kaggle.com/aliabdin1/calculate-mean-std-of-images
", "votes": null, "replies": [] }, { "id": 1155779, "author_name": "etagiev", "author_url": "", "post_date": "01/16/2021 17:07:20", "content": "Ok, guys, I think I get it now, I just forgot to delete by 255, as @woshifym mentioned, and @aliabdin1 showed in his notebook. Thanks a lot!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1152790": "Greetings to everyone! While studying the forum, I've came across a few times with advice to recalculate mean and std for normalizing procedure, rather than using \"default\" ImageNet ones. I tried to do that, but my numbers differ from ImageNet one significantly and lead to bad score. I am sharing the code that I used to do that. Could anyone share the methods or code that was used to calculate these numbers? If you find any mistakes in my calculations, I would be grateful, if you write me about them:\n\n```\ndef get_mean_std_opencv(train_df):\n sum_mean = 0.0\n sum_std = 0.0\n file_name_list = train_df[\"image_id\"].values\n for file_name in file_name_list:\n # print(file_name)\n file_path = f\"{CFG.TRAIN_PATH}/{file_name}\"\n image = cv2.imread(file_path)\n image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n resized_image = cv2.resize(image, (CFG.size, CFG.size))\n mean, std = cv2.meanStdDev(resized_image)\n sum_mean += mean\n sum_std += std\n avg_mean = sum_mean / len(file_name_list)\n avg_std = sum_std / len(file_name_list)\n return avg_mean, avg_std\n\n\ndef get_mean_std(loader):\n # var[X] = E[X**2] - E[X]**2\n mean = 0.0\n std = 0.0\n nb_samples = 0.0\n for data, _ in tqdm(loader):\n data = torch.Tensor.float(data)\n batch_samples = data.size(0)\n data = data.view(batch_samples, data.size(1), -1)\n mean += data.mean(2).sum(0)\n std += data.std(2).sum(0)\n nb_samples += batch_samples\n\n mean /= nb_samples\n std /= nb_samples\n return mean, std\n```", "1153115": "You can have a look at the code in this video [here](https://www.youtube.com/watch?v=y6IEcEBRZks)", "1153176": "Did you divide values by 255?", "1154766": "I published a notebook on this topic:\n\nhttps://www.kaggle.com/aliabdin1/calculate-mean-std-of-images", "1155779": "Ok, guys, I think I get it now, I just forgot to delete by 255, as @woshifym mentioned, and @aliabdin1 showed in his notebook. Thanks a lot!" }, "source": "meta" }