I'm using an efficient net b3 backbone with an Image size of 256x256 it achieves accuracy of somewhere around 0.85
\nbut using image resolution of 512x512 accuracy dips significantly to 0.65
\n(PS. due to the large image size, I need to reduce the batch size while training model with 512x512 size)
\nI was assuming that accuracy should increase or must be somewhere around 256x256
\nAny particular reason for the above failure
\nAlso, it would be nice if you can mention your CV or LB score
I'm using an efficient net b3 backbone with an Image size of 256x256 it achieves accuracy of somewhere around 0.85
\nbut using image resolution of 512x512 accuracy dips significantly to 0.65
\n(PS. due to the large image size, I need to reduce the batch size while training model with 512x512 size)
\nI was assuming that accuracy should increase or must be somewhere around 256x256
\nAny particular reason for the above failure
\nAlso, it would be nice if you can mention your CV or LB score
Hello!
\nI'm sure there must be a mistake in your code, as numerous users here reported that larger image size, ceteris paribus, leads to higher accuracy. To make your experiments with image size clear, make sure that you seed every operation including randomness in your code and manually initialize the model's head.
\nOne of the reasons for this accuracy drop is the learning rate which, generally speaking, should not be kept the same if you change the batch size: smaller batches mean more stochasticity while larger batches are more stable. A rule of thumb I've seen throughout many notebooks is setting the max value of the learning rate equal to 3e-5 - 1e-4 times the number of cores (GPU = 1 core, TPU = 8 cores) when you fine-tuning or to 0.1 * batch_size / 256 when you are training from scratch.
As for me, switching from 384x384 to 512x512 resulted in +0.03 CV & LB gain.
", "rawMarkdown": "Hello!\n\nI'm sure there must be a mistake in your code, as numerous users here reported that larger image size, ceteris paribus, leads to higher accuracy. To make your experiments with image size clear, make sure that you seed every operation including randomness in your code and manually initialize the model's head. \n\nOne of the reasons for this accuracy drop is the learning rate which, generally speaking, should not be kept the same if you change the batch size: smaller batches mean more stochasticity while larger batches are more stable. A rule of thumb I've seen throughout many notebooks is setting the max value of the learning rate equal to `3e-5 - 1e-4` times the number of cores (GPU = 1 core, TPU = 8 cores) when you fine-tuning or to `0.1 * batch_size / 256` when you are training from scratch.\n\nAs for me, switching from 384x384 to 512x512 resulted in +0.03 CV & LB gain.", "votes": null }, { "id": "1201839", "postDate": "02/15/2021 17:24:39", "content": "I noticed an improvement on B3 with a bigger image size, so I would check our code and print out the image shape during training to make sure some augments are not causing the image size to be something other than what you wanted.
", "rawMarkdown": "I noticed an improvement on B3 with a bigger image size, so I would check our code and print out the image shape during training to make sure some augments are not causing the image size to be something other than what you wanted.", "votes": null }, { "id": "1202089", "postDate": "02/15/2021 20:35:37", "content": "I am using 512 size and it worked best. 256 size is giving less accuracy. You must have made a mistake somewhere.
\nIf it is helpful you can compare it with my basic notebooks here
Most pretrained image neural networks have BatchNormalizations. So if you have a very small batch size, then when you finetune the BatchNormalizations you are using a very small sample amount to estimate mean and variance. Maybe look into gradient accumulation. Also Learning Rate must be tuned mutually with Batch size.
", "rawMarkdown": "Most pretrained image neural networks have BatchNormalizations. So if you have a very small batch size, then when you finetune the BatchNormalizations you are using a very small sample amount to estimate mean and variance. Maybe look into gradient accumulation. Also Learning Rate must be tuned mutually with Batch size.", "votes": null }, { "id": "1204308", "postDate": "02/16/2021 03:42:05", "content": "You can try different learning rate schedules. Like cosine annealing or exponential decay and see if the CV improves. Also when applying the learning rate schedules, don't use early stopping because it may stop training before the accuracy is improved.
", "rawMarkdown": "You can try different learning rate schedules. Like cosine annealing or exponential decay and see if the CV improves. Also when applying the learning rate schedules, don't use early stopping because it may stop training before the accuracy is improved.", "votes": null } ], "comments": [ { "id": 1201757, "author_name": "nickuzmenkov", "author_url": "", "post_date": "02/15/2021 16:23:37", "content": "Hello!
\nI'm sure there must be a mistake in your code, as numerous users here reported that larger image size, ceteris paribus, leads to higher accuracy. To make your experiments with image size clear, make sure that you seed every operation including randomness in your code and manually initialize the model's head.
\nOne of the reasons for this accuracy drop is the learning rate which, generally speaking, should not be kept the same if you change the batch size: smaller batches mean more stochasticity while larger batches are more stable. A rule of thumb I've seen throughout many notebooks is setting the max value of the learning rate equal to 3e-5 - 1e-4 times the number of cores (GPU = 1 core, TPU = 8 cores) when you fine-tuning or to 0.1 * batch_size / 256 when you are training from scratch.
As for me, switching from 384x384 to 512x512 resulted in +0.03 CV & LB gain.
", "votes": null, "replies": [] }, { "id": 1201839, "author_name": "trushk", "author_url": "", "post_date": "02/15/2021 17:24:39", "content": "I noticed an improvement on B3 with a bigger image size, so I would check our code and print out the image shape during training to make sure some augments are not causing the image size to be something other than what you wanted.
", "votes": null, "replies": [] }, { "id": 1202089, "author_name": "vickygoyal", "author_url": "", "post_date": "02/15/2021 20:35:37", "content": "I am using 512 size and it worked best. 256 size is giving less accuracy. You must have made a mistake somewhere.
\nIf it is helpful you can compare it with my basic notebooks here
Most pretrained image neural networks have BatchNormalizations. So if you have a very small batch size, then when you finetune the BatchNormalizations you are using a very small sample amount to estimate mean and variance. Maybe look into gradient accumulation. Also Learning Rate must be tuned mutually with Batch size.
", "votes": null, "replies": [] }, { "id": 1204308, "author_name": "sigmasix", "author_url": "", "post_date": "02/16/2021 03:42:05", "content": "You can try different learning rate schedules. Like cosine annealing or exponential decay and see if the CV improves. Also when applying the learning rate schedules, don't use early stopping because it may stop training before the accuracy is improved.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1201680": "I'm using an efficient net b3 backbone with an Image size of 256x256 it achieves accuracy of somewhere around 0.85 \n\nbut using image resolution of 512x512 accuracy dips significantly to 0.65\n\n(PS. due to the large image size, I need to reduce the batch size while training model with 512x512 size)\n\nI was assuming that accuracy should increase or must be somewhere around 256x256\n\nAny particular reason for the above failure\nAlso, it would be nice if you can mention your CV or LB score", "1201757": "Hello!\n\nI'm sure there must be a mistake in your code, as numerous users here reported that larger image size, ceteris paribus, leads to higher accuracy. To make your experiments with image size clear, make sure that you seed every operation including randomness in your code and manually initialize the model's head. \n\nOne of the reasons for this accuracy drop is the learning rate which, generally speaking, should not be kept the same if you change the batch size: smaller batches mean more stochasticity while larger batches are more stable. A rule of thumb I've seen throughout many notebooks is setting the max value of the learning rate equal to `3e-5 - 1e-4` times the number of cores (GPU = 1 core, TPU = 8 cores) when you fine-tuning or to `0.1 * batch_size / 256` when you are training from scratch.\n\nAs for me, switching from 384x384 to 512x512 resulted in +0.03 CV & LB gain.", "1201839": "I noticed an improvement on B3 with a bigger image size, so I would check our code and print out the image shape during training to make sure some augments are not causing the image size to be something other than what you wanted.", "1202089": "I am using 512 size and it worked best. 256 size is giving less accuracy. You must have made a mistake somewhere. \nIf it is helpful you can compare it with my basic notebooks [here](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/216461)", "1204181": "Most pretrained image neural networks have BatchNormalizations. So if you have a very small batch size, then when you finetune the BatchNormalizations you are using a very small sample amount to estimate mean and variance. Maybe look into gradient accumulation. Also Learning Rate must be tuned mutually with Batch size.", "1204308": "You can try different learning rate schedules. Like cosine annealing or exponential decay and see if the CV improves. Also when applying the learning rate schedules, don't use early stopping because it may stop training before the accuracy is improved." }, "source": "meta" }