On unfreezing layers:
\nBathcNormalization layers need to be kept frozen. If they are also turned to trainable, the first epoch after unfreezing will significantly reduce accuracy.B7.Some other tips for utilizing EfficientNet:
\nEfficientNet do not guarantee improved performance, especially for tasks with less data or fewer classes. In such a case, the larger variant of EfficientNet chosen, the harder it is to tune hyperparameters.RMSprop setup as in the original paper for transfer learning. The momentum and learning rate are too high for transfer learning. It will easily corrupt the pretrained weight and blow up the loss. A quick check is to see if loss (as categorical cross-entropy) is getting significantly larger than log(NUM_CLASSES) after the same epoch. If so, the initial learning rate/momentum is too high.On unfreezing layers:
\nBathcNormalization layers need to be kept frozen. If they are also turned to trainable, the first epoch after unfreezing will significantly reduce accuracy.B7.Some other tips for utilizing EfficientNet:
\nEfficientNet do not guarantee improved performance, especially for tasks with less data or fewer classes. In such a case, the larger variant of EfficientNet chosen, the harder it is to tune hyperparameters.RMSprop setup as in the original paper for transfer learning. The momentum and learning rate are too high for transfer learning. It will easily corrupt the pretrained weight and blow up the loss. A quick check is to see if loss (as categorical cross-entropy) is getting significantly larger than log(NUM_CLASSES) after the same epoch. If so, the initial learning rate/momentum is too high.Why would you keep BatchNorm frozen? BatchNorm contains values that are trained to approximate the mean and standard deviation of the dataset the model was originally trained on...ImageNet, not the dataset you are trying to fine-tune on. BatchNorm will try to correct your input of melanoma data as it tries to differ from ImageNet data. I would think that if you were going to unfreeze anything it would be the BatchNorm layers.
", "rawMarkdown": "Why would you keep BatchNorm frozen? BatchNorm contains values that are trained to approximate the mean and standard deviation of the dataset the model was originally trained on...ImageNet, not the dataset you are trying to fine-tune on. BatchNorm will try to correct your input of melanoma data as it tries to differ from ImageNet data. I would think that if you were going to unfreeze anything it would be the BatchNorm layers.", "votes": null }, { "id": "961155", "postDate": "08/07/2020 00:38:22", "content": "I think it's a common practice to freeze the bn layer while fine-tuning. When we unfreeze a model that contains BatchNormalization layers in order to do fine-tuning, we should keep the BatchNormalization layers in inference mode by passing training=False when calling the base model. Otherwise, the updates applied to the non-trainable weights will suddenly destroy what the model has learned.
IMO, the bn_layer should be unfrozen if I can use a larger batch size, otherwise, it might not be stable properly.
I agree with @brianfeeny, can't understand why freezing batch norm layer would create more stability or better results.
\nNot saying that this is not true but it seems very counter-intuitive. Would you have a link to some content talking about this?
@optimo hmm, I see. I've already given the source link at the very end of this post, please follow the link.
\nHowever, IMO, it actually depends on the custom data set. If it's way more different than the original ImageNet data set and we need to fine-tune the pre-trained weights on it and if we have to set lower batch size then, in this case, I think it's a bit risky to set bn layer unfreeze. But if we can accumulate a larger batch size (Gradient Accumulation), we can then set the bn layer unfreeze. Also, with lower batch size, we can also replace BatchNormalization with Group or Instance normalization (if we want).
Also, anyone can check directly from here, written by François Chollet about fine-tuning - section:Fine-tuning, title Important notes about BatchNormalization layer.
@brianfeeny rather than downvoting, simple giving constructive feedback gives more sense, I believe. I can easily ignore that but It seems, it's happening in other places. 😅
", "rawMarkdown": "brianfeeny rather than downvoting, simple giving constructive feedback gives more sense, I believe. I can easily ignore that but It seems, it's happening [in other places](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/172892#961105). 😅", "votes": null }, { "id": "961665", "postDate": "08/07/2020 11:23:34", "content": "@ipythonx I don't downvote. If somehow something said I downvoted something then it was by accident. Edit: So I just read up on downvoting and if I had downvoted it would show in blue (to me), but your downvoted comments are in grey.
", "rawMarkdown": "ipythonx I don't downvote. If somehow something said I downvoted something then it was by accident. Edit: So I just read up on downvoting and if I had downvoted it would show in blue (to me), but your downvoted comments are in grey.", "votes": null }, { "id": "961932", "postDate": "08/07/2020 16:19:39", "content": "I found adding dense head improved the model performance. Adding three Fully Connected Layers of sizes 512, 256 and 128 helped a lot. \nRectified Adam optimizer seems to work slightly better than adam. \nCyclical Learning rates and Focal loss also work better.
", "rawMarkdown": "I found adding dense head improved the model performance. Adding three Fully Connected Layers of sizes 512, 256 and 128 helped a lot. \nRectified Adam optimizer seems to work slightly better than adam. \nCyclical Learning rates and Focal loss also work better.", "votes": null }, { "id": "962057", "postDate": "08/07/2020 19:00:00", "content": "@ajaykumar7778 So you are saying that you found doing 512, 256, 128 FC layers and then finally 1 output, worked better than just a dense head? This was for just the image data? with or without dropout and batch normalization? My own experiments showed better scores without batch norm or dropout.
", "rawMarkdown": "ajaykumar7778 So you are saying that you found doing 512, 256, 128 FC layers and then finally 1 output, worked better than just a dense head? This was for just the image data? with or without dropout and batch normalization? My own experiments showed better scores without batch norm or dropout.", "votes": null }, { "id": "962119", "postDate": "08/07/2020 20:31:47", "content": "Interesting. For me adding dense + bn + relu + dropout blocks at the head decreased performance (by a small amount) compared to just a single linear classifier. Maybe I can try 2 layer Dense + Relu/Mish instead.
", "rawMarkdown": "Interesting. For me adding dense + bn + relu + dropout blocks at the head decreased performance (by a small amount) compared to just a single linear classifier. Maybe I can try 2 layer Dense + Relu/Mish instead.", "votes": null }, { "id": "962154", "postDate": "08/07/2020 21:16:40", "content": "Interesting, I tried adding dropout layer but haven't tried dense layer yet…
", "rawMarkdown": "Interesting, I tried adding dropout layer but haven't tried dense layer yet...", "votes": null }, { "id": "962316", "postDate": "08/08/2020 03:31:40", "content": "Yes. Adding 3 FC layers instead of a single dense head worked better. This is only for image data. I did not use BatchNorm, but used a dropout of 0.3 for each of the FC layers.
", "rawMarkdown": "Yes. Adding 3 FC layers instead of a single dense head worked better. This is only for image data. I did not use BatchNorm, but used a dropout of 0.3 for each of the FC layers.", "votes": null }, { "id": "962504", "postDate": "08/08/2020 07:15:59", "content": "@ajaykumar7778 thanks for sharing. do your experiments show that BN is no needed in your FC layers
\n\n@arroqc i'm using similar head - 2-3 blocks of dense + prelu + bn + dropout. the improvement is tiny. i wonder where's the best place for BN? after dense or after activation layer?
", "rawMarkdown": "ajaykumar7778 thanks for sharing. do your experiments show that BN is no needed in your FC layers \n\n@arroqc i'm using similar head - 2-3 blocks of dense + prelu + bn + dropout. the improvement is tiny. i wonder where's the best place for BN? after dense or after activation layer?", "votes": null }, { "id": "963004", "postDate": "08/08/2020 15:35:17", "content": "It's an open question. If you look for it on the web you'll find a lot of different answer. I think it has become common practice to do Dense + BN + activation + dropout. But like many things in machine learning you can just try it :)
", "rawMarkdown": "It's an open question. If you look for it on the web you'll find a lot of different answer. I think it has become common practice to do Dense + BN + activation + dropout. But like many things in machine learning you can just try it :)", "votes": null }, { "id": "963207", "postDate": "08/08/2020 18:50:23", "content": "@arroqc indeed, there are different views from big figures in the DL field. so one more thing to try.
", "rawMarkdown": "arroqc indeed, there are different views from big figures in the DL field. so one more thing to try.", "votes": null }, { "id": "1159798", "postDate": "01/19/2021 13:27:26", "content": "Do you know a Kaggle notebook applying fine-tuning on EfficientNet using PyTorch? I would be very interested to see a working example.
", "rawMarkdown": "Do you know a Kaggle notebook applying fine-tuning on EfficientNet using PyTorch? I would be very interested to see a working example.", "votes": null } ], "comments": [ { "id": 961623, "author_name": "ipythonx", "author_url": "", "post_date": "08/07/2020 10:54:01", "content": "@optimo hmm, I see. I've already given the source link at the very end of this post, please follow the link.
\nHowever, IMO, it actually depends on the custom data set. If it's way more different than the original ImageNet data set and we need to fine-tune the pre-trained weights on it and if we have to set lower batch size then, in this case, I think it's a bit risky to set bn layer unfreeze. But if we can accumulate a larger batch size (Gradient Accumulation), we can then set the bn layer unfreeze. Also, with lower batch size, we can also replace BatchNormalization with Group or Instance normalization (if we want).
Also, anyone can check directly from here, written by François Chollet about fine-tuning - section:Fine-tuning, title Important notes about BatchNormalization layer.
Do you know a Kaggle notebook applying fine-tuning on EfficientNet using PyTorch? I would be very interested to see a working example.
", "votes": null, "replies": [] }, { "id": 961106, "author_name": "brianfeeny", "author_url": "", "post_date": "08/06/2020 23:12:39", "content": "Why would you keep BatchNorm frozen? BatchNorm contains values that are trained to approximate the mean and standard deviation of the dataset the model was originally trained on...ImageNet, not the dataset you are trying to fine-tune on. BatchNorm will try to correct your input of melanoma data as it tries to differ from ImageNet data. I would think that if you were going to unfreeze anything it would be the BatchNorm layers.
", "votes": null, "replies": [ { "id": 961155, "author_name": "ipythonx", "author_url": "", "post_date": "08/07/2020 00:38:22", "content": "I think it's a common practice to freeze the bn layer while fine-tuning. When we unfreeze a model that contains BatchNormalization layers in order to do fine-tuning, we should keep the BatchNormalization layers in inference mode by passing training=False when calling the base model. Otherwise, the updates applied to the non-trainable weights will suddenly destroy what the model has learned.
IMO, the bn_layer should be unfrozen if I can use a larger batch size, otherwise, it might not be stable properly.
I agree with @brianfeeny, can't understand why freezing batch norm layer would create more stability or better results.
\nNot saying that this is not true but it seems very counter-intuitive. Would you have a link to some content talking about this?
@brianfeeny rather than downvoting, simple giving constructive feedback gives more sense, I believe. I can easily ignore that but It seems, it's happening in other places. 😅
", "votes": null, "replies": [] }, { "id": 961665, "author_name": "brianfeeny", "author_url": "", "post_date": "08/07/2020 11:23:34", "content": "@ipythonx I don't downvote. If somehow something said I downvoted something then it was by accident. Edit: So I just read up on downvoting and if I had downvoted it would show in blue (to me), but your downvoted comments are in grey.
", "votes": null, "replies": [] } ] }, { "id": 961932, "author_name": "ajaykumar7778", "author_url": "", "post_date": "08/07/2020 16:19:39", "content": "I found adding dense head improved the model performance. Adding three Fully Connected Layers of sizes 512, 256 and 128 helped a lot. \nRectified Adam optimizer seems to work slightly better than adam. \nCyclical Learning rates and Focal loss also work better.
", "votes": null, "replies": [ { "id": 962057, "author_name": "brianfeeny", "author_url": "", "post_date": "08/07/2020 19:00:00", "content": "@ajaykumar7778 So you are saying that you found doing 512, 256, 128 FC layers and then finally 1 output, worked better than just a dense head? This was for just the image data? with or without dropout and batch normalization? My own experiments showed better scores without batch norm or dropout.
", "votes": null, "replies": [] }, { "id": 962119, "author_name": "arroqc", "author_url": "", "post_date": "08/07/2020 20:31:47", "content": "Interesting. For me adding dense + bn + relu + dropout blocks at the head decreased performance (by a small amount) compared to just a single linear classifier. Maybe I can try 2 layer Dense + Relu/Mish instead.
", "votes": null, "replies": [] }, { "id": 962154, "author_name": "datafan07", "author_url": "", "post_date": "08/07/2020 21:16:40", "content": "Interesting, I tried adding dropout layer but haven't tried dense layer yet…
", "votes": null, "replies": [] }, { "id": 962316, "author_name": "ajaykumar7778", "author_url": "", "post_date": "08/08/2020 03:31:40", "content": "Yes. Adding 3 FC layers instead of a single dense head worked better. This is only for image data. I did not use BatchNorm, but used a dropout of 0.3 for each of the FC layers.
", "votes": null, "replies": [] }, { "id": 962504, "author_name": "yimacs", "author_url": "", "post_date": "08/08/2020 07:15:59", "content": "@ajaykumar7778 thanks for sharing. do your experiments show that BN is no needed in your FC layers
\n\n@arroqc i'm using similar head - 2-3 blocks of dense + prelu + bn + dropout. the improvement is tiny. i wonder where's the best place for BN? after dense or after activation layer?
", "votes": null, "replies": [] }, { "id": 963004, "author_name": "arroqc", "author_url": "", "post_date": "08/08/2020 15:35:17", "content": "It's an open question. If you look for it on the web you'll find a lot of different answer. I think it has become common practice to do Dense + BN + activation + dropout. But like many things in machine learning you can just try it :)
", "votes": null, "replies": [] }, { "id": 963207, "author_name": "yimacs", "author_url": "", "post_date": "08/08/2020 18:50:23", "content": "@arroqc indeed, there are different views from big figures in the DL field. so one more thing to try.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "960993": "# Tips for fine-tuning EfficientNet\nOn unfreezing layers:\n\n- The `BathcNormalization` layers need to be kept frozen. If they are also turned to trainable, the first epoch after unfreezing will significantly reduce accuracy.\n- In some cases, it may be beneficial to open up only a portion of layers instead of unfreezing all. This will make fine-tuning much faster when going to larger models like `B7`.\n- **Each block needs to be all turned on or off**. This is because the architecture includes a shortcut from the first layer to the last layer for each block. Not respecting blocks also significantly harms the final performance.\n\nSome other tips for utilizing EfficientNet:\n- Larger variants of `EfficientNet` **do not guarantee improved performance**, especially for tasks **with less data** or **fewer classes**. In such a case, the larger variant of EfficientNet chosen, the harder it is to tune hyperparameters.\n- EMA (**Exponential Moving Average**) is very helpful in training EfficientNet from scratch, but not so much for transfer learning.\n- Do not use the `RMSprop` setup as in the original paper for transfer learning. The momentum and learning rate are too high for transfer learning. It will easily corrupt the pretrained weight and blow up the loss. A quick check is to see if loss (as `categorical cross-entropy`) is getting significantly larger than `log(NUM_CLASSES`) after the same epoch. If so, the initial `learning rate/momentum` is too high.\n- Smaller batch size benefit validation accuracy, possibly due to effectively providing regularization.\n- Using the latest EfficientNet weights.\n\n[Source](https://keras.io/examples/vision/image_classification_efficientnet_fine_tuning/).", "961106": "Why would you keep BatchNorm frozen? BatchNorm contains values that are trained to approximate the mean and standard deviation of the dataset the model was originally trained on...ImageNet, not the dataset you are trying to fine-tune on. BatchNorm will try to correct your input of melanoma data as it tries to differ from ImageNet data. I would think that if you were going to unfreeze anything it would be the BatchNorm layers.", "961155": "I think it's a common practice to freeze the `bn layer` while fine-tuning. When we unfreeze a model that contains `BatchNormalization` layers in order to do fine-tuning, we should keep the `BatchNormalization` layers in inference mode by passing `training=False` when calling the base model. Otherwise, the updates applied to the non-trainable weights will suddenly destroy what the model has learned.\n\nIMO, the `bn_layer` should be unfrozen if I can use a larger `batch size`, otherwise, it might not be stable properly.", "961544": "I agree with @brianfeeny, can't understand why freezing batch norm layer would create more stability or better results.\nNot saying that this is not true but it seems very counter-intuitive. Would you have a link to some content talking about this?", "961623": "optimo hmm, I see. I've already given the source link at the very end of this post, please follow the link. \n\nHowever, IMO, it actually depends on the custom data set. If it's way more **different** than the original ImageNet data set and we need to **fine-tune** the pre-trained weights on it and if we have to set **lower batch size** then, in this case, I think it's a bit risky to set `bn` layer **unfreeze**. But if we can accumulate a larger `batch size` (`Gradient Accumulation`), we can then set the `bn` layer unfreeze. Also, with lower `batch size`, we can also replace `BatchNormalization` with `Group` or `Instance` normalization (if we want).\n\nAlso, anyone can check directly from [here](https://keras.io/guides/transfer_learning/), written by François Chollet about fine-tuning - section:**Fine-tuning**, title `Important notes about BatchNormalization layer`.", "961642": "brianfeeny rather than downvoting, simple giving constructive feedback gives more sense, I believe. I can easily ignore that but It seems, it's happening [in other places](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/172892#961105). 😅", "961665": "ipythonx I don't downvote. If somehow something said I downvoted something then it was by accident. Edit: So I just read up on downvoting and if I had downvoted it would show in blue (to me), but your downvoted comments are in grey.", "961932": "I found adding dense head improved the model performance. Adding three Fully Connected Layers of sizes 512, 256 and 128 helped a lot. \nRectified Adam optimizer seems to work slightly better than adam. \nCyclical Learning rates and Focal loss also work better.", "962057": "ajaykumar7778 So you are saying that you found doing 512, 256, 128 FC layers and then finally 1 output, worked better than just a dense head? This was for just the image data? with or without dropout and batch normalization? My own experiments showed better scores without batch norm or dropout.", "962119": "Interesting. For me adding dense + bn + relu + dropout blocks at the head decreased performance (by a small amount) compared to just a single linear classifier. Maybe I can try 2 layer Dense + Relu/Mish instead.", "962154": "Interesting, I tried adding dropout layer but haven't tried dense layer yet...", "962316": "Yes. Adding 3 FC layers instead of a single dense head worked better. This is only for image data. I did not use BatchNorm, but used a dropout of 0.3 for each of the FC layers.", "962504": "ajaykumar7778 thanks for sharing. do your experiments show that BN is no needed in your FC layers \n\n@arroqc i'm using similar head - 2-3 blocks of dense + prelu + bn + dropout. the improvement is tiny. i wonder where's the best place for BN? after dense or after activation layer?", "963004": "It's an open question. If you look for it on the web you'll find a lot of different answer. I think it has become common practice to do Dense + BN + activation + dropout. But like many things in machine learning you can just try it :)", "963207": "arroqc indeed, there are different views from big figures in the DL field. so one more thing to try.", "1159798": "Do you know a Kaggle notebook applying fine-tuning on EfficientNet using PyTorch? I would be very interested to see a working example." }, "source": "meta" }