According to YOLOv5 official, 300 is recommended for default epoch number.
\nBut, partly due to data size or training setting, training saturates early. For some simple situation, epoch number ~30 is enough to get a best validation result.
How do you set the training schedule including epoch number?
\nAlthough \"small epoch number is better.\" sounds like gospel for guys with poor resource (like me), it should be considered carefully.
Thanks.
", "messages": [ { "id": "1652730", "postDate": "01/16/2022 23:59:32", "content": "According to YOLOv5 official, 300 is recommended for default epoch number.
\nBut, partly due to data size or training setting, training saturates early. For some simple situation, epoch number ~30 is enough to get a best validation result.
How do you set the training schedule including epoch number?
\nAlthough \"small epoch number is better.\" sounds like gospel for guys with poor resource (like me), it should be considered carefully.
Thanks.
", "rawMarkdown": "According to [YOLOv5 official](https://github.com/ultralytics/yolov5/wiki/Train-Custom-Data), 300 is recommended for default epoch number.\nBut, partly due to data size or training setting, training saturates early. For some simple situation, epoch number ~30 is enough to get a best validation result.\n\nHow do you set the training schedule including epoch number?\nAlthough \"small epoch number is better.\" sounds like gospel for guys with poor resource (like me), it should be considered carefully.\n\nThanks.", "votes": null }, { "id": "1652874", "postDate": "01/17/2022 04:28:25", "content": "with pretrained weight, fine tuning around 10 epochs would get pretty good result.
\nfor better result, GPU resource is a must.
Thank you for your reply. I have a similar feel with you.
\nI wonder: If we have much more epochs (100~), a model just gets overfitted even if along with strong data-augmentation methods.
", "rawMarkdown": "Thank you for your reply. I have a similar feel with you.\n\nI wonder: If we have much more epochs (100~), a model just gets overfitted even if along with strong data-augmentation methods.", "votes": null }, { "id": "1652984", "postDate": "01/17/2022 06:35:50", "content": "Keep in mind that the 300-epoch recommendation is for 80 objects in COCO. For 1 object, you don't need as many epochs.
\nTrain until validation loss does not improve with further cycles. It is hard to say how many is correct, as heavier augmentation or lower learning rate will lead to slower learning but may result in better mAP.
\nOvertraining the model may result in higher precision and lower recall, which is bad for this competition as the F2 metric favors recall over precision.
", "rawMarkdown": "Keep in mind that the 300-epoch recommendation is for 80 objects in COCO. For 1 object, you don't need as many epochs.\n\nTrain until validation loss does not improve with further cycles. It is hard to say how many is correct, as heavier augmentation or lower learning rate will lead to slower learning but may result in better mAP.\n\nOvertraining the model may result in higher precision and lower recall, which is bad for this competition as the F2 metric favors recall over precision.", "votes": null }, { "id": "1653018", "postDate": "01/17/2022 07:09:50", "content": "Thanks a lot. Makes sense to me.
\nBecause I was skeptical about my understanding, your comment made me trustable. :)
It is very hard to determine training schedule, such as epoch number, learning rate, and heaviness of DA method. To estimate delicate balance among these, I think trust CV method is most important.
", "rawMarkdown": "Thanks a lot. Makes sense to me.\nBecause I was skeptical about my understanding, your comment made me trustable. :)\n\nIt is very hard to determine training schedule, such as epoch number, learning rate, and heaviness of DA method. To estimate delicate balance among these, I think trust CV method is most important.", "votes": null }, { "id": "1653076", "postDate": "01/17/2022 08:27:41", "content": "How much epoch? It depends ….. You should look into metrics … and choose correct model (according to metrics and your goal). Just train yolov5 with --save-period 1 and choose model.
I am grateful for your kind advice. I agree with you.
\nI'll try using --save-period option.
Second one in fitness strategy in yolov5. You can automate process of \"best.pt\" model (https://github.com/ultralytics/yolov5/blob/affa284352fa6d094d32fe2be69dbffe36bd20f8/utils/metrics.py#L15-L19)
\ndef fitness(x):\n w = [0.0, 0.0, 0.1, 0.9] # weights for [P, R, mAP@0.5, mAP@0.5:0.95]\n return (x[:, :4] * w).sum(1)\nfor really custom things you can modify code here: https://github.com/ultralytics/yolov5/blob/affa284352fa6d094d32fe2be69dbffe36bd20f8/train.py#L376-L395
", "rawMarkdown": "Second one in fitness strategy in yolov5. You can automate process of \"best.pt\" model (https://github.com/ultralytics/yolov5/blob/affa284352fa6d094d32fe2be69dbffe36bd20f8/utils/metrics.py#L15-L19)\n\n```python\ndef fitness(x):\n w = [0.0, 0.0, 0.1, 0.9] # weights for [P, R, mAP@0.5, mAP@0.5:0.95]\n return (x[:, :4] * w).sum(1)\n```\n\nfor really custom things you can modify code here: https://github.com/ultralytics/yolov5/blob/affa284352fa6d094d32fe2be69dbffe36bd20f8/train.py#L376-L395", "votes": null }, { "id": "1653487", "postDate": "01/17/2022 15:19:37", "content": "You can use YOLO's --evolve parameter to tune hyperparameters.
", "rawMarkdown": "You can use YOLO's --evolve parameter to tune hyperparameters.", "votes": null }, { "id": "1653520", "postDate": "01/17/2022 15:55:49", "content": "Great information.
\nAre there some recommendations about these weights?
\nIn GBR condition, recall may be worth to focus on, but it's hard to define better ratio;(
Thank you for your suggestion.
\nWould you tell me how expensive is it in computational cost?
It's the same compute as training. It just tries to tune params to the 'fitness' metrics.
\nHere's the overview from Ultralytics -> https://docs.ultralytics.com/tutorials/hyperparameter-evolution/
", "rawMarkdown": "It's the same compute as training. It just tries to tune params to the 'fitness' metrics.\n\nHere's the overview from Ultralytics -> https://docs.ultralytics.com/tutorials/hyperparameter-evolution/", "votes": null }, { "id": "1653611", "postDate": "01/17/2022 17:31:00", "content": "But …. evolve requires GPU resources (a lot) and good baseline model. You do not improve poor model (maybe a little bit) - it is bed strategy. So firstly work on dataset, create strong model and then tune.
", "rawMarkdown": "But …. evolve requires GPU resources (a lot) and good baseline model. You do not improve poor model (maybe a little bit) - it is bed strategy. So firstly work on dataset, create strong model and then tune.", "votes": null }, { "id": "1653858", "postDate": "01/17/2022 23:40:02", "content": "I thought evolve option was so expensive!
\nI'm thinking of challenge it if I get good models.
Thank you for giving me a lot of advice on my question.
\nI say.. First thing's first. We have to hurry! 🔥
Evolve doesn't require more GPU memory, just more time, because it does multiple runs. I found it easier to smooth out my training with evolve than by manually trying random hyperparameters. Of course if you're training on Kaggle or Colab, time is an issue.
", "rawMarkdown": "Evolve doesn't require more GPU memory, just more time, because it does multiple runs. I found it easier to smooth out my training with evolve than by manually trying random hyperparameters. Of course if you're training on Kaggle or Colab, time is an issue.", "votes": null } ], "comments": [ { "id": 1652874, "author_name": "dragonzhang", "author_url": "", "post_date": "01/17/2022 04:28:25", "content": "with pretrained weight, fine tuning around 10 epochs would get pretty good result.
\nfor better result, GPU resource is a must.
Thank you for your reply. I have a similar feel with you.
\nI wonder: If we have much more epochs (100~), a model just gets overfitted even if along with strong data-augmentation methods.
", "votes": null, "replies": [] } ] }, { "id": 1652984, "author_name": "alexchwong", "author_url": "", "post_date": "01/17/2022 06:35:50", "content": "Keep in mind that the 300-epoch recommendation is for 80 objects in COCO. For 1 object, you don't need as many epochs.
\nTrain until validation loss does not improve with further cycles. It is hard to say how many is correct, as heavier augmentation or lower learning rate will lead to slower learning but may result in better mAP.
\nOvertraining the model may result in higher precision and lower recall, which is bad for this competition as the F2 metric favors recall over precision.
", "votes": null, "replies": [ { "id": 1653018, "author_name": "okunot", "author_url": "", "post_date": "01/17/2022 07:09:50", "content": "Thanks a lot. Makes sense to me.
\nBecause I was skeptical about my understanding, your comment made me trustable. :)
It is very hard to determine training schedule, such as epoch number, learning rate, and heaviness of DA method. To estimate delicate balance among these, I think trust CV method is most important.
", "votes": null, "replies": [] }, { "id": 1653487, "author_name": "davidbroberts", "author_url": "", "post_date": "01/17/2022 15:19:37", "content": "You can use YOLO's --evolve parameter to tune hyperparameters.
", "votes": null, "replies": [] }, { "id": 1653526, "author_name": "okunot", "author_url": "", "post_date": "01/17/2022 16:01:54", "content": "Thank you for your suggestion.
\nWould you tell me how expensive is it in computational cost?
It's the same compute as training. It just tries to tune params to the 'fitness' metrics.
\nHere's the overview from Ultralytics -> https://docs.ultralytics.com/tutorials/hyperparameter-evolution/
", "votes": null, "replies": [ { "id": 1653858, "author_name": "okunot", "author_url": "", "post_date": "01/17/2022 23:40:02", "content": "I thought evolve option was so expensive!
\nI'm thinking of challenge it if I get good models.
But …. evolve requires GPU resources (a lot) and good baseline model. You do not improve poor model (maybe a little bit) - it is bed strategy. So firstly work on dataset, create strong model and then tune.
", "votes": null, "replies": [] }, { "id": 1653868, "author_name": "okunot", "author_url": "", "post_date": "01/17/2022 23:46:54", "content": "Thank you for giving me a lot of advice on my question.
\nI say.. First thing's first. We have to hurry! 🔥
Evolve doesn't require more GPU memory, just more time, because it does multiple runs. I found it easier to smooth out my training with evolve than by manually trying random hyperparameters. Of course if you're training on Kaggle or Colab, time is an issue.
", "votes": null, "replies": [] } ] }, { "id": 1653076, "author_name": "remekkinas", "author_url": "", "post_date": "01/17/2022 08:27:41", "content": "How much epoch? It depends ….. You should look into metrics … and choose correct model (according to metrics and your goal). Just train yolov5 with --save-period 1 and choose model.
I am grateful for your kind advice. I agree with you.
\nI'll try using --save-period option.
Second one in fitness strategy in yolov5. You can automate process of \"best.pt\" model (https://github.com/ultralytics/yolov5/blob/affa284352fa6d094d32fe2be69dbffe36bd20f8/utils/metrics.py#L15-L19)
\ndef fitness(x):\n w = [0.0, 0.0, 0.1, 0.9] # weights for [P, R, mAP@0.5, mAP@0.5:0.95]\n return (x[:, :4] * w).sum(1)\nfor really custom things you can modify code here: https://github.com/ultralytics/yolov5/blob/affa284352fa6d094d32fe2be69dbffe36bd20f8/train.py#L376-L395
", "votes": null, "replies": [] }, { "id": 1653520, "author_name": "okunot", "author_url": "", "post_date": "01/17/2022 15:55:49", "content": "Great information.
\nAre there some recommendations about these weights?
\nIn GBR condition, recall may be worth to focus on, but it's hard to define better ratio;(