![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2151083%2F84dd6a24f8a8bc5eae8fe630e4da3387%2FScreenshot%20from%202019-08-19%2016-39-53.png?generation=1566223556655606&alt=media\")
I start to train model with my GeForce RTX 2070 and RAM 16 gb but it takes a lot of time. For example, 1 epoch takes more then 10 hours. Is it normal? Because I want to train at least 50 epochs but it will take more then 3 weeks. I am using Tiny-yolo model.
", "messages": [ { "id": "602639", "postDate": "08/19/2019 10:01:00", "content": "I start to train model with my GeForce RTX 2070 and RAM 16 gb but it takes a lot of time. For example, 1 epoch takes more then 10 hours. Is it normal? Because I want to train at least 50 epochs but it will take more then 3 weeks. I am using Tiny-yolo model.
", "rawMarkdown": "I start to train model with my GeForce RTX 2070 and RAM 16 gb but it takes a lot of time. For example, 1 epoch takes more then 10 hours. Is it normal? Because I want to train at least 50 epochs but it will take more then 3 weeks. I am using Tiny-yolo model.", "votes": null }, { "id": "602749", "postDate": "08/19/2019 12:57:55", "content": "Did you check whether the cuda is being utilized during training or not. \nWhich framework are you using for your task?
", "rawMarkdown": "Did you check whether the cuda is being utilized during training or not. \nWhich framework are you using for your task?", "votes": null }, { "id": "602803", "postDate": "08/19/2019 14:05:15", "content": "@thanatoz I am using YOLO (in darknet) with weights tiny-yolov2. Before starting training:\n
After:\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2151083%2Fe0a4c2bf2812e043b0ae475579aac2f9%2FScreenshot%20from%202019-08-19%2016-39-22.png?generation=1566223598385985&alt=media\")
I can clearly see that your model has been using the Cuda. As the size of the images in the competition is fairly large, so it could be the reason for such a slow training time. I should suggest you try some other algorithms other than Yolo for the same. That should help (Although the model is going to take longer timings to train. My model took about 21 hours on colab which I trained on the testing dataset and validated on the validation dataset.)
", "rawMarkdown": "I can clearly see that your model has been using the Cuda. As the size of the images in the competition is fairly large, so it could be the reason for such a slow training time. I should suggest you try some other algorithms other than Yolo for the same. That should help (Although the model is going to take longer timings to train. My model took about 21 hours on colab which I trained on the testing dataset and validated on the validation dataset.)", "votes": null }, { "id": "603132", "postDate": "08/19/2019 22:27:41", "content": "unfortunately this is actually not a bad timing at all on this dataset. Only reading the dataset takes a few hours with no processing...\nsome commonts - tiny yolo will not get you very far. unless you are doing it just for fun to see how training works, I doubt you can go over 0.30 with it. It does not have enough depth for so many classes. you will be MUCH better off fine tuning the yolov3 model that was already trained on v3 or v4 of the open images dataset (see yolov3 homepage, search for open images)
", "rawMarkdown": "unfortunately this is actually not a bad timing at all on this dataset. Only reading the dataset takes a few hours with no processing...\nsome commonts - tiny yolo will not get you very far. unless you are doing it just for fun to see how training works, I doubt you can go over 0.30 with it. It does not have enough depth for so many classes. you will be MUCH better off fine tuning the yolov3 model that was already trained on v3 or v4 of the open images dataset (see yolov3 homepage, search for open images)", "votes": null } ], "comments": [ { "id": 602749, "author_name": "thanatoz", "author_url": "", "post_date": "08/19/2019 12:57:55", "content": "Did you check whether the cuda is being utilized during training or not. \nWhich framework are you using for your task?
", "votes": null, "replies": [] }, { "id": 602803, "author_name": "ikikiki", "author_url": "", "post_date": "08/19/2019 14:05:15", "content": "@thanatoz I am using YOLO (in darknet) with weights tiny-yolov2. Before starting training:\n
I can clearly see that your model has been using the Cuda. As the size of the images in the competition is fairly large, so it could be the reason for such a slow training time. I should suggest you try some other algorithms other than Yolo for the same. That should help (Although the model is going to take longer timings to train. My model took about 21 hours on colab which I trained on the testing dataset and validated on the validation dataset.)
", "votes": null, "replies": [] } ] }, { "id": 602805, "author_name": "ikikiki", "author_url": "", "post_date": "08/19/2019 14:06:40", "content": "After:\n
unfortunately this is actually not a bad timing at all on this dataset. Only reading the dataset takes a few hours with no processing...\nsome commonts - tiny yolo will not get you very far. unless you are doing it just for fun to see how training works, I doubt you can go over 0.30 with it. It does not have enough depth for so many classes. you will be MUCH better off fine tuning the yolov3 model that was already trained on v3 or v4 of the open images dataset (see yolov3 homepage, search for open images)
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "602639": "I start to train model with my GeForce RTX 2070 and RAM 16 gb but it takes a lot of time. For example, 1 epoch takes more then 10 hours. Is it normal? Because I want to train at least 50 epochs but it will take more then 3 weeks. I am using Tiny-yolo model.", "602749": "Did you check whether the cuda is being utilized during training or not. \nWhich framework are you using for your task?", "602803": "thanatoz I am using YOLO (in darknet) with weights tiny-yolov2. Before starting training:\n", "602805": "After:\n", "603030": "I can clearly see that your model has been using the Cuda. As the size of the images in the competition is fairly large, so it could be the reason for such a slow training time. I should suggest you try some other algorithms other than Yolo for the same. That should help (Although the model is going to take longer timings to train. My model took about 21 hours on colab which I trained on the testing dataset and validated on the validation dataset.)", "603132": "unfortunately this is actually not a bad timing at all on this dataset. Only reading the dataset takes a few hours with no processing...\nsome commonts - tiny yolo will not get you very far. unless you are doing it just for fun to see how training works, I doubt you can go over 0.30 with it. It does not have enough depth for so many classes. you will be MUCH better off fine tuning the yolov3 model that was already trained on v3 or v4 of the open images dataset (see yolov3 homepage, search for open images)" }, "source": "meta" }