I am trying to check various things like different models, image augmentations, etc. For checking purpose only, I am training for 2 epochs and it takes more than 12 Min for ResNet50 to train. Any ideas to speed up things.
", "messages": [ { "id": "568078", "postDate": "07/04/2019 11:16:01", "content": "I am trying to check various things like different models, image augmentations, etc. For checking purpose only, I am training for 2 epochs and it takes more than 12 Min for ResNet50 to train. Any ideas to speed up things.
", "rawMarkdown": "I am trying to check various things like different models, image augmentations, etc. For checking purpose only, I am training for 2 epochs and it takes more than 12 Min for ResNet50 to train. Any ideas to speed up things.", "votes": null }, { "id": "568232", "postDate": "07/04/2019 15:15:39", "content": "Do you use kernel for training? If yes, you can try make I/O overhead smaller. I mean:\n1. Read all images to memory (of course resize them). Not sure if will fir to memory\n2. Download dataset locally, resize it, upload to you kernel.
\n\nI use already resized image locally and then I have 4x speed up of training. One epoch take 30s at 1080.
", "rawMarkdown": "Do you use kernel for training? If yes, you can try make I/O overhead smaller. I mean:\n1. Read all images to memory (of course resize them). Not sure if will fir to memory\n2. Download dataset locally, resize it, upload to you kernel. \n\nI use already resized image locally and then I have 4x speed up of training. One epoch take 30s at 1080.", "votes": null }, { "id": "568670", "postDate": "07/05/2019 09:29:08", "content": "After downloading the dataset locally, to what size you are resizing to it?
", "rawMarkdown": "After downloading the dataset locally, to what size you are resizing to it?", "votes": null }, { "id": "568826", "postDate": "07/05/2019 13:50:09", "content": "It helped me by bringing 1 epoch to 60 sec from 400+ sec. Thanks
", "rawMarkdown": "It helped me by bringing 1 epoch to 60 sec from 400+ sec. Thanks", "votes": null }, { "id": "569509", "postDate": "07/06/2019 19:25:57", "content": "Hi! @nitin29, I'm already thinking to run GCP ML Engine or AWS SageMaker with Hyperparameter tunning in a powerfull VM with GPUs, other options are reducing the image size, datasets size, or the number of epochs but they can generate different results and point you in the wrong direction.
", "rawMarkdown": "Hi! @nitin29, I'm already thinking to run GCP ML Engine or AWS SageMaker with Hyperparameter tunning in a powerfull VM with GPUs, other options are reducing the image size, datasets size, or the number of epochs but they can generate different results and point you in the wrong direction.", "votes": null }, { "id": "569556", "postDate": "07/06/2019 23:03:34", "content": "Pytorch and Apex for training with mixed precision can be a big time saver, if you have appropriate GPU :)
\n\nhttps://github.com/NVIDIA/apex
", "rawMarkdown": "Pytorch and Apex for training with mixed precision can be a big time saver, if you have appropriate GPU :)\n\nhttps://github.com/NVIDIA/apex", "votes": null } ], "comments": [ { "id": 568232, "author_name": "melgor", "author_url": "", "post_date": "07/04/2019 15:15:39", "content": "Do you use kernel for training? If yes, you can try make I/O overhead smaller. I mean:\n1. Read all images to memory (of course resize them). Not sure if will fir to memory\n2. Download dataset locally, resize it, upload to you kernel.
\n\nI use already resized image locally and then I have 4x speed up of training. One epoch take 30s at 1080.
", "votes": null, "replies": [ { "id": 568670, "author_name": "nitin29", "author_url": "", "post_date": "07/05/2019 09:29:08", "content": "After downloading the dataset locally, to what size you are resizing to it?
", "votes": null, "replies": [] }, { "id": 568826, "author_name": "nitin29", "author_url": "", "post_date": "07/05/2019 13:50:09", "content": "It helped me by bringing 1 epoch to 60 sec from 400+ sec. Thanks
", "votes": null, "replies": [] } ] }, { "id": 569509, "author_name": "cv13j0", "author_url": "", "post_date": "07/06/2019 19:25:57", "content": "Hi! @nitin29, I'm already thinking to run GCP ML Engine or AWS SageMaker with Hyperparameter tunning in a powerfull VM with GPUs, other options are reducing the image size, datasets size, or the number of epochs but they can generate different results and point you in the wrong direction.
", "votes": null, "replies": [] }, { "id": 569556, "author_name": "taindow", "author_url": "", "post_date": "07/06/2019 23:03:34", "content": "Pytorch and Apex for training with mixed precision can be a big time saver, if you have appropriate GPU :)
\n\n", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "568078": "I am trying to check various things like different models, image augmentations, etc. For checking purpose only, I am training for 2 epochs and it takes more than 12 Min for ResNet50 to train. Any ideas to speed up things.", "568232": "Do you use kernel for training? If yes, you can try make I/O overhead smaller. I mean:\n1. Read all images to memory (of course resize them). Not sure if will fir to memory\n2. Download dataset locally, resize it, upload to you kernel. \n\nI use already resized image locally and then I have 4x speed up of training. One epoch take 30s at 1080.", "568670": "After downloading the dataset locally, to what size you are resizing to it?", "568826": "It helped me by bringing 1 epoch to 60 sec from 400+ sec. Thanks", "569509": "Hi! @nitin29, I'm already thinking to run GCP ML Engine or AWS SageMaker with Hyperparameter tunning in a powerfull VM with GPUs, other options are reducing the image size, datasets size, or the number of epochs but they can generate different results and point you in the wrong direction.", "569556": "Pytorch and Apex for training with mixed precision can be a big time saver, if you have appropriate GPU :)\n\nhttps://github.com/NVIDIA/apex" }, "source": "meta" }