I am trying to do 5fold CV on kaggle notebooks, it stops midway after 9hrs, I only get 2 folds done in that time.
\nI use efficientnetb4, img size 512, batch size 8(or 2 sometimes).
It just takes too much time even with mixed-precision training.
", "messages": [ { "id": "1192497", "postDate": "02/09/2021 06:23:36", "content": "I am trying to do 5fold CV on kaggle notebooks, it stops midway after 9hrs, I only get 2 folds done in that time.
\nI use efficientnetb4, img size 512, batch size 8(or 2 sometimes).
It just takes too much time even with mixed-precision training.
", "rawMarkdown": "I am trying to do 5fold CV on kaggle notebooks, it stops midway after 9hrs, I only get 2 folds done in that time.\nI use efficientnetb4, img size 512, batch size 8(or 2 sometimes).\n\nIt just takes too much time even with mixed-precision training.", "votes": null }, { "id": "1192510", "postDate": "02/09/2021 06:36:55", "content": "Hello!
\nThere may be several reasons for such a long runtime. Consider the following, top to bottom:
\n(1-2) can be found in this community notebook, (3) can be found in this notebook by DimitreOliveira.
\nWith all the above done training EfficientNetB4 (image_size=(512, 512), batch_size=128, i.e. 16 samples per core) should take 2 minutes per epoch, up to 4 hours total on 5 KFold.
Woah! that's cool. I don't really know much about using TPU's, but I will try to learn it as this is quite an improvement over GPU.
\nDo I have to change my whole code to be able to use TPU or just the Training loop part?
You can turn on TPU anytime without changing anything in your code, but this won't give any notable difference, because the code should be optimized first. Alternatively, you can optimize your code so that it runs faster even without TPU.
\nUntil you decide to go for a custom training loop, the most important part to optimize is data workflow. In the both notebooks mentioned above you will see tf.data.TFRecordDataset class is utilized instead of more straightforward ImageDataGenerator class (which you probably used a lot).
And, of course, don't forget to parallellize your model wrapping it by with strategy.scope(): model = get_model(params) statement when running on TPU.
Some tips to reduce time:
\nnadam optimizer, and reduce epochs to 5 or 7I have implemented these in my notebook here. You can take inspiration from it, and please feel free to give suggestions.
\nHope this helps.
", "rawMarkdown": "Some tips to reduce time:\n\n * Do augmentation in another notebook. Make a dataset of those augmented images and feed those into your training notebook. Demo [here](https://www.kaggle.com/adityakane/cassava-stratified-k-folds-tfrecords-maker). \n * Use TPUs if you can\n * Use `nadam` optimizer, and reduce epochs to 5 or 7\n * Batch size: 16 or 32 for 299x299 images\n * Use tf.data module and @tf.function if ypu are using TensorFlow.\n\nI have implemented these in my notebook [here](https://www.kaggle.com/adityakane/cassava-gpu-preprocessed-trainer). You can take inspiration from it, and please feel free to give suggestions.\n\nHope this helps.", "votes": null }, { "id": "1193168", "postDate": "02/09/2021 13:49:32", "content": "Train 1 fold at a time
\nYou don't need to train all the folds in one go
\nYou can also use gradient accumulation to increase your batch size
Thank you very much I will try it out.
", "rawMarkdown": "Thank you very much I will try it out.", "votes": null }, { "id": "1193783", "postDate": "02/09/2021 21:18:09", "content": "Hi,
\nYour input preparation might be inefficient. Because, I did experiments with like exactly your parameters except I had gradient accumulation as well. If you can share your preparation script as well, we can take a look.
\nRegards
", "rawMarkdown": "Hi,\n\nYour input preparation might be inefficient. Because, I did experiments with like exactly your parameters except I had gradient accumulation as well. If you can share your preparation script as well, we can take a look. \n\nRegards", "votes": null } ], "comments": [ { "id": 1192510, "author_name": "nickuzmenkov", "author_url": "", "post_date": "02/09/2021 06:36:55", "content": "Hello!
\nThere may be several reasons for such a long runtime. Consider the following, top to bottom:
\n(1-2) can be found in this community notebook, (3) can be found in this notebook by DimitreOliveira.
\nWith all the above done training EfficientNetB4 (image_size=(512, 512), batch_size=128, i.e. 16 samples per core) should take 2 minutes per epoch, up to 4 hours total on 5 KFold.
Woah! that's cool. I don't really know much about using TPU's, but I will try to learn it as this is quite an improvement over GPU.
\nDo I have to change my whole code to be able to use TPU or just the Training loop part?
You can turn on TPU anytime without changing anything in your code, but this won't give any notable difference, because the code should be optimized first. Alternatively, you can optimize your code so that it runs faster even without TPU.
\nUntil you decide to go for a custom training loop, the most important part to optimize is data workflow. In the both notebooks mentioned above you will see tf.data.TFRecordDataset class is utilized instead of more straightforward ImageDataGenerator class (which you probably used a lot).
And, of course, don't forget to parallellize your model wrapping it by with strategy.scope(): model = get_model(params) statement when running on TPU.
Some tips to reduce time:
\nnadam optimizer, and reduce epochs to 5 or 7I have implemented these in my notebook here. You can take inspiration from it, and please feel free to give suggestions.
\nHope this helps.
", "votes": null, "replies": [] }, { "id": 1193168, "author_name": "debarshichanda", "author_url": "", "post_date": "02/09/2021 13:49:32", "content": "Train 1 fold at a time
\nYou don't need to train all the folds in one go
\nYou can also use gradient accumulation to increase your batch size
Thank you very much I will try it out.
", "votes": null, "replies": [] } ] }, { "id": 1193783, "author_name": "snnclsr", "author_url": "", "post_date": "02/09/2021 21:18:09", "content": "Hi,
\nYour input preparation might be inefficient. Because, I did experiments with like exactly your parameters except I had gradient accumulation as well. If you can share your preparation script as well, we can take a look.
\nRegards
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1192497": "I am trying to do 5fold CV on kaggle notebooks, it stops midway after 9hrs, I only get 2 folds done in that time.\nI use efficientnetb4, img size 512, batch size 8(or 2 sometimes).\n\nIt just takes too much time even with mixed-precision training.", "1192510": "Hello!\n\nThere may be several reasons for such a long runtime. Consider the following, top to bottom:\n1. Switch from GPU to TPU\n2. Use TFRecords instead of Jpegs\n3. Use custom training loop\n\n(1-2) can be found in this **[community notebook](https://www.kaggle.com/jessemostipak/getting-started-tpus-cassava-leaf-disease)**, (3) can be found in this **[notebook by DimitreOliveira](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-training-with-tpu-v2-pods)**.\nWith all the above done training EfficientNetB4 (`image_size=(512, 512)`, `batch_size=128`, i.e. 16 samples per core) should take 2 minutes per epoch, up to 4 hours total on 5 KFold.", "1192514": "Woah! that's cool. I don't really know much about using TPU's, but I will try to learn it as this is quite an improvement over GPU.\nDo I have to change my whole code to be able to use TPU or just the Training loop part?", "1192524": "You can turn on TPU anytime without changing anything in your code, but this won't give any notable difference, because the code should be optimized first. Alternatively, you can optimize your code so that it runs faster even without TPU.\n\nUntil you decide to go for a custom training loop, the most important part to optimize is data workflow. In the both notebooks mentioned above you will see `tf.data.TFRecordDataset` class is utilized instead of more straightforward `ImageDataGenerator` class (which you probably used a lot). \n\nAnd, of course, don't forget to parallellize your model wrapping it by `with strategy.scope(): model = get_model(params)` statement when running on TPU.", "1192863": "Some tips to reduce time:\n\n * Do augmentation in another notebook. Make a dataset of those augmented images and feed those into your training notebook. Demo [here](https://www.kaggle.com/adityakane/cassava-stratified-k-folds-tfrecords-maker). \n * Use TPUs if you can\n * Use `nadam` optimizer, and reduce epochs to 5 or 7\n * Batch size: 16 or 32 for 299x299 images\n * Use tf.data module and @tf.function if ypu are using TensorFlow.\n\nI have implemented these in my notebook [here](https://www.kaggle.com/adityakane/cassava-gpu-preprocessed-trainer). You can take inspiration from it, and please feel free to give suggestions.\n\nHope this helps.", "1193168": "Train 1 fold at a time\nYou don't need to train all the folds in one go\nYou can also use gradient accumulation to increase your batch size", "1193579": "Thank you very much I will try it out.", "1193783": "Hi,\n\nYour input preparation might be inefficient. Because, I did experiments with like exactly your parameters except I had gradient accumulation as well. If you can share your preparation script as well, we can take a look. \n\nRegards" }, "source": "meta" }