![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2F8dc2e33050deb235fa70a2514525d887%2FScreen%20Shot%202020-07-31%20at%208.09.28%20AM.png?generation=1596208333848945&alt=media\")
Hi.
\n\nLets say I want to do 5 image augmentations. Is there any benefit to creating new images and having 6 times the number of images in my training dataset? Or is it better to just randomly do a transformation on an image, and then replace the original image in your training dataset with the transformed image?
\n\nI guess I could try both, but having ~200,000 images in training would be a real pain.
\n\nThanks in advance : )
", "messages": [ { "id": "953008", "postDate": "07/31/2020 12:31:00", "content": "Hi.
\n\nLets say I want to do 5 image augmentations. Is there any benefit to creating new images and having 6 times the number of images in my training dataset? Or is it better to just randomly do a transformation on an image, and then replace the original image in your training dataset with the transformed image?
\n\nI guess I could try both, but having ~200,000 images in training would be a real pain.
\n\nThanks in advance : )
", "rawMarkdown": "Hi.\n\nLets say I want to do 5 image augmentations. Is there any benefit to creating new images and having 6 times the number of images in my training dataset? Or is it better to just randomly do a transformation on an image, and then replace the original image in your training dataset with the transformed image?\n\nI guess I could try both, but having ~200,000 images in training would be a real pain.\n\nThanks in advance : )", "votes": null }, { "id": "953079", "postDate": "07/31/2020 14:11:04", "content": "Pros:\n- Reduced computation by performing augmentations beforehand
\n\nCons:\n- Increased storage and bigger dataset to train\n- Instead of seeing different input images in different iterations, now the model can potentially see the same data (image rotated or flipped) at different iterations, leading to lower accuracy
", "rawMarkdown": "Pros:\n- Reduced computation by performing augmentations beforehand\n\nCons:\n- Increased storage and bigger dataset to train\n- Instead of seeing different input images in different iterations, now the model can potentially see the same data (image rotated or flipped) at different iterations, leading to **lower accuracy**", "votes": null }, { "id": "953102", "postDate": "07/31/2020 14:32:41", "content": "The standard way I've seen people doing it in this competition is to do it once at train time. I think during each epoch a random transformation is applied to a given image so the model will see different 'versions' of the same image.
\n\nI don't think this necessarily slows things down, as the batches are pre-loaded (and augmented) by the CPU before training is performed on the GPU/TPU.
\n\nI maybe wrong, but this is my understanding!
", "rawMarkdown": "The standard way I've seen people doing it in this competition is to do it once at train time. I think during each epoch a random transformation is applied to a given image so the model will see different 'versions' of the same image. \n\nI don't think this necessarily slows things down, as the batches are pre-loaded (and augmented) by the CPU before training is performed on the GPU/TPU.\n\nI maybe wrong, but this is my understanding!", "votes": null }, { "id": "953136", "postDate": "07/31/2020 14:56:37", "content": "\"Is there any benefit?\". Most likely No.
\n\nThe only possible benefit is faster training. You can check whether there would be a speed increase. Run your pipeline with and without augmentation. If the time to train each epoch is the same, then there is \"no benefit\". If your model trains faster without augmentation, then you could get a speed benefit from doing many augmentations ahead of time (and saving to disk), but you will also get disadvantages listed in other comments such as less variety of train images and lower model accuracy afterward.
\n\nI only recommend saving augmentation to disk beforehand in the case of extreme augmentation like GAN augmentation. Any other normal augmentations are best done randomly each epoch.
\n\nBoth GPU and TPU do augmentation on the CPU in parallel with training on GPU. So while your model is training one batch on GPU/TPU, your dataloader is augmenting and preparing the next batch for training. If your code is written efficiently, this CPU augmentation time will be less than GPU/TPU train time and not slow down training. Thus \"most likely no benefit\".
", "rawMarkdown": "\"Is there any benefit?\". Most likely No. \n\nThe only possible benefit is faster training. You can check whether there would be a speed increase. Run your pipeline **with** and **without** augmentation. If the time to train each epoch is the same, then there is \"no benefit\". If your model trains faster **without** augmentation, then you could get a speed benefit from doing many augmentations ahead of time (and saving to disk), but you will also get disadvantages listed in other comments such as less variety of train images and lower model accuracy afterward.\n\nI only recommend saving augmentation to disk beforehand in the case of extreme augmentation like GAN augmentation. Any other normal augmentations are best done randomly each epoch.\n\nBoth GPU and TPU do augmentation on the CPU in parallel with training on GPU. So while your model is training one batch on GPU/TPU, your dataloader is augmenting and preparing the next batch for training. If your code is written efficiently, this CPU augmentation time will be less than GPU/TPU train time and not slow down training. Thus \"most likely no benefit\".", "votes": null }, { "id": "953142", "postDate": "07/31/2020 14:59:52", "content": "Chris, is my understanding of how augmentations work across epochs correct? For a given image will it have different augmentations per epoch because of the df.repeat() call?
", "rawMarkdown": "Chris, is my understanding of how augmentations work across epochs correct? For a given image will it have different augmentations per epoch because of the df.repeat() call?", "votes": null }, { "id": "953153", "postDate": "07/31/2020 15:14:22", "content": "Yes. That is correct. When you train 30,000 images for 10 epochs with data augmentation, you will get 300,000 unique images! Here is an example. Start with a dataset of 1 element via ds.take(1). Then if you repeat this 1 element via ds.repeat() and apply random augmentation, you will see that tf.data.Dataset makes a new image each time:
ds = tf.data.TFRecordDataset(files_train, num_parallel_reads=AUTO).shuffle(1024)\n ds = ds.take(1).cache().repeat()\n ds = ds.map(read_unlabeled_tfrecord, num_parallel_calls=AUTO)\n ds = ds.map(lambda img, target: (data_augmentation(img), target) )\n
\n\n\nduring each epoch a random transformation is applied to a given image so the model will see different 'versions' of the same image.
\n
Thank you for your comment. I have a better understanding of this now. It seems I had not thought about this before - during each epoch, an augmentation is randomly applied. (My original question was not with respect to pre-computing augmentations - It was more to do with whether I need to increase the size of my training set by \"adding\" augmented images to my training set. But what you said makes sense - instead of \"adding\" new images, it'll just see an augmented version of the image in a different epoch).
", "rawMarkdown": "> during each epoch a random transformation is applied to a given image so the model will see different 'versions' of the same image.\n\nThank you for your comment. I have a better understanding of this now. It seems I had not thought about this before - during each epoch, an augmentation is randomly applied. (My original question was not with respect to pre-computing augmentations - It was more to do with whether I need to increase the size of my training set by \"adding\" augmented images to my training set. But what you said makes sense - instead of \"adding\" new images, it'll just see an augmented version of the image in a different epoch).", "votes": null }, { "id": "953982", "postDate": "08/01/2020 09:35:54", "content": "Thank you for your detailed comment, Chris. Just for completeness, I was not talking about pre-computing the augmentation (maybe my language was a bit confusing), but just about increasing the size of my training set by adding the augmentations. What you said about less variety of training images in doing such a thing makes sense. I guess it's best to apply augmentations randomly in each epoch.
", "rawMarkdown": "Thank you for your detailed comment, Chris. Just for completeness, I was not talking about pre-computing the augmentation (maybe my language was a bit confusing), but just about increasing the size of my training set by adding the augmentations. What you said about less variety of training images in doing such a thing makes sense. I guess it's best to apply augmentations randomly in each epoch.", "votes": null }, { "id": "953993", "postDate": "08/01/2020 09:46:26", "content": "Yes, randomly each epoch is best. If you want more training, just use more epochs.
\n\nBeing aware of precomputing augmentation is helpful though. There are some augmentations that are complex and slow down training. For example, Albumentations nonlinear augmentations like OpticalDistortion, GridDistortion, ElasticTransform will slow down training because our CPUs cannot generate batches fast enough. So if you want to use nonlinear augmentations, you could build 5-10 copies of the train data with random nonlinear augmentations and save to disk. Then train with this 5-10x larger train set and apply the fast augmentations to these images each epoch.
Pros:\n- Reduced computation by performing augmentations beforehand
\n\nCons:\n- Increased storage and bigger dataset to train\n- Instead of seeing different input images in different iterations, now the model can potentially see the same data (image rotated or flipped) at different iterations, leading to lower accuracy
", "votes": null, "replies": [] }, { "id": 953102, "author_name": "fchmiel", "author_url": "", "post_date": "07/31/2020 14:32:41", "content": "The standard way I've seen people doing it in this competition is to do it once at train time. I think during each epoch a random transformation is applied to a given image so the model will see different 'versions' of the same image.
\n\nI don't think this necessarily slows things down, as the batches are pre-loaded (and augmented) by the CPU before training is performed on the GPU/TPU.
\n\nI maybe wrong, but this is my understanding!
", "votes": null, "replies": [ { "id": 953973, "author_name": "teamaker", "author_url": "", "post_date": "08/01/2020 09:30:17", "content": "\n\n\nduring each epoch a random transformation is applied to a given image so the model will see different 'versions' of the same image.
\n
Thank you for your comment. I have a better understanding of this now. It seems I had not thought about this before - during each epoch, an augmentation is randomly applied. (My original question was not with respect to pre-computing augmentations - It was more to do with whether I need to increase the size of my training set by \"adding\" augmented images to my training set. But what you said makes sense - instead of \"adding\" new images, it'll just see an augmented version of the image in a different epoch).
", "votes": null, "replies": [] } ] }, { "id": 953136, "author_name": "cdeotte", "author_url": "", "post_date": "07/31/2020 14:56:37", "content": "\"Is there any benefit?\". Most likely No.
\n\nThe only possible benefit is faster training. You can check whether there would be a speed increase. Run your pipeline with and without augmentation. If the time to train each epoch is the same, then there is \"no benefit\". If your model trains faster without augmentation, then you could get a speed benefit from doing many augmentations ahead of time (and saving to disk), but you will also get disadvantages listed in other comments such as less variety of train images and lower model accuracy afterward.
\n\nI only recommend saving augmentation to disk beforehand in the case of extreme augmentation like GAN augmentation. Any other normal augmentations are best done randomly each epoch.
\n\nBoth GPU and TPU do augmentation on the CPU in parallel with training on GPU. So while your model is training one batch on GPU/TPU, your dataloader is augmenting and preparing the next batch for training. If your code is written efficiently, this CPU augmentation time will be less than GPU/TPU train time and not slow down training. Thus \"most likely no benefit\".
", "votes": null, "replies": [ { "id": 953142, "author_name": "fchmiel", "author_url": "", "post_date": "07/31/2020 14:59:52", "content": "Chris, is my understanding of how augmentations work across epochs correct? For a given image will it have different augmentations per epoch because of the df.repeat() call?
", "votes": null, "replies": [] }, { "id": 953153, "author_name": "cdeotte", "author_url": "", "post_date": "07/31/2020 15:14:22", "content": "Yes. That is correct. When you train 30,000 images for 10 epochs with data augmentation, you will get 300,000 unique images! Here is an example. Start with a dataset of 1 element via ds.take(1). Then if you repeat this 1 element via ds.repeat() and apply random augmentation, you will see that tf.data.Dataset makes a new image each time:
ds = tf.data.TFRecordDataset(files_train, num_parallel_reads=AUTO).shuffle(1024)\n ds = ds.take(1).cache().repeat()\n ds = ds.map(read_unlabeled_tfrecord, num_parallel_calls=AUTO)\n ds = ds.map(lambda img, target: (data_augmentation(img), target) )\n
Thank you for your detailed comment, Chris. Just for completeness, I was not talking about pre-computing the augmentation (maybe my language was a bit confusing), but just about increasing the size of my training set by adding the augmentations. What you said about less variety of training images in doing such a thing makes sense. I guess it's best to apply augmentations randomly in each epoch.
", "votes": null, "replies": [] }, { "id": 953993, "author_name": "cdeotte", "author_url": "", "post_date": "08/01/2020 09:46:26", "content": "Yes, randomly each epoch is best. If you want more training, just use more epochs.
\n\nBeing aware of precomputing augmentation is helpful though. There are some augmentations that are complex and slow down training. For example, Albumentations nonlinear augmentations like OpticalDistortion, GridDistortion, ElasticTransform will slow down training because our CPUs cannot generate batches fast enough. So if you want to use nonlinear augmentations, you could build 5-10 copies of the train data with random nonlinear augmentations and save to disk. Then train with this 5-10x larger train set and apply the fast augmentations to these images each epoch.