Apologize if this question is too elementary . I would like to understand whether by doing data augmentation , we are actually increasing the number of images in the training dataset. For example,If I do random crop to an image , then will the augmented dataset have the actual image and randomly cropped image ? .What is the effect of augmentation on the number of training images ?.Thanks in advance for the help .
", "messages": [ { "id": "752515", "postDate": "02/21/2020 05:23:57", "content": "Apologize if this question is too elementary . I would like to understand whether by doing data augmentation , we are actually increasing the number of images in the training dataset. For example,If I do random crop to an image , then will the augmented dataset have the actual image and randomly cropped image ? .What is the effect of augmentation on the number of training images ?.Thanks in advance for the help .
", "rawMarkdown": "Apologize if this question is too elementary . I would like to understand whether by doing data augmentation , we are actually increasing the number of images in the training dataset. For example,If I do random crop to an image , then will the augmented dataset have the actual image and randomly cropped image ? .What is the effect of augmentation on the number of training images ?.Thanks in advance for the help .", "votes": null }, { "id": "752670", "postDate": "02/21/2020 09:32:22", "content": "All the original images are just transformed (i.e. rotation, zooming, etc.) every epoch and then used for training. Therefore, the number of images in each epoch is equal to the number of original images you have.\nVia this answer on Stack Overflow
", "rawMarkdown": "All the original images are just transformed (i.e. rotation, zooming, etc.) every epoch and then used for training. Therefore, the number of images in each epoch is equal to the number of original images you have.\n[Via this answer on Stack Overflow](https://stackoverflow.com/questions/51748514/does-imagedatagenerator-add-more-images-to-my-dataset)", "votes": null }, { "id": "752814", "postDate": "02/21/2020 12:38:10", "content": "Following up on what @atamazian said, the thing is, having a random variant of the transformation applied for each epoch makes it look like every epoch was trained on a different set of images enabling the model to focus on the important features that makes up the label and ignore effects that might have been confusing to like translation, rotation, etc...
", "rawMarkdown": "Following up on what @atamazian said, the thing is, having a random variant of the transformation applied for each epoch makes it look like every epoch was trained on a different set of images enabling the model to focus on the important features that makes up the label and ignore effects that might have been confusing to like translation, rotation, etc...", "votes": null }, { "id": "755431", "postDate": "02/24/2020 19:13:35", "content": "The augmented dataset will have more images only if you decide to generate more images. Applying transformation to existing images does not change the number. In the tf,data,Dataset API, you can use flat_map togenerate multiple images from a single input and merge them into the dataset. If you use map, then you are applying a transformation in place and the number of elements in the dataset is not affected.
", "rawMarkdown": "The augmented dataset will have more images only if you decide to generate more images. Applying transformation to existing images does not change the number. In the tf,data,Dataset API, you can use [flat_map](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#flat_map) togenerate multiple images from a single input and merge them into the dataset. If you use [map](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#map), then you are applying a transformation in place and the number of elements in the dataset is not affected.", "votes": null }, { "id": "755683", "postDate": "02/25/2020 03:00:51", "content": "Thank you @mgornergoogle ..This is much helpful and I will try to use both the options and see how the score behaves ..
", "rawMarkdown": "Thank you @mgornergoogle ..This is much helpful and I will try to use both the options and see how the score behaves ..", "votes": null }, { "id": "755684", "postDate": "02/25/2020 03:01:09", "content": "Thank you @msheriey
", "rawMarkdown": "Thank you @msheriey", "votes": null }, { "id": "755685", "postDate": "02/25/2020 03:01:26", "content": "Thanks for the link @atamazian
", "rawMarkdown": "Thanks for the link @atamazian", "votes": null }, { "id": "755787", "postDate": "02/25/2020 06:00:14", "content": "Yes. Using data augmentation creates more images. The training dataset has 12,753 images. If you augment them, then each epoch you have a new set of 12,753. Therefore over 20 epochs, you will train with 255,060 images! This is why data augmentation increases model accuracy.
", "rawMarkdown": "Yes. Using data augmentation creates more images. The training dataset has 12,753 images. If you augment them, then each epoch you have a new set of 12,753. Therefore over 20 epochs, you will train with 255,060 images! This is why data augmentation increases model accuracy.", "votes": null }, { "id": "756186", "postDate": "02/25/2020 13:42:48", "content": "Thank you @cdeotte for the answer .. I have a followup question..
\n\n\n\n\nIf you augment them, then each epoch you have a new set of 12,753
\n
1.The augmentation is done for the training data and lets say for example I do random crop , increase brightness for rose flower . Therefore , in one epoch random crop is done and in next epoch the same rose flower is considered and increase brightness augmentation is applied ? Is this how the augmentation works ?
\n\nThanks in advance.
", "rawMarkdown": "Thank you @cdeotte for the answer .. I have a followup question..\n\n> If you augment them, then each epoch you have a new set of 12,753\n\n1.The augmentation is done for the training data and lets say for example I do random crop , increase brightness for rose flower . Therefore , in one epoch random crop is done and in next epoch the same rose flower is considered and increase brightness augmentation is applied ? Is this how the augmentation works ?\n\nThanks in advance.", "votes": null }, { "id": "756230", "postDate": "02/25/2020 14:31:05", "content": "Just happened to see your related answer for a similar question . I guess the answer for the above question can be inferred by understanding what is happening with
\n\n\n\n\ndataset.repeat()
\n
function . Pls correct me if I am wrong.
", "rawMarkdown": "Just happened to see your related answer for a [similar question ](https://www.kaggle.com/c/flower-classification-with-tpus/discussion/132127). I guess the answer for the above question can be inferred by understanding what is happening with \n> dataset.repeat() \n\nfunction . Pls correct me if I am wrong.", "votes": null }, { "id": "756360", "postDate": "02/25/2020 16:48:35", "content": "The best way to know what TensorFlow is doing each epoch is to build your chain of augmentation and then view the images. I posted code in this discussion for viewing. Also when you are viewing your augmentation, set the random ranges very high like tf.image.adjust_contrast(x,LARGE_NUMBER), that way you can see what is happening, then lower the LARGE_NUMBER until it looks reasonable and use the more reasonable number for training.
Thank you for the tips @cdeotte ..I will play around with the number as mentioned and check its behaviour ..
", "rawMarkdown": "Thank you for the tips @cdeotte ..I will play around with the number as mentioned and check its behaviour ..", "votes": null } ], "comments": [ { "id": 752670, "author_name": "atamazian", "author_url": "", "post_date": "02/21/2020 09:32:22", "content": "All the original images are just transformed (i.e. rotation, zooming, etc.) every epoch and then used for training. Therefore, the number of images in each epoch is equal to the number of original images you have.\nVia this answer on Stack Overflow
", "votes": null, "replies": [ { "id": 755685, "author_name": "gsdeepakkumar", "author_url": "", "post_date": "02/25/2020 03:01:26", "content": "Thanks for the link @atamazian
", "votes": null, "replies": [] } ] }, { "id": 752814, "author_name": "msheriey", "author_url": "", "post_date": "02/21/2020 12:38:10", "content": "Following up on what @atamazian said, the thing is, having a random variant of the transformation applied for each epoch makes it look like every epoch was trained on a different set of images enabling the model to focus on the important features that makes up the label and ignore effects that might have been confusing to like translation, rotation, etc...
", "votes": null, "replies": [ { "id": 755684, "author_name": "gsdeepakkumar", "author_url": "", "post_date": "02/25/2020 03:01:09", "content": "Thank you @msheriey
", "votes": null, "replies": [] } ] }, { "id": 755431, "author_name": "mgorner", "author_url": "", "post_date": "02/24/2020 19:13:35", "content": "The augmented dataset will have more images only if you decide to generate more images. Applying transformation to existing images does not change the number. In the tf,data,Dataset API, you can use flat_map togenerate multiple images from a single input and merge them into the dataset. If you use map, then you are applying a transformation in place and the number of elements in the dataset is not affected.
", "votes": null, "replies": [ { "id": 755683, "author_name": "gsdeepakkumar", "author_url": "", "post_date": "02/25/2020 03:00:51", "content": "Thank you @mgornergoogle ..This is much helpful and I will try to use both the options and see how the score behaves ..
", "votes": null, "replies": [] } ] }, { "id": 755787, "author_name": "cdeotte", "author_url": "", "post_date": "02/25/2020 06:00:14", "content": "Yes. Using data augmentation creates more images. The training dataset has 12,753 images. If you augment them, then each epoch you have a new set of 12,753. Therefore over 20 epochs, you will train with 255,060 images! This is why data augmentation increases model accuracy.
", "votes": null, "replies": [ { "id": 756186, "author_name": "gsdeepakkumar", "author_url": "", "post_date": "02/25/2020 13:42:48", "content": "Thank you @cdeotte for the answer .. I have a followup question..
\n\n\n\n\nIf you augment them, then each epoch you have a new set of 12,753
\n
1.The augmentation is done for the training data and lets say for example I do random crop , increase brightness for rose flower . Therefore , in one epoch random crop is done and in next epoch the same rose flower is considered and increase brightness augmentation is applied ? Is this how the augmentation works ?
\n\nThanks in advance.
", "votes": null, "replies": [] }, { "id": 756230, "author_name": "gsdeepakkumar", "author_url": "", "post_date": "02/25/2020 14:31:05", "content": "Just happened to see your related answer for a similar question . I guess the answer for the above question can be inferred by understanding what is happening with
\n\n\n\n\ndataset.repeat()
\n
function . Pls correct me if I am wrong.
", "votes": null, "replies": [] }, { "id": 756360, "author_name": "cdeotte", "author_url": "", "post_date": "02/25/2020 16:48:35", "content": "The best way to know what TensorFlow is doing each epoch is to build your chain of augmentation and then view the images. I posted code in this discussion for viewing. Also when you are viewing your augmentation, set the random ranges very high like tf.image.adjust_contrast(x,LARGE_NUMBER), that way you can see what is happening, then lower the LARGE_NUMBER until it looks reasonable and use the more reasonable number for training.
Thank you for the tips @cdeotte ..I will play around with the number as mentioned and check its behaviour ..
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "752515": "Apologize if this question is too elementary . I would like to understand whether by doing data augmentation , we are actually increasing the number of images in the training dataset. For example,If I do random crop to an image , then will the augmented dataset have the actual image and randomly cropped image ? .What is the effect of augmentation on the number of training images ?.Thanks in advance for the help .", "752670": "All the original images are just transformed (i.e. rotation, zooming, etc.) every epoch and then used for training. Therefore, the number of images in each epoch is equal to the number of original images you have.\n[Via this answer on Stack Overflow](https://stackoverflow.com/questions/51748514/does-imagedatagenerator-add-more-images-to-my-dataset)", "752814": "Following up on what @atamazian said, the thing is, having a random variant of the transformation applied for each epoch makes it look like every epoch was trained on a different set of images enabling the model to focus on the important features that makes up the label and ignore effects that might have been confusing to like translation, rotation, etc...", "755431": "The augmented dataset will have more images only if you decide to generate more images. Applying transformation to existing images does not change the number. In the tf,data,Dataset API, you can use [flat_map](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#flat_map) togenerate multiple images from a single input and merge them into the dataset. If you use [map](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#map), then you are applying a transformation in place and the number of elements in the dataset is not affected.", "755683": "Thank you @mgornergoogle ..This is much helpful and I will try to use both the options and see how the score behaves ..", "755684": "Thank you @msheriey", "755685": "Thanks for the link @atamazian", "755787": "Yes. Using data augmentation creates more images. The training dataset has 12,753 images. If you augment them, then each epoch you have a new set of 12,753. Therefore over 20 epochs, you will train with 255,060 images! This is why data augmentation increases model accuracy.", "756186": "Thank you @cdeotte for the answer .. I have a followup question..\n\n> If you augment them, then each epoch you have a new set of 12,753\n\n1.The augmentation is done for the training data and lets say for example I do random crop , increase brightness for rose flower . Therefore , in one epoch random crop is done and in next epoch the same rose flower is considered and increase brightness augmentation is applied ? Is this how the augmentation works ?\n\nThanks in advance.", "756230": "Just happened to see your related answer for a [similar question ](https://www.kaggle.com/c/flower-classification-with-tpus/discussion/132127). I guess the answer for the above question can be inferred by understanding what is happening with \n> dataset.repeat() \n\nfunction . Pls correct me if I am wrong.", "756360": "The best way to know what TensorFlow is doing each epoch is to build your chain of augmentation and then view the images. I posted code in [this discussion][1] for viewing. Also when you are viewing your augmentation, set the random ranges very high like `tf.image.adjust_contrast(x,LARGE_NUMBER)`, that way you can see what is happening, then lower the `LARGE_NUMBER` until it looks reasonable and use the more reasonable number for training.\n\n[1]: https://www.kaggle.com/c/flower-classification-with-tpus/discussion/132127#755445", "756727": "Thank you for the tips @cdeotte ..I will play around with the number as mentioned and check its behaviour .." }, "source": "meta" }