![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2Ff1fc243fb7c4c2373902a9e26f653f22%2FScreen%20Shot%202020-02-24%20at%2011.19.15%20AM.png?generation=1582571973827383&alt=media\")
Hallo,\nI am rather new to tf and I have a question about augmentation.\nWhen I have a function like
\n\ndef get_training_dataset():\n dataset = load_dataset(TRAINING_FILENAMES, labeled=True)\n dataset = dataset.map(data_augment, num_parallel_calls=AUTO)\n dataset = dataset.repeat() # the training dataset must repeat for several epochs\n dataset = dataset.shuffle(2048)\n dataset = dataset.batch(BATCH_SIZE)\n dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n return dataset
\n\n...which is used in the model.fit() method, is the dataset calculated new for each epoch and does it make a difference if I call the map() function after repeat.\nI would like to get new augmentations for every epoch, otherwise the variance of the training data would be very poor.
\n\nregards
", "messages": [ { "id": "755053", "postDate": "02/24/2020 11:40:43", "content": "Hallo,\nI am rather new to tf and I have a question about augmentation.\nWhen I have a function like
\n\ndef get_training_dataset():\n dataset = load_dataset(TRAINING_FILENAMES, labeled=True)\n dataset = dataset.map(data_augment, num_parallel_calls=AUTO)\n dataset = dataset.repeat() # the training dataset must repeat for several epochs\n dataset = dataset.shuffle(2048)\n dataset = dataset.batch(BATCH_SIZE)\n dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n return dataset
\n\n...which is used in the model.fit() method, is the dataset calculated new for each epoch and does it make a difference if I call the map() function after repeat.\nI would like to get new augmentations for every epoch, otherwise the variance of the training data would be very poor.
\n\nregards
", "rawMarkdown": "Hallo,\nI am rather new to tf and I have a question about augmentation.\nWhen I have a function like\n\ndef get_training_dataset():\n dataset = load_dataset(TRAINING_FILENAMES, labeled=True)\n dataset = dataset.map(data_augment, num_parallel_calls=AUTO)\n dataset = dataset.repeat() # the training dataset must repeat for several epochs\n dataset = dataset.shuffle(2048)\n dataset = dataset.batch(BATCH_SIZE)\n dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n return dataset\n\n...which is used in the model.fit() method, is the dataset calculated new for each epoch and does it make a difference if I call the map() function after repeat.\nI would like to get new augmentations for every epoch, otherwise the variance of the training data would be very poor.\n\nregards", "votes": null }, { "id": "755369", "postDate": "02/24/2020 17:57:36", "content": "As much as I know, the data is augmented and feed to the model.fit() every epoch and so is will keep changing every epoch. The get_training_data() function returns the new random variant of augmented data for each epoch.
", "rawMarkdown": "As much as I know, the data is augmented and feed to the model.fit() every epoch and so is will keep changing every epoch. The get_training_data() function returns the new random variant of augmented data for each epoch.", "votes": null }, { "id": "755421", "postDate": "02/24/2020 19:02:07", "content": "This is a great question. The best way to answer this it to make a small dataset of 10 images, and display what is happening. Because the TensorFlow documentation may say one thing but what is actually happening may be different.
\n\nI think you may be right, that dataset = dataset.repeat() may need to be before dataset = dataset.map(dataaugment). Because the repeat command turns the dataset from size 12,000 to size infinite. We want to do that before augmentation. As opposed to taking and repeating 12,000 augmented images.
EDIT: The order doesn't matter. I posted a test above.
", "rawMarkdown": "This is a great question. The best way to answer this it to make a small dataset of 10 images, and display what is happening. Because the TensorFlow documentation may say one thing but what is actually happening may be different.\n\nI think you may be right, that `dataset = dataset.repeat()` may need to be before `dataset = dataset.map(dataaugment)`. Because the repeat command turns the dataset from size 12,000 to size infinite. We want to do that before augmentation. As opposed to taking and repeating 12,000 augmented images.\n\nEDIT: The order doesn't matter. I posted a test above.", "votes": null }, { "id": "755436", "postDate": "02/24/2020 19:20:10", "content": "CORRECTION: The TF team confirmed that map and repeat are commutative. Repeat repeats the dataset, including whatever transformations are applied to it, wherever it is placed in the dataset definition.
[original (wrong) comment]
\n\nYou are correct, as defined, the dataset is augmented then repeated. Entropy will be much better if you repeat and then augment.
\n\nThe tf.data.Dataset API does not create a set of data as such, it creates a set of transformations to be applied to data. These transformations are then applied as you train. For training data, it is recommended to use an infinitely repeating dataset that will cover all epochs of training. That's what the getting started notebook does. But there are indeed two ways of getting an infinitely repeating dataset: augmenting the dataset, then repeating it and the other way around.
", "rawMarkdown": "CORRECTION: The TF team confirmed that `map` and `repeat` are commutative. Repeat repeats the dataset, including whatever transformations are applied to it, wherever it is placed in the dataset definition.\n\n------\n[original (wrong) comment]\n\nYou are correct, as defined, the dataset is augmented then repeated. Entropy will be much better if you repeat and then augment.\n\nThe tf.data.Dataset API does not create a set of data as such, it creates a set of transformations to be applied to data. These transformations are then applied as you train. For training data, it is recommended to use an infinitely repeating dataset that will cover all epochs of training. That's what the [getting started notebook](https://www.kaggle.com/mgornergoogle/getting-started-with-100-flowers-on-tpu/#Datasets) does. But there are indeed two ways of getting an infinitely repeating dataset: augmenting the dataset, then repeating it and the other way around.", "votes": null }, { "id": "755437", "postDate": "02/24/2020 19:22:14", "content": "CORRECTION: The TF team confirmed that map and repeat are commutative. Repeat repeats the dataset, including whatever transformations are applied to it, wherever it is placed in the dataset definition.
\n\n[original (wrong) comment]
\n\nNot in the getting started notebook. Since the dataset is first augmented, then repeated indefinitely to cover all epochs, the same initially augmented images are being looped through at each epoch. Not ideal indeed.
", "rawMarkdown": "CORRECTION: The TF team confirmed that map and repeat are commutative. Repeat repeats the dataset, including whatever transformations are applied to it, wherever it is placed in the dataset definition.\n\n---\n[original (wrong) comment]\n\nNot in the [getting started notebook](https://www.kaggle.com/mgornergoogle/getting-started-with-100-flowers-on-tpu/#Datasets). Since the dataset is first augmented, then repeated indefinitely to cover all epochs, the same initially augmented images are being looped through at each epoch. Not ideal indeed.", "votes": null }, { "id": "755445", "postDate": "02/24/2020 19:26:50", "content": "I just ran some tests. We get different augmentation each epoch. It doesn't matter where we make the call map(data_augment) in the chain. Upon reflection, this makes sense. TensorFlow doesn't compute anything until when it is needed. Augmentation includes a random number tensor. So every time augmentation is asked for, TensorFlow produces a random number Tensor first.
all = get_training_dataset(load_dataset(TRAINING_FILENAMES)).unbatch().batch(4)\nbatch = next(iter(all))\nfour_images = tf.data.Dataset.from_tensors( batch ).unbatch().repeat()\nrow = 2; col = 4;\nplt.figure(figsize=(15,int(15*row/col))\nfor j,(img,label) in enumerate(four_images):\n plt.subplot(row,col,j+1)\n plt.axis('off'); plt.imshow(img)\n if j>=7: break\nplt.show()\n
Look closely and you will see that some images are flipped left to right.
\n\nfour_images = tf.data.Dataset.from_tensors( batch ).unbatch().repeat().map(data_augment)\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2F83d6c6badc4c6c6f5749fd91b7d54b25%2FScreen%20Shot%202020-02-24%20at%2011.22.32%20AM.png?generation=1582572192907792&alt=media\")
Look closely and you will see that some images are flipped left to right.
\n\nfour_images = tf.data.Dataset.from_tensors( batch ).unbatch().map(data_augment).repeat()\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2F3a99ea476c156a9e5fbda2a73ea9f6ff%2FScreen%20Shot%202020-02-24%20at%2011.24.01%20AM.png?generation=1582572256025671&alt=media\")
Hey Martin, I just did a test. It doesn't appear to matter whether you augment or repeat first.
", "rawMarkdown": "Hey Martin, I just did a test. It doesn't appear to matter whether you augment or repeat first.", "votes": null }, { "id": "755467", "postDate": "02/24/2020 19:51:38", "content": "Thanks for the input guys.\nHmm. I would say entropy matters :)\nAnyway - can one tell from 2 epochs, if there is a difference?
\n\nI guess one can :)
", "rawMarkdown": "Thanks for the input guys.\nHmm. I would say entropy matters :)\nAnyway - can one tell from 2 epochs, if there is a difference?\n\nI guess one can :)", "votes": null }, { "id": "755472", "postDate": "02/24/2020 19:55:47", "content": "Yes, 2 epochs is enough. If TensorFlow augmented all the images once and then repeated them without augmenting the new epoch, we would see that in my experiment below using only 2 epochs. (But my experiments show that each epoch is getting new images regardless of the location of dataaugment in chain of calls).
Indeed. I tried as well in this Colab notebook.\nYou seem to be right, the dataset is augmented correctly, with fresh data augmentation at each epoch, no matter the order of augment and repeat.
", "rawMarkdown": "Indeed. I tried as well in this [Colab notebook](https://colab.research.google.com/drive/1dw2M866LyaiKWTDamgrGPBsYIEMWUW_U).\nYou seem to be right, the dataset is augmented correctly, with fresh data augmentation at each epoch, no matter the order of augment and repeat.", "votes": null }, { "id": "755612", "postDate": "02/25/2020 00:09:56", "content": "Let me ask the TF team wether this is normal.
", "rawMarkdown": "Let me ask the TF team wether this is normal.", "votes": null }, { "id": "756379", "postDate": "02/25/2020 17:08:14", "content": "The TF team confirmed that map and repeat are indeed commutative. Repeat repeats the dataset, including whatever transformations are applied to it, wherever it is placed in the dataset definition.
Great you could find out - thx
", "rawMarkdown": "Great you could find out - thx", "votes": null } ], "comments": [ { "id": 755369, "author_name": "shank885", "author_url": "", "post_date": "02/24/2020 17:57:36", "content": "As much as I know, the data is augmented and feed to the model.fit() every epoch and so is will keep changing every epoch. The get_training_data() function returns the new random variant of augmented data for each epoch.
", "votes": null, "replies": [ { "id": 755437, "author_name": "mgorner", "author_url": "", "post_date": "02/24/2020 19:22:14", "content": "CORRECTION: The TF team confirmed that map and repeat are commutative. Repeat repeats the dataset, including whatever transformations are applied to it, wherever it is placed in the dataset definition.
\n\n[original (wrong) comment]
\n\nNot in the getting started notebook. Since the dataset is first augmented, then repeated indefinitely to cover all epochs, the same initially augmented images are being looped through at each epoch. Not ideal indeed.
", "votes": null, "replies": [] } ] }, { "id": 755421, "author_name": "cdeotte", "author_url": "", "post_date": "02/24/2020 19:02:07", "content": "This is a great question. The best way to answer this it to make a small dataset of 10 images, and display what is happening. Because the TensorFlow documentation may say one thing but what is actually happening may be different.
\n\nI think you may be right, that dataset = dataset.repeat() may need to be before dataset = dataset.map(dataaugment). Because the repeat command turns the dataset from size 12,000 to size infinite. We want to do that before augmentation. As opposed to taking and repeating 12,000 augmented images.
EDIT: The order doesn't matter. I posted a test above.
", "votes": null, "replies": [] }, { "id": 755436, "author_name": "mgorner", "author_url": "", "post_date": "02/24/2020 19:20:10", "content": "CORRECTION: The TF team confirmed that map and repeat are commutative. Repeat repeats the dataset, including whatever transformations are applied to it, wherever it is placed in the dataset definition.
[original (wrong) comment]
\n\nYou are correct, as defined, the dataset is augmented then repeated. Entropy will be much better if you repeat and then augment.
\n\nThe tf.data.Dataset API does not create a set of data as such, it creates a set of transformations to be applied to data. These transformations are then applied as you train. For training data, it is recommended to use an infinitely repeating dataset that will cover all epochs of training. That's what the getting started notebook does. But there are indeed two ways of getting an infinitely repeating dataset: augmenting the dataset, then repeating it and the other way around.
", "votes": null, "replies": [ { "id": 755448, "author_name": "cdeotte", "author_url": "", "post_date": "02/24/2020 19:27:56", "content": "Hey Martin, I just did a test. It doesn't appear to matter whether you augment or repeat first.
", "votes": null, "replies": [] }, { "id": 755610, "author_name": "mgorner", "author_url": "", "post_date": "02/25/2020 00:09:35", "content": "Indeed. I tried as well in this Colab notebook.\nYou seem to be right, the dataset is augmented correctly, with fresh data augmentation at each epoch, no matter the order of augment and repeat.
", "votes": null, "replies": [] }, { "id": 755612, "author_name": "mgorner", "author_url": "", "post_date": "02/25/2020 00:09:56", "content": "Let me ask the TF team wether this is normal.
", "votes": null, "replies": [] }, { "id": 756379, "author_name": "mgorner", "author_url": "", "post_date": "02/25/2020 17:08:14", "content": "The TF team confirmed that map and repeat are indeed commutative. Repeat repeats the dataset, including whatever transformations are applied to it, wherever it is placed in the dataset definition.
Great you could find out - thx
", "votes": null, "replies": [] } ] }, { "id": 755445, "author_name": "cdeotte", "author_url": "", "post_date": "02/24/2020 19:26:50", "content": "I just ran some tests. We get different augmentation each epoch. It doesn't matter where we make the call map(data_augment) in the chain. Upon reflection, this makes sense. TensorFlow doesn't compute anything until when it is needed. Augmentation includes a random number tensor. So every time augmentation is asked for, TensorFlow produces a random number Tensor first.
all = get_training_dataset(load_dataset(TRAINING_FILENAMES)).unbatch().batch(4)\nbatch = next(iter(all))\nfour_images = tf.data.Dataset.from_tensors( batch ).unbatch().repeat()\nrow = 2; col = 4;\nplt.figure(figsize=(15,int(15*row/col))\nfor j,(img,label) in enumerate(four_images):\n plt.subplot(row,col,j+1)\n plt.axis('off'); plt.imshow(img)\n if j>=7: break\nplt.show()\n
Look closely and you will see that some images are flipped left to right.
\n\nfour_images = tf.data.Dataset.from_tensors( batch ).unbatch().repeat().map(data_augment)\n
Look closely and you will see that some images are flipped left to right.
\n\nfour_images = tf.data.Dataset.from_tensors( batch ).unbatch().map(data_augment).repeat()\n
Thanks for the input guys.\nHmm. I would say entropy matters :)\nAnyway - can one tell from 2 epochs, if there is a difference?
\n\nI guess one can :)
", "votes": null, "replies": [ { "id": 755472, "author_name": "cdeotte", "author_url": "", "post_date": "02/24/2020 19:55:47", "content": "Yes, 2 epochs is enough. If TensorFlow augmented all the images once and then repeated them without augmenting the new epoch, we would see that in my experiment below using only 2 epochs. (But my experiments show that each epoch is getting new images regardless of the location of dataaugment in chain of calls).