If I want to perform data augmentation using an external library like albumentations. Where is the best place to it ? In the data augment function from the stater code ? Can I make a image data generator using keras ? Which one would work and which is the efficient one ?
", "messages": [ { "id": 755465, "postDate": "2020-02-24T19:50:45.223Z", "content": "Unfortunately most external libraries will not work. You must perform your augmentation on the GPU/TPU and not the CPU.
\n\nI posted a starter notebook showing how to do rotation, shear, zoom, and shift augmentation using GPU/TPU and TensorFlow here. And a discussion here.
", "rawMarkdown": "Unfortunately most external libraries will not work. You must perform your augmentation on the GPU/TPU and not the CPU.\n\nI posted a starter notebook showing how to do rotation, shear, zoom, and shift augmentation using GPU/TPU and TensorFlow [here][2]. And a discussion [here][1].\n\n[1]: https://www.kaggle.com/c/flower-classification-with-tpus/discussion/132191\n[2]: https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96", "votes": 3 }, { "id": 755029, "postDate": "2020-02-24T11:11:27.487Z", "content": "In my understanding, data will remain in tensor when we are using .tfrec format. So maybe we need to perform the augmentation in tensor format, which means we need to implement in tensorflow style. I had implemented some data augmentation methods in this public kernel. Hope it can help :)
\n\nhttps://www.kaggle.com/xiejialun/customize-data-augmentation-with-tensorflow
", "rawMarkdown": "In my understanding, data will remain in tensor when we are using .tfrec format. So maybe we need to perform the augmentation in tensor format, which means we need to implement in tensorflow style. I had implemented some data augmentation methods in this public kernel. Hope it can help :)\n\nhttps://www.kaggle.com/xiejialun/customize-data-augmentation-with-tensorflow ", "votes": 1, "replies": [ { "id": 755127, "postDate": "2020-02-24T13:26:19.473Z", "content": "Thanks for four answer. This means that I need to implement some augmentation techniques that are not implemented in tensorflow.
", "rawMarkdown": "Thanks for four answer. This means that I need to implement some augmentation techniques that are not implemented in tensorflow. " }, { "id": 755132, "postDate": "2020-02-24T13:28:44.487Z", "content": "Yes, you are right. This is why I published this kernel. The augmentation methods in the kernel are different from the methods implemented in tensorflow.
", "rawMarkdown": "Yes, you are right. This is why I published this kernel. The augmentation methods in the kernel are different from the methods implemented in tensorflow." }, { "id": 755134, "postDate": "2020-02-24T13:32:14.817Z", "content": "But normally, you can use an excellent library called tensorflow_addons. This library implement the augmentation methods in pure tensorflow, which means it can directly apply on tensor. But I think there are some problems of it when using it under TPU environment. But GPU/CPU is okay though.
", "rawMarkdown": "But normally, you can use an excellent library called tensorflow_addons. This library implement the augmentation methods in pure tensorflow, which means it can directly apply on tensor. But I think there are some problems of it when using it under TPU environment. But GPU/CPU is okay though." }, { "id": 755137, "postDate": "2020-02-24T13:38:49.577Z", "content": "I checked your kernel, it shows some great examples. I also know about the tensorflow addons problems. Thanks for your help.
", "rawMarkdown": "I checked your kernel, it shows some great examples. I also know about the tensorflow addons problems. Thanks for your help.", "votes": 1 } ] }, { "id": 754632, "postDate": "2020-02-23T21:18:52.073Z", "content": "If I want to perform data augmentation using an external library like albumentations. Where is the best place to it ? In the data augment function from the stater code ? Can I make a image data generator using keras ? Which one would work and which is the efficient one ?
", "rawMarkdown": "If I want to perform data augmentation using an external library like albumentations. Where is the best place to it ? In the data augment function from the stater code ? Can I make a image data generator using keras ? Which one would work and which is the efficient one ?", "votes": 1 }, { "id": 754805, "postDate": "2020-02-24T04:41:42.823Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ], "comments": [ { "id": 755465, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-02-24T19:50:45.223000", "content": "Unfortunately most external libraries will not work. You must perform your augmentation on the GPU/TPU and not the CPU.
\n\nI posted a starter notebook showing how to do rotation, shear, zoom, and shift augmentation using GPU/TPU and TensorFlow here. And a discussion here.
", "votes": 3, "replies": [] }, { "id": 755029, "author_name": "Tsai29", "author_url": "", "post_date": "2020-02-24T11:11:27.487000", "content": "In my understanding, data will remain in tensor when we are using .tfrec format. So maybe we need to perform the augmentation in tensor format, which means we need to implement in tensorflow style. I had implemented some data augmentation methods in this public kernel. Hope it can help :)
\n\nhttps://www.kaggle.com/xiejialun/customize-data-augmentation-with-tensorflow
", "votes": 1, "replies": [ { "id": 755127, "author_name": "Ibrahim Sherif", "author_url": "", "post_date": "2020-02-24T13:26:19.473000", "content": "Thanks for four answer. This means that I need to implement some augmentation techniques that are not implemented in tensorflow.
", "votes": 0, "replies": [] }, { "id": 755132, "author_name": "Tsai29", "author_url": "", "post_date": "2020-02-24T13:28:44.487000", "content": "Yes, you are right. This is why I published this kernel. The augmentation methods in the kernel are different from the methods implemented in tensorflow.
", "votes": 0, "replies": [] }, { "id": 755134, "author_name": "Tsai29", "author_url": "", "post_date": "2020-02-24T13:32:14.817000", "content": "But normally, you can use an excellent library called tensorflow_addons. This library implement the augmentation methods in pure tensorflow, which means it can directly apply on tensor. But I think there are some problems of it when using it under TPU environment. But GPU/CPU is okay though.
", "votes": 0, "replies": [] }, { "id": 755137, "author_name": "Ibrahim Sherif", "author_url": "", "post_date": "2020-02-24T13:38:49.577000", "content": "I checked your kernel, it shows some great examples. I also know about the tensorflow addons problems. Thanks for your help.
", "votes": 1, "replies": [] } ] }, { "id": 754805, "author_name": "", "author_url": "", "post_date": "2020-02-24T04:41:42.823000", "content": "", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "755465": "Unfortunately most external libraries will not work. You must perform your augmentation on the GPU/TPU and not the CPU.\n\nI posted a starter notebook showing how to do rotation, shear, zoom, and shift augmentation using GPU/TPU and TensorFlow [here][2]. And a discussion [here][1].\n\n[1]: https://www.kaggle.com/c/flower-classification-with-tpus/discussion/132191\n[2]: https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96", "755029": "In my understanding, data will remain in tensor when we are using .tfrec format. So maybe we need to perform the augmentation in tensor format, which means we need to implement in tensorflow style. I had implemented some data augmentation methods in this public kernel. Hope it can help :)\n\nhttps://www.kaggle.com/xiejialun/customize-data-augmentation-with-tensorflow ", "754632": "If I want to perform data augmentation using an external library like albumentations. Where is the best place to it ? In the data augment function from the stater code ? Can I make a image data generator using keras ? Which one would work and which is the efficient one ?", "754805": "" } }