{
  "id": 155367,
  "title": "tf.dataset tpu  pipeline augmentation other than tf.image",
  "url": "/competitions/siim-isic-melanoma-classification/discussion/155367",
  "author_name": "",
  "post_date": "2020-06-01T12:15:08.135753200Z",
  "votes": 2,
  "comment_count": 6,
  "views": 0,
  "content": "<p>Is there a way to apply augmentation other than tf.image in tf.dataset tpu pipelinesuch as (cv2 ,albumentation)</p>",
  "messages": [
    {
      "id": "869978",
      "postDate": "06/01/2020 12:15:08",
      "content": "<p>Is there a way to apply augmentation other than tf.image in tf.dataset tpu pipelinesuch as (cv2 ,albumentation)</p>",
      "rawMarkdown": "Is there a way to apply augmentation other than tf.image in tf.dataset tpu pipelinesuch as (cv2 ,albumentation)",
      "votes": null
    },
    {
      "id": "870297",
      "postDate": "06/01/2020 16:06:16",
      "content": "<p>Yes , If you use the JPEG data . </p>",
      "rawMarkdown": "Yes , If you use the JPEG data .",
      "votes": null
    },
    {
      "id": "870353",
      "postDate": "06/01/2020 16:34:53",
      "content": "<p>can you give some reffrences it will be very helpfull</p>",
      "rawMarkdown": "can you give some reffrences it will be very helpfull",
      "votes": null
    },
    {
      "id": "870588",
      "postDate": "06/01/2020 18:52:17",
      "content": "<p>Hey I've actually did some looking into this and apparently when creating a tf.dataset pipeline you have to use only tf operations one solution to this is that you have to wrap up your transformation in tf.py_func and then use it (src =&gt; <a href=\"https://github.com/tensorflow/tensorflow/issues/30112\">https://github.com/tensorflow/tensorflow/issues/30112</a>) I guess this is one downside to dataloaders in tf versus pytorch.</p>",
      "rawMarkdown": "Hey I've actually did some looking into this and apparently when creating a tf.dataset pipeline you have to use only tf operations one solution to this is that you have to wrap up your transformation in tf.py_func and then use it (src =&gt; https://github.com/tensorflow/tensorflow/issues/30112) I guess this is one downside to dataloaders in tf versus pytorch.",
      "votes": null
    },
    {
      "id": "871013",
      "postDate": "06/02/2020 05:04:57",
      "content": "<p>If you use <code>tf.data.dataset</code> then your augmentation must be written in TensorFlow language. There is much discussion and examples in Flower comp <a href=\"https://www.kaggle.com/c/flower-classification-with-tpus/discussion\">here</a>. For example, <a href=\"https://www.kaggle.com/c/flower-classification-with-tpus/discussion/132191\">How To Rotate</a>, and <a href=\"https://www.kaggle.com/c/flower-classification-with-tpus/discussion/132935\">How To CutMix</a>.</p>\n\n<p>If you want to use cv2 and albumentations, then you must create your own dataloader with Keras.Sequence, PyTorch data.DataLoader or similar. </p>",
      "rawMarkdown": "If you use `tf.data.dataset` then your augmentation must be written in TensorFlow language. There is much discussion and examples in Flower comp [here][1]. For example, [How To Rotate][2], and [How To CutMix][3].\n\nIf you want to use cv2 and albumentations, then you must create your own dataloader with Keras.Sequence, PyTorch data.DataLoader or similar. \n\n[1]: https://www.kaggle.com/c/flower-classification-with-tpus/discussion\n[2]: https://www.kaggle.com/c/flower-classification-with-tpus/discussion/132191\n[3]: https://www.kaggle.com/c/flower-classification-with-tpus/discussion/132935",
      "votes": null
    },
    {
      "id": "871236",
      "postDate": "06/02/2020 08:04:18",
      "content": "<p>I was able to \"excise\" a few functions that I needed from this <a href=\"https://github.com/tensorflow/tpu/blob/master/models/official/efficientnet/autoaugment.py\">set</a> of augmentations. It's written in Tensorflow V1 I think but if you add ...</p>\n\n<p><code>\nimport tensorflow as tf\ntf.to_float = lambda x: tf.cast(x, tf.float32)\n</code>\nIt will run, or at least the bits I used ran.</p>",
      "rawMarkdown": "I was able to \"excise\" a few functions that I needed from this [set](https://github.com/tensorflow/tpu/blob/master/models/official/efficientnet/autoaugment.py) of augmentations. It's written in Tensorflow V1 I think but if you add ...\n\n```\nimport tensorflow as tf\ntf.to_float = lambda x: tf.cast(x, tf.float32)\n```\nIt will run, or at least the bits I used ran.",
      "votes": null
    },
    {
      "id": "871362",
      "postDate": "06/02/2020 09:43:15",
      "content": "<p>thanks for share it</p>",
      "rawMarkdown": "thanks for share it",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 870297,
      "author_name": "prudhvi9999",
      "author_url": "",
      "post_date": "06/01/2020 16:06:16",
      "content": "<p>Yes , If you use the JPEG data . </p>",
      "votes": null,
      "replies": [
        {
          "id": 870353,
          "author_name": "ajax0564",
          "author_url": "",
          "post_date": "06/01/2020 16:34:53",
          "content": "<p>can you give some reffrences it will be very helpfull</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 870588,
      "author_name": "aziz69",
      "author_url": "",
      "post_date": "06/01/2020 18:52:17",
      "content": "<p>Hey I've actually did some looking into this and apparently when creating a tf.dataset pipeline you have to use only tf operations one solution to this is that you have to wrap up your transformation in tf.py_func and then use it (src =&gt; <a href=\"https://github.com/tensorflow/tensorflow/issues/30112\">https://github.com/tensorflow/tensorflow/issues/30112</a>) I guess this is one downside to dataloaders in tf versus pytorch.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 871013,
      "author_name": "cdeotte",
      "author_url": "",
      "post_date": "06/02/2020 05:04:57",
      "content": "<p>If you use <code>tf.data.dataset</code> then your augmentation must be written in TensorFlow language. There is much discussion and examples in Flower comp <a href=\"https://www.kaggle.com/c/flower-classification-with-tpus/discussion\">here</a>. For example, <a href=\"https://www.kaggle.com/c/flower-classification-with-tpus/discussion/132191\">How To Rotate</a>, and <a href=\"https://www.kaggle.com/c/flower-classification-with-tpus/discussion/132935\">How To CutMix</a>.</p>\n\n<p>If you want to use cv2 and albumentations, then you must create your own dataloader with Keras.Sequence, PyTorch data.DataLoader or similar. </p>",
      "votes": null,
      "replies": [
        {
          "id": 871362,
          "author_name": "ajax0564",
          "author_url": "",
          "post_date": "06/02/2020 09:43:15",
          "content": "<p>thanks for share it</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 871236,
      "author_name": "mutantspore",
      "author_url": "",
      "post_date": "06/02/2020 08:04:18",
      "content": "<p>I was able to \"excise\" a few functions that I needed from this <a href=\"https://github.com/tensorflow/tpu/blob/master/models/official/efficientnet/autoaugment.py\">set</a> of augmentations. It's written in Tensorflow V1 I think but if you add ...</p>\n\n<p><code>\nimport tensorflow as tf\ntf.to_float = lambda x: tf.cast(x, tf.float32)\n</code>\nIt will run, or at least the bits I used ran.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "869978": "Is there a way to apply augmentation other than tf.image in tf.dataset tpu pipelinesuch as (cv2 ,albumentation)",
    "870297": "Yes , If you use the JPEG data .",
    "870353": "can you give some reffrences it will be very helpfull",
    "870588": "Hey I've actually did some looking into this and apparently when creating a tf.dataset pipeline you have to use only tf operations one solution to this is that you have to wrap up your transformation in tf.py_func and then use it (src =&gt; https://github.com/tensorflow/tensorflow/issues/30112) I guess this is one downside to dataloaders in tf versus pytorch.",
    "871013": "If you use `tf.data.dataset` then your augmentation must be written in TensorFlow language. There is much discussion and examples in Flower comp [here][1]. For example, [How To Rotate][2], and [How To CutMix][3].\n\nIf you want to use cv2 and albumentations, then you must create your own dataloader with Keras.Sequence, PyTorch data.DataLoader or similar. \n\n[1]: https://www.kaggle.com/c/flower-classification-with-tpus/discussion\n[2]: https://www.kaggle.com/c/flower-classification-with-tpus/discussion/132191\n[3]: https://www.kaggle.com/c/flower-classification-with-tpus/discussion/132935",
    "871236": "I was able to \"excise\" a few functions that I needed from this [set](https://github.com/tensorflow/tpu/blob/master/models/official/efficientnet/autoaugment.py) of augmentations. It's written in Tensorflow V1 I think but if you add ...\n\n```\nimport tensorflow as tf\ntf.to_float = lambda x: tf.cast(x, tf.float32)\n```\nIt will run, or at least the bits I used ran.",
    "871362": "thanks for share it"
  },
  "source": "meta"
}