{
  "id": 169102,
  "title": "ImageDataGenerator with TPU",
  "url": "/competitions/siim-isic-melanoma-classification/discussion/169102",
  "author_name": "LGreig",
  "post_date": "2020-07-23T00:07:03.607000",
  "votes": 1,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Hi,</p>\n\n<p>I'm relatively new to neural networks and have been trying to use keras to do image augmentation. I've used ImageDataGenerator to do this and it works well with the CPU. However when using the TPU, I find that it keeps throwing up an error. Using model.fit works fine but model.fit_generator wont work (<a href=\"https://github.com/tensorflow/tensorflow/issues/34346\">https://github.com/tensorflow/tensorflow/issues/34346</a>) . In order to use keras with a TPU, does image augmentation have to be done manually with custom code instead or are there better ways of doing this?</p>",
  "messages": [
    {
      "id": 940421,
      "postDate": "2020-07-23T00:07:03.607Z",
      "content": "<p>Hi,</p>\n\n<p>I'm relatively new to neural networks and have been trying to use keras to do image augmentation. I've used ImageDataGenerator to do this and it works well with the CPU. However when using the TPU, I find that it keeps throwing up an error. Using model.fit works fine but model.fit_generator wont work (<a href=\"https://github.com/tensorflow/tensorflow/issues/34346\">https://github.com/tensorflow/tensorflow/issues/34346</a>) . In order to use keras with a TPU, does image augmentation have to be done manually with custom code instead or are there better ways of doing this?</p>",
      "rawMarkdown": "Hi,\n\nI'm relatively new to neural networks and have been trying to use keras to do image augmentation. I've used ImageDataGenerator to do this and it works well with the CPU. However when using the TPU, I find that it keeps throwing up an error. Using model.fit works fine but model.fit_generator wont work (https://github.com/tensorflow/tensorflow/issues/34346) . In order to use keras with a TPU, does image augmentation have to be done manually with custom code instead or are there better ways of doing this?",
      "votes": 1
    },
    {
      "id": 950538,
      "postDate": "2020-07-29T13:05:32.213Z",
      "content": "<p>Yes you are write our general Keras ImageDataGenerators won't work on TPU directly. \nYou need to call them inside the stratergy.scope() to make them able to be used on TPU.</p>\n\n<p>Whereas , other than you doubt. I would recommend you to use tfrecords with there GCS paths on TPu to get the exact  performance what a TPU can provide. Using individual images would just limit it to use to CPUs and not actual TPU.</p>",
      "rawMarkdown": "Yes you are write our general Keras ImageDataGenerators won't work on TPU directly. \nYou need to call them inside the stratergy.scope() to make them able to be used on TPU.\n\nWhereas , other than you doubt. I would recommend you to use tfrecords with there GCS paths on TPu to get the exact  performance what a TPU can provide. Using individual images would just limit it to use to CPUs and not actual TPU.",
      "votes": 2,
      "replies": [
        {
          "id": 953612,
          "postDate": "2020-07-31T23:14:36.310Z",
          "content": "<p>Thanks; this works!</p>",
          "rawMarkdown": "Thanks; this works!"
        }
      ]
    },
    {
      "id": 941559,
      "postDate": "2020-07-23T09:47:52.030Z",
      "rawMarkdown": "",
      "votes": 1,
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 950538,
      "author_name": "Prashant Arora",
      "author_url": "",
      "post_date": "2020-07-29T13:05:32.213000",
      "content": "<p>Yes you are write our general Keras ImageDataGenerators won't work on TPU directly. \nYou need to call them inside the stratergy.scope() to make them able to be used on TPU.</p>\n\n<p>Whereas , other than you doubt. I would recommend you to use tfrecords with there GCS paths on TPu to get the exact  performance what a TPU can provide. Using individual images would just limit it to use to CPUs and not actual TPU.</p>",
      "votes": 2,
      "replies": [
        {
          "id": 953612,
          "author_name": "LGreig",
          "author_url": "",
          "post_date": "2020-07-31T23:14:36.310000",
          "content": "<p>Thanks; this works!</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 941559,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-07-23T09:47:52.030000",
      "content": "",
      "votes": 1,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "940421": "Hi,\n\nI'm relatively new to neural networks and have been trying to use keras to do image augmentation. I've used ImageDataGenerator to do this and it works well with the CPU. However when using the TPU, I find that it keeps throwing up an error. Using model.fit works fine but model.fit_generator wont work (https://github.com/tensorflow/tensorflow/issues/34346) . In order to use keras with a TPU, does image augmentation have to be done manually with custom code instead or are there better ways of doing this?",
    "950538": "Yes you are write our general Keras ImageDataGenerators won't work on TPU directly. \nYou need to call them inside the stratergy.scope() to make them able to be used on TPU.\n\nWhereas , other than you doubt. I would recommend you to use tfrecords with there GCS paths on TPu to get the exact  performance what a TPU can provide. Using individual images would just limit it to use to CPUs and not actual TPU.",
    "941559": ""
  }
}