Hi, I saw many people have normalized their images with global mean and std. deviation, but I couldn't find a way to use them in my tf.data pipeline. Is there a way to easily normalize the dataset while using tf.data pipeline.
\nPS: I tried using tf.keras.layers.experimental.preprocessing.Normalization() layer in my pipeline but its adapt method always threw error
", "messages": [ { "id": "1104170", "postDate": "12/06/2020 17:10:24", "content": "Hi, I saw many people have normalized their images with global mean and std. deviation, but I couldn't find a way to use them in my tf.data pipeline. Is there a way to easily normalize the dataset while using tf.data pipeline.
\nPS: I tried using tf.keras.layers.experimental.preprocessing.Normalization() layer in my pipeline but its adapt method always threw error
", "rawMarkdown": "Hi, I saw many people have normalized their images with global mean and std. deviation, but I couldn't find a way to use them in my tf.data pipeline. Is there a way to easily normalize the dataset while using tf.data pipeline.\n\nPS: I tried using tf.keras.layers.experimental.preprocessing.Normalization() layer in my pipeline but its adapt method always threw error", "votes": null }, { "id": "1107392", "postDate": "12/09/2020 17:00:12", "content": "Although I didn't join this competition, it seems they provide the raw images (e.g. JPG) as well as TFRecords for processing. In the case of working with raw images, can you use the ImageDataGenerator as a means to normalize your images when you feed into your model.fit? You could also make use of the preprocessing_function parameter to pass a function on how you want to normalise the data.
", "rawMarkdown": "Although I didn't join this competition, it seems they provide the raw images (e.g. JPG) as well as TFRecords for processing. In the case of working with raw images, can you use the [ImageDataGenerator](https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/image/ImageDataGenerator?hl=en) as a means to normalize your images when you feed into your model.fit? You could also make use of the **preprocessing_function** parameter to pass a function on how you want to normalise the data.", "votes": null }, { "id": "1107440", "postDate": "12/09/2020 17:33:12", "content": "Thanks for the reply, but unfortunately I am using TFRecords and looking for a way to normalize the dataset while using tf.data pipeline
", "rawMarkdown": "Thanks for the reply, but unfortunately I am using TFRecords and looking for a way to normalize the dataset while using tf.data pipeline", "votes": null }, { "id": "1107485", "postDate": "12/09/2020 18:19:23", "content": "Can you use the map method on your tf.data and feed in a lambda function for your normalization?
\ndataset = tf.data.TFRecordDataset([\"image1.tfrecords\", \"image2.tfrecords\"])\ndataset = dataset.map(lambda x: x/255)\n
Yes, I am aware of this, but what I have is the channel wise mean and standard deviations. So is there any way to normalize (or Standardize would be the right term?) using them?
", "rawMarkdown": "Yes, I am aware of this, but what I have is the channel wise mean and standard deviations. So is there any way to normalize (or Standardize would be the right term?) using them?", "votes": null } ], "comments": [ { "id": 1107392, "author_name": "alankmwong", "author_url": "", "post_date": "12/09/2020 17:00:12", "content": "Although I didn't join this competition, it seems they provide the raw images (e.g. JPG) as well as TFRecords for processing. In the case of working with raw images, can you use the ImageDataGenerator as a means to normalize your images when you feed into your model.fit? You could also make use of the preprocessing_function parameter to pass a function on how you want to normalise the data.
", "votes": null, "replies": [ { "id": 1107440, "author_name": "ashish2001", "author_url": "", "post_date": "12/09/2020 17:33:12", "content": "Thanks for the reply, but unfortunately I am using TFRecords and looking for a way to normalize the dataset while using tf.data pipeline
", "votes": null, "replies": [] }, { "id": 1107485, "author_name": "alankmwong", "author_url": "", "post_date": "12/09/2020 18:19:23", "content": "Can you use the map method on your tf.data and feed in a lambda function for your normalization?
\ndataset = tf.data.TFRecordDataset([\"image1.tfrecords\", \"image2.tfrecords\"])\ndataset = dataset.map(lambda x: x/255)\n
Yes, I am aware of this, but what I have is the channel wise mean and standard deviations. So is there any way to normalize (or Standardize would be the right term?) using them?
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1104170": "Hi, I saw many people have normalized their images with global mean and std. deviation, but I couldn't find a way to use them in my tf.data pipeline. Is there a way to easily normalize the dataset while using tf.data pipeline.\n\nPS: I tried using tf.keras.layers.experimental.preprocessing.Normalization() layer in my pipeline but its adapt method always threw error", "1107392": "Although I didn't join this competition, it seems they provide the raw images (e.g. JPG) as well as TFRecords for processing. In the case of working with raw images, can you use the [ImageDataGenerator](https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/image/ImageDataGenerator?hl=en) as a means to normalize your images when you feed into your model.fit? You could also make use of the **preprocessing_function** parameter to pass a function on how you want to normalise the data.", "1107440": "Thanks for the reply, but unfortunately I am using TFRecords and looking for a way to normalize the dataset while using tf.data pipeline", "1107485": "Can you use the **map** method on your tf.data and feed in a lambda function for your normalization?\n\n\n`dataset = tf.data.TFRecordDataset([\"image1.tfrecords\", \"image2.tfrecords\"])\ndataset = dataset.map(lambda x: x/255)\n`", "1107515": "Yes, I am aware of this, but what I have is the channel wise mean and standard deviations. So is there any way to normalize (or Standardize would be the right term?) using them?" }, "source": "meta" }