Hello,
\nI am trying to read the images using TensorFlow, following the guide from: https://www.tensorflow.org/guide/data#decoding_image_data_and_resizing_it
I obtain the following code:
\nlist_ds = tf.data.Dataset.list_files('../input/plant-pathology-2021-fgvc8/train_images/*')\n\ndf = pd.read_csv('../input/plant-pathology-2021-fgvc8/train.csv')\n\ndef parse_image(filepath):\n parts = tf.strings.split(filepath, os.sep)\n\n filename = tf.get_static_value(parts[-1]).decode('utf-8')\n\n image = tf.io.read_file(filepath)\n image = tf.image.decode_jpeg(image)\n image = tf.image.convert_image_dtype(image, tf.float32)\n image = tf.image.resize(image, [128, 128])\n\n label = df[df['image'] == filename]['labels']\n\n return image, label\n\nimages_ds = list_ds.map(lambda x: tf.py_function(parse_image, [x], [tf.float32, tf.string]))\nThen I create a simple model just to try if everything works:
\nimport tensorflow as tf\nprint(tf.__version__)\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Dense, Flatten, Conv2D, MaxPooling2D\n\nmodel = Sequential([\n Conv2D(16,(3,3), activation='relu', input_shape=(128,128,3)),\n MaxPooling2D((3,3)),\n Flatten(),\n Dense(64, activation='relu'),\n Dense(12, activation='softmax') \n])\n\nmodel.compile(optimizer='adam',\n loss='sparse_categorical_crossentropy',\n metrics=['accuracy'])\nHowever when I try to fit the model to images_ds:
\nhistory = model.fit(images_ds, epochs=8, batch_size=256)\nI get the following error:
\nValueError: as_list() is not defined on an unknown TensorShape.
From what I have read online it is due to the use of tf.py_function and the tf.get_static_value(), but I am not sure how to load the images with their correspondig labels without using it.
", "messages": [ { "id": "1264608", "postDate": "04/06/2021 09:50:13", "content": "Hello,
\nI am trying to read the images using TensorFlow, following the guide from: https://www.tensorflow.org/guide/data#decoding_image_data_and_resizing_it
I obtain the following code:
\nlist_ds = tf.data.Dataset.list_files('../input/plant-pathology-2021-fgvc8/train_images/*')\n\ndf = pd.read_csv('../input/plant-pathology-2021-fgvc8/train.csv')\n\ndef parse_image(filepath):\n parts = tf.strings.split(filepath, os.sep)\n\n filename = tf.get_static_value(parts[-1]).decode('utf-8')\n\n image = tf.io.read_file(filepath)\n image = tf.image.decode_jpeg(image)\n image = tf.image.convert_image_dtype(image, tf.float32)\n image = tf.image.resize(image, [128, 128])\n\n label = df[df['image'] == filename]['labels']\n\n return image, label\n\nimages_ds = list_ds.map(lambda x: tf.py_function(parse_image, [x], [tf.float32, tf.string]))\nThen I create a simple model just to try if everything works:
\nimport tensorflow as tf\nprint(tf.__version__)\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Dense, Flatten, Conv2D, MaxPooling2D\n\nmodel = Sequential([\n Conv2D(16,(3,3), activation='relu', input_shape=(128,128,3)),\n MaxPooling2D((3,3)),\n Flatten(),\n Dense(64, activation='relu'),\n Dense(12, activation='softmax') \n])\n\nmodel.compile(optimizer='adam',\n loss='sparse_categorical_crossentropy',\n metrics=['accuracy'])\nHowever when I try to fit the model to images_ds:
\nhistory = model.fit(images_ds, epochs=8, batch_size=256)\nI get the following error:
\nValueError: as_list() is not defined on an unknown TensorShape.
From what I have read online it is due to the use of tf.py_function and the tf.get_static_value(), but I am not sure how to load the images with their correspondig labels without using it.
", "rawMarkdown": "Hello,\nI am trying to read the images using TensorFlow, following the guide from: https://www.tensorflow.org/guide/data#decoding_image_data_and_resizing_it\n\nI obtain the following code:\n\n```py\nlist_ds = tf.data.Dataset.list_files('../input/plant-pathology-2021-fgvc8/train_images/*')\n\ndf = pd.read_csv('../input/plant-pathology-2021-fgvc8/train.csv')\n\ndef parse_image(filepath):\n parts = tf.strings.split(filepath, os.sep)\n\n filename = tf.get_static_value(parts[-1]).decode('utf-8')\n\n image = tf.io.read_file(filepath)\n image = tf.image.decode_jpeg(image)\n image = tf.image.convert_image_dtype(image, tf.float32)\n image = tf.image.resize(image, [128, 128])\n \n label = df[df['image'] == filename]['labels']\n \n return image, label\n\nimages_ds = list_ds.map(lambda x: tf.py_function(parse_image, [x], [tf.float32, tf.string]))\n```\n\nThen I create a simple model just to try if everything works:\n\n```\nimport tensorflow as tf\nprint(tf.__version__)\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Dense, Flatten, Conv2D, MaxPooling2D\n\nmodel = Sequential([\n Conv2D(16,(3,3), activation='relu', input_shape=(128,128,3)),\n MaxPooling2D((3,3)),\n Flatten(),\n Dense(64, activation='relu'),\n Dense(12, activation='softmax') \n])\n\nmodel.compile(optimizer='adam',\n loss='sparse_categorical_crossentropy',\n metrics=['accuracy'])\n``` \n\nHowever when I try to fit the model to images_ds:\n\n```\nhistory = model.fit(images_ds, epochs=8, batch_size=256)\n```\n\nI get the following error:\n\n``` ValueError: as_list() is not defined on an unknown TensorShape.```\n\nFrom what I have read online it is due to the use of tf.py_function and the tf.get_static_value(), but I am not sure how to load the images with their correspondig labels without using it.", "votes": null }, { "id": "1264963", "postDate": "04/06/2021 14:20:57", "content": "Hey Even at first tried with images_ds = list_ds.map(lambda x: tf.py_function(parse_image, [x], [tf.float32, tf.string])) but got many errors. I suggest you not to use this method. (I don't know why this error occured.)
You can go through my notebooks for different data pipeline.
\nTPU Training : https://www.kaggle.com/shanmukh05/plant-pathology-2k21-baseline-tpu-training
\nGPU Training : https://www.kaggle.com/shanmukh05/plant-pathology-2k21-baseline-gpu-training
It's better to prepare TFRecords beforehand, and use them. It will be faster.
", "rawMarkdown": "It's better to prepare TFRecords beforehand, and use them. It will be faster.", "votes": null }, { "id": "1265133", "postDate": "04/06/2021 16:33:57", "content": "Any good notebook that shows how this would work?
", "rawMarkdown": "Any good notebook that shows how this would work?", "votes": null }, { "id": "1265224", "postDate": "04/06/2021 17:33:47", "content": "For example, https://www.kaggle.com/nickuzmenkov/plant-pathology-2021-making-tfrecords
", "rawMarkdown": "For example, https://www.kaggle.com/nickuzmenkov/plant-pathology-2021-making-tfrecords", "votes": null }, { "id": "1310654", "postDate": "05/16/2021 19:41:28", "content": "you can create the labels in the format you want in the dataframe in different column and then use it in ds=tf.data.Dataset.from_tensor_slices(file_paths_columns,label_column)
\nfrom there you can simply map like
\nds=ds.map(decode_file)
and then the normal pipeline in which ever way you want.
", "rawMarkdown": "you can create the labels in the format you want in the dataframe in different column and then use it in ds=tf.data.Dataset.from_tensor_slices(file_paths_columns,label_column)\n\nfrom there you can simply map like \nds=ds.map(decode_file)\n\nand then the normal pipeline in which ever way you want.", "votes": null } ], "comments": [ { "id": 1264963, "author_name": "shanmukh05", "author_url": "", "post_date": "04/06/2021 14:20:57", "content": "Hey Even at first tried with images_ds = list_ds.map(lambda x: tf.py_function(parse_image, [x], [tf.float32, tf.string])) but got many errors. I suggest you not to use this method. (I don't know why this error occured.)
You can go through my notebooks for different data pipeline.
\nTPU Training : https://www.kaggle.com/shanmukh05/plant-pathology-2k21-baseline-tpu-training
\nGPU Training : https://www.kaggle.com/shanmukh05/plant-pathology-2k21-baseline-gpu-training
It's better to prepare TFRecords beforehand, and use them. It will be faster.
", "votes": null, "replies": [ { "id": 1265133, "author_name": "coilysum", "author_url": "", "post_date": "04/06/2021 16:33:57", "content": "Any good notebook that shows how this would work?
", "votes": null, "replies": [] }, { "id": 1265224, "author_name": "atamazian", "author_url": "", "post_date": "04/06/2021 17:33:47", "content": "For example, https://www.kaggle.com/nickuzmenkov/plant-pathology-2021-making-tfrecords
", "votes": null, "replies": [] } ] }, { "id": 1310654, "author_name": "sagarkarki136", "author_url": "", "post_date": "05/16/2021 19:41:28", "content": "you can create the labels in the format you want in the dataframe in different column and then use it in ds=tf.data.Dataset.from_tensor_slices(file_paths_columns,label_column)
\nfrom there you can simply map like
\nds=ds.map(decode_file)
and then the normal pipeline in which ever way you want.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1264608": "Hello,\nI am trying to read the images using TensorFlow, following the guide from: https://www.tensorflow.org/guide/data#decoding_image_data_and_resizing_it\n\nI obtain the following code:\n\n```py\nlist_ds = tf.data.Dataset.list_files('../input/plant-pathology-2021-fgvc8/train_images/*')\n\ndf = pd.read_csv('../input/plant-pathology-2021-fgvc8/train.csv')\n\ndef parse_image(filepath):\n parts = tf.strings.split(filepath, os.sep)\n\n filename = tf.get_static_value(parts[-1]).decode('utf-8')\n\n image = tf.io.read_file(filepath)\n image = tf.image.decode_jpeg(image)\n image = tf.image.convert_image_dtype(image, tf.float32)\n image = tf.image.resize(image, [128, 128])\n \n label = df[df['image'] == filename]['labels']\n \n return image, label\n\nimages_ds = list_ds.map(lambda x: tf.py_function(parse_image, [x], [tf.float32, tf.string]))\n```\n\nThen I create a simple model just to try if everything works:\n\n```\nimport tensorflow as tf\nprint(tf.__version__)\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Dense, Flatten, Conv2D, MaxPooling2D\n\nmodel = Sequential([\n Conv2D(16,(3,3), activation='relu', input_shape=(128,128,3)),\n MaxPooling2D((3,3)),\n Flatten(),\n Dense(64, activation='relu'),\n Dense(12, activation='softmax') \n])\n\nmodel.compile(optimizer='adam',\n loss='sparse_categorical_crossentropy',\n metrics=['accuracy'])\n``` \n\nHowever when I try to fit the model to images_ds:\n\n```\nhistory = model.fit(images_ds, epochs=8, batch_size=256)\n```\n\nI get the following error:\n\n``` ValueError: as_list() is not defined on an unknown TensorShape.```\n\nFrom what I have read online it is due to the use of tf.py_function and the tf.get_static_value(), but I am not sure how to load the images with their correspondig labels without using it.", "1264963": "Hey Even at first tried with `images_ds = list_ds.map(lambda x: tf.py_function(parse_image, [x], [tf.float32, tf.string]))` but got many errors. I suggest you not to use this method. (I don't know why this error occured.)\n\nYou can go through my notebooks for different data pipeline.\nTPU Training : https://www.kaggle.com/shanmukh05/plant-pathology-2k21-baseline-tpu-training\nGPU Training : https://www.kaggle.com/shanmukh05/plant-pathology-2k21-baseline-gpu-training", "1265119": "It's better to prepare TFRecords beforehand, and use them. It will be faster.", "1265133": "Any good notebook that shows how this would work?", "1265224": "For example, https://www.kaggle.com/nickuzmenkov/plant-pathology-2021-making-tfrecords", "1310654": "you can create the labels in the format you want in the dataframe in different column and then use it in ds=tf.data.Dataset.from_tensor_slices(file_paths_columns,label_column)\n\nfrom there you can simply map like \nds=ds.map(decode_file)\n\nand then the normal pipeline in which ever way you want." }, "source": "meta" }