{ "id": 168632, "title": "Help in TFRecords", "url": "/competitions/siim-isic-melanoma-classification/discussion/168632", "author_name": "Aditya Baurai", "post_date": "2020-07-21T10:25:24.986000", "votes": 0, "comment_count": 11, "views": 0, "content": "

I am having some trouble understanding TFRecords! Having checked the google TF documentation everywhere to access them they defined feature set in advance.

\n

So, how would one get the idea of the features present in TFRecord files beforehand?

\n

Like in this competition, folks have written :

\n

tfrecord_format = {
\n \"image\": tf.io.FixedLenFeature([], tf.string),
\n \"target\": tf.io.FixedLenFeature([], tf.int64)
\n }

\n

How do we know the underlying format and the exact feature name present? In TF documentation they first made their own TFRecord files and then used the features(they knew because they defined the file) to read the TFRecord.

\n

How would an outsider know the name and type of the features present?

", "messages": [ { "id": 938112, "postDate": "2020-07-21T10:58:44.993Z", "content": "

check this notebook you will get all answers How To Create TFRecords.

\n\n

How would an outsider know the name and type of the features present?

\n\n

you have mansion tfrecord_format along with TFRecords files

", "rawMarkdown": "check this notebook you will get all answers [How To Create TFRecords](https://www.kaggle.com/cdeotte/how-to-create-tfrecords).\n\n`How would an outsider know the name and type of the features present?`\n\nyou have mansion **tfrecord_format** along with TFRecords files", "votes": 1, "replies": [ { "id": 938280, "postDate": "2020-07-21T12:35:48.663Z", "content": "

See, here also the GM defined tfrecords first. He wrote them and then read. However, if I want to use the tfrecords available in the dataset, i would need to know the features it has, right?

\n

That's what I'm asking how to get the content of that file so I may know the tfrecord features in order to use them.

\n

For example here :

\n

https://www.kaggle.com/amyjang/tensorflow-transfer-learning-melanoma

\n

The author used \"image\" and \"target\" features from the record. But how did one come by those?

", "rawMarkdown": "See, here also the GM defined tfrecords first. He wrote them and then read. However, if I want to use the tfrecords available in the dataset, i would need to know the features it has, right?\n\nThat's what I'm asking how to get the content of that file so I may know the tfrecord features in order to use them. \n\nFor example here :\n\nhttps://www.kaggle.com/amyjang/tensorflow-transfer-learning-melanoma \n\nThe author used \"image\" and \"target\" features from the record. But how did one come by those?" } ] }, { "id": 938069, "postDate": "2020-07-21T10:25:24.987Z", "content": "

I am having some trouble understanding TFRecords! Having checked the google TF documentation everywhere to access them they defined feature set in advance.

\n

So, how would one get the idea of the features present in TFRecord files beforehand?

\n

Like in this competition, folks have written :

\n

tfrecord_format = {
\n \"image\": tf.io.FixedLenFeature([], tf.string),
\n \"target\": tf.io.FixedLenFeature([], tf.int64)
\n }

\n

How do we know the underlying format and the exact feature name present? In TF documentation they first made their own TFRecord files and then used the features(they knew because they defined the file) to read the TFRecord.

\n

How would an outsider know the name and type of the features present?

", "rawMarkdown": "I am having some trouble understanding TFRecords! Having checked the google TF documentation everywhere to access them they defined feature set in advance. \n\nSo, how would one get the idea of the features present in TFRecord files beforehand?\n\nLike in this competition, folks have written : \n\ntfrecord_format = {\n \"image\": tf.io.FixedLenFeature([], tf.string),\n \"target\": tf.io.FixedLenFeature([], tf.int64)\n } \n\nHow do we know the underlying format and the exact feature name present? In TF documentation they first made their own TFRecord files and then used the features(they knew because they defined the file) to read the TFRecord.\n\nHow would an outsider know the name and type of the features present?" }, { "id": 939650, "postDate": "2020-07-22T11:01:16.363Z", "content": "

https://www.kaggle.com/cdeotte/how-to-create-tfrecords

\n

this is how they make tf records

\n

You can access the inner features present using
\nimport tensorflow as tf

\n

for example in tf.compat.v1.pythonio.tfrecord_iterator(\"data/foobar.tfrecord\"):
\nprint(tf.train.Example.FromString(example))

\n

use the path accordingly

\n

And the a tf record comprises both image and features present in csv file, the image is saved as image feature thats why you can see the feature name in it

", "rawMarkdown": "https://www.kaggle.com/cdeotte/how-to-create-tfrecords\n\nthis is how they make tf records\n\nYou can access the inner features present using\nimport tensorflow as tf\n\nfor example in tf.compat.v1.pythonio.tfrecord_iterator(\"data/foobar.tfrecord\"):\nprint(tf.train.Example.FromString(example))\n\nuse the path accordingly\n\n\nAnd the a tf record comprises both image and features present in csv file, the image is saved as image feature thats why you can see the feature name in it" }, { "id": 938307, "postDate": "2020-07-21T12:54:45.650Z", "content": "

did you try thr code that i gave you, i worked for me in tf v2

", "rawMarkdown": "did you try thr code that i gave you, i worked for me in tf v2", "replies": [ { "id": 939653, "postDate": "2020-07-22T11:03:07.420Z", "content": "

No., it didn't

", "rawMarkdown": "No., it didn't" } ] }, { "id": 938262, "postDate": "2020-07-21T12:28:09.477Z", "content": "

USE THIS CODE TO ACCESS FEATURES OF TF records

\n\n

**import tensorflow as tf

\n\n

for example in tf.compat.v1.python_io.tf_record_iterator(\"data/foobar.tfrecord\"):\n print(tf.train.Example.FromString(example))**

", "rawMarkdown": "USE THIS CODE TO ACCESS FEATURES OF TF records\n\n**import tensorflow as tf\n\nfor example in tf.compat.v1.python_io.tf_record_iterator(\"data/foobar.tfrecord\"):\n print(tf.train.Example.FromString(example))**", "replies": [ { "id": 938272, "postDate": "2020-07-21T12:33:25.477Z", "content": "

Is there a Tensorflow 2 version available?

", "rawMarkdown": "Is there a Tensorflow 2 version available?" } ] }, { "id": 939418, "postDate": "2020-07-22T08:28:38.010Z", "rawMarkdown": "", "votes": -1, "isDeleted": true, "replies": [ { "id": 939634, "postDate": "2020-07-22T10:46:51.903Z", "content": "

I thought that too initially, however check this notebook :

\n

*https://www.kaggle.com/amyjang/tensorflow-transfer-learning-melanoma

\n

in the line :

\n

tfrecord_format = {
\n\"image\": tf.io.FixedLenFeature([], tf.string),
\n\"target\": tf.io.FixedLenFeature([], tf.int64)
\n}*

\n

\"image\" is not a name in csv file. It's \"image_name\"

", "rawMarkdown": "I thought that too initially, however check this notebook : \n\n*https://www.kaggle.com/amyjang/tensorflow-transfer-learning-melanoma \n\nin the line : \n\ntfrecord_format = {\n\"image\": tf.io.FixedLenFeature([], tf.string),\n\"target\": tf.io.FixedLenFeature([], tf.int64)\n}*\n\n\"image\" is not a name in csv file. It's \"image_name\"" }, { "id": 939659, "postDate": "2020-07-22T11:10:16.167Z", "content": "

Yes, this works!! Thank you

", "rawMarkdown": "Yes, this works!! Thank you" }, { "id": 939724, "postDate": "2020-07-22T12:09:42.160Z", "rawMarkdown": "", "isDeleted": true } ] } ], "comments": [ { "id": 938112, "author_name": "Vatsal Parsaniya", "author_url": "", "post_date": "2020-07-21T10:58:44.993000", "content": "

check this notebook you will get all answers How To Create TFRecords.

\n\n

How would an outsider know the name and type of the features present?

\n\n

you have mansion tfrecord_format along with TFRecords files

", "votes": 1, "replies": [ { "id": 938280, "author_name": "Aditya Baurai", "author_url": "", "post_date": "2020-07-21T12:35:48.663000", "content": "

See, here also the GM defined tfrecords first. He wrote them and then read. However, if I want to use the tfrecords available in the dataset, i would need to know the features it has, right?

\n

That's what I'm asking how to get the content of that file so I may know the tfrecord features in order to use them.

\n

For example here :

\n

https://www.kaggle.com/amyjang/tensorflow-transfer-learning-melanoma

\n

The author used \"image\" and \"target\" features from the record. But how did one come by those?

", "votes": 0, "replies": [] } ] }, { "id": 939650, "author_name": "khailash santhakumar", "author_url": "", "post_date": "2020-07-22T11:01:16.363000", "content": "

https://www.kaggle.com/cdeotte/how-to-create-tfrecords

\n

this is how they make tf records

\n

You can access the inner features present using
\nimport tensorflow as tf

\n

for example in tf.compat.v1.pythonio.tfrecord_iterator(\"data/foobar.tfrecord\"):
\nprint(tf.train.Example.FromString(example))

\n

use the path accordingly

\n

And the a tf record comprises both image and features present in csv file, the image is saved as image feature thats why you can see the feature name in it

", "votes": 0, "replies": [] }, { "id": 938307, "author_name": "khailash santhakumar", "author_url": "", "post_date": "2020-07-21T12:54:45.650000", "content": "

did you try thr code that i gave you, i worked for me in tf v2

", "votes": 0, "replies": [ { "id": 939653, "author_name": "Aditya Baurai", "author_url": "", "post_date": "2020-07-22T11:03:07.420000", "content": "

No., it didn't

", "votes": 0, "replies": [] } ] }, { "id": 938262, "author_name": "khailash santhakumar", "author_url": "", "post_date": "2020-07-21T12:28:09.477000", "content": "

USE THIS CODE TO ACCESS FEATURES OF TF records

\n\n

**import tensorflow as tf

\n\n

for example in tf.compat.v1.python_io.tf_record_iterator(\"data/foobar.tfrecord\"):\n print(tf.train.Example.FromString(example))**

", "votes": 0, "replies": [ { "id": 938272, "author_name": "Aditya Baurai", "author_url": "", "post_date": "2020-07-21T12:33:25.477000", "content": "

Is there a Tensorflow 2 version available?

", "votes": 0, "replies": [] } ] }, { "id": 939418, "author_name": "", "author_url": "", "post_date": "2020-07-22T08:28:38.010000", "content": "", "votes": -1, "replies": [ { "id": 939634, "author_name": "Aditya Baurai", "author_url": "", "post_date": "2020-07-22T10:46:51.903000", "content": "

I thought that too initially, however check this notebook :

\n

*https://www.kaggle.com/amyjang/tensorflow-transfer-learning-melanoma

\n

in the line :

\n

tfrecord_format = {
\n\"image\": tf.io.FixedLenFeature([], tf.string),
\n\"target\": tf.io.FixedLenFeature([], tf.int64)
\n}*

\n

\"image\" is not a name in csv file. It's \"image_name\"

", "votes": 0, "replies": [] }, { "id": 939659, "author_name": "Aditya Baurai", "author_url": "", "post_date": "2020-07-22T11:10:16.167000", "content": "

Yes, this works!! Thank you

", "votes": 0, "replies": [] }, { "id": 939724, "author_name": "", "author_url": "", "post_date": "2020-07-22T12:09:42.160000", "content": "", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "938112": "check this notebook you will get all answers [How To Create TFRecords](https://www.kaggle.com/cdeotte/how-to-create-tfrecords).\n\n`How would an outsider know the name and type of the features present?`\n\nyou have mansion **tfrecord_format** along with TFRecords files", "938069": "I am having some trouble understanding TFRecords! Having checked the google TF documentation everywhere to access them they defined feature set in advance. \n\nSo, how would one get the idea of the features present in TFRecord files beforehand?\n\nLike in this competition, folks have written : \n\ntfrecord_format = {\n \"image\": tf.io.FixedLenFeature([], tf.string),\n \"target\": tf.io.FixedLenFeature([], tf.int64)\n } \n\nHow do we know the underlying format and the exact feature name present? In TF documentation they first made their own TFRecord files and then used the features(they knew because they defined the file) to read the TFRecord.\n\nHow would an outsider know the name and type of the features present?", "939650": "https://www.kaggle.com/cdeotte/how-to-create-tfrecords\n\nthis is how they make tf records\n\nYou can access the inner features present using\nimport tensorflow as tf\n\nfor example in tf.compat.v1.pythonio.tfrecord_iterator(\"data/foobar.tfrecord\"):\nprint(tf.train.Example.FromString(example))\n\nuse the path accordingly\n\n\nAnd the a tf record comprises both image and features present in csv file, the image is saved as image feature thats why you can see the feature name in it", "938307": "did you try thr code that i gave you, i worked for me in tf v2", "938262": "USE THIS CODE TO ACCESS FEATURES OF TF records\n\n**import tensorflow as tf\n\nfor example in tf.compat.v1.python_io.tf_record_iterator(\"data/foobar.tfrecord\"):\n print(tf.train.Example.FromString(example))**", "939418": "" } }