Hi, I am trying to create a TFRecord and I am slightly confused on how they should be formatted. Should they be images and then have probabilities for each number class (1-203K) or should they be formatted a different way? I have tried looking at previous challenges involving TPUs but I wasn't able to figure out how targets were presented? Please Help!
", "messages": [ { "id": 937037, "postDate": "2020-07-20T17:22:46.433Z", "content": "Have you tried the following?
\n", "rawMarkdown": "Have you tried the following?\n\nhttps://github.com/tensorflow/models/tree/master/research/delf/delf/python/training#prepare-the-data-for-training", "votes": 1 }, { "id": 937587, "postDate": "2020-07-21T05:03:28.183Z", "content": "Any luck with this?
", "rawMarkdown": "Any luck with this?", "votes": 2, "replies": [ { "id": 938386, "postDate": "2020-07-21T13:48:17.570Z", "content": "Well since the project is so big I have to go out and buy a hard drive to store it. I think I should be able to get a TFRecord dataset out buy Thurs./Fri.
", "rawMarkdown": "Well since the project is so big I have to go out and buy a hard drive to store it. I think I should be able to get a TFRecord dataset out buy Thurs./Fri.", "votes": 2 }, { "id": 938486, "postDate": "2020-07-21T15:09:30.323Z", "content": "Are you trying to create tfrecords for the dataset available for download through the data section of the competition(~108 GB) or the official gldv2 dataset which is worth ~500 GB?
", "rawMarkdown": "Are you trying to create tfrecords for the dataset available for download through the data section of the competition(~108 GB) or the official gldv2 dataset which is worth ~500 GB?", "votes": 6 }, { "id": 938559, "postDate": "2020-07-21T15:52:42.923Z", "content": "To be honest, any would work. Is there a big difference between the 2? Is the official gldv2 dataset just an extension? I'm still fairly uncertain about the differences between the 2, such as the difference between train.csv and train_clean.csv?
", "rawMarkdown": "To be honest, any would work. Is there a big difference between the 2? Is the official gldv2 dataset just an extension? I'm still fairly uncertain about the differences between the 2, such as the difference between train.csv and train_clean.csv?", "votes": 2 }, { "id": 938896, "postDate": "2020-07-21T20:50:51.397Z", "content": "train_clean.csv does the job. It takes into account the current competition data(~108 GB) and creates tfrecords for that.
", "rawMarkdown": "train_clean.csv does the job. It takes into account the current competition data(~108 GB) and creates tfrecords for that.", "votes": 6 } ] }, { "id": 937256, "postDate": "2020-07-20T21:30:10.750Z", "content": "@pukkinming it seems to me that the landmark ids are encoded as an _int64_feature but when I try using my own dataset encoded in the same way during training the loss equates to nan and the accuracy stays at 0. What am I missing?
", "rawMarkdown": "@pukkinming it seems to me that the landmark ids are encoded as an _int64_feature but when I try using my own dataset encoded in the same way during training the loss equates to nan and the accuracy stays at 0. What am I missing?" }, { "id": 937123, "postDate": "2020-07-20T18:29:03.060Z", "content": "Would I be able to simply plug in the dataset for the train directories?
", "rawMarkdown": "Would I be able to simply plug in the dataset for the train directories?" }, { "id": 937082, "postDate": "2020-07-20T17:57:23.310Z", "content": "Hi, slightly confused here, in build_tfrecords.py they use the dataset GLDv2. Is this the same thing as the dataset here? If not, how would I convert this to work with the dataset from this competition?
", "rawMarkdown": "Hi, slightly confused here, in build_tfrecords.py they use the dataset GLDv2. Is this the same thing as the dataset here? If not, how would I convert this to work with the dataset from this competition?", "replies": [ { "id": 937113, "postDate": "2020-07-20T18:20:18.467Z", "content": "I believe you can manipulate the parameters you pass to the following:
\npython3 buildimagedataset.py \\
\n --traincsvpath=gldv2dataset/train/train.csv \\\n --traincleancsvpath=gldv2dataset/train/trainclean.csv \\
\n --traindirectory=gldv2dataset/train///*/ \\
\n --outputdirectory=gldv2dataset/tfrecord/ \\
\n --numshards=128 \\\n --generatetrainvalidationsplits \\
\n --validationsplitsize=0.2
You can change the parameters accordingly. For example, you can replace the train.csv with the files you specify.
", "rawMarkdown": "You can change the parameters accordingly. For example, you can replace the train.csv with the files you specify. " } ] }, { "id": 936971, "postDate": "2020-07-20T16:23:16.430Z", "content": "Hi, I am trying to create a TFRecord and I am slightly confused on how they should be formatted. Should they be images and then have probabilities for each number class (1-203K) or should they be formatted a different way? I have tried looking at previous challenges involving TPUs but I wasn't able to figure out how targets were presented? Please Help!
", "rawMarkdown": "Hi, I am trying to create a TFRecord and I am slightly confused on how they should be formatted. Should they be images and then have probabilities for each number class (1-203K) or should they be formatted a different way? I have tried looking at previous challenges involving TPUs but I wasn't able to figure out how targets were presented? Please Help!" }, { "id": 937081, "postDate": "2020-07-20T17:56:47.203Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 937037, "author_name": "FP", "author_url": "", "post_date": "2020-07-20T17:22:46.433000", "content": "Have you tried the following?
\n", "votes": 1, "replies": [] }, { "id": 937587, "author_name": "Chandan Verma", "author_url": "", "post_date": "2020-07-21T05:03:28.183000", "content": "Any luck with this?
", "votes": 2, "replies": [ { "id": 938386, "author_name": "Fateh Aliyev", "author_url": "", "post_date": "2020-07-21T13:48:17.570000", "content": "Well since the project is so big I have to go out and buy a hard drive to store it. I think I should be able to get a TFRecord dataset out buy Thurs./Fri.
", "votes": 2, "replies": [] }, { "id": 938486, "author_name": "Chandan Verma", "author_url": "", "post_date": "2020-07-21T15:09:30.323000", "content": "Are you trying to create tfrecords for the dataset available for download through the data section of the competition(~108 GB) or the official gldv2 dataset which is worth ~500 GB?
", "votes": 6, "replies": [] }, { "id": 938559, "author_name": "Fateh Aliyev", "author_url": "", "post_date": "2020-07-21T15:52:42.923000", "content": "To be honest, any would work. Is there a big difference between the 2? Is the official gldv2 dataset just an extension? I'm still fairly uncertain about the differences between the 2, such as the difference between train.csv and train_clean.csv?
", "votes": 2, "replies": [] }, { "id": 938896, "author_name": "Chandan Verma", "author_url": "", "post_date": "2020-07-21T20:50:51.397000", "content": "train_clean.csv does the job. It takes into account the current competition data(~108 GB) and creates tfrecords for that.
", "votes": 6, "replies": [] } ] }, { "id": 937256, "author_name": "Fateh Aliyev", "author_url": "", "post_date": "2020-07-20T21:30:10.750000", "content": "@pukkinming it seems to me that the landmark ids are encoded as an _int64_feature but when I try using my own dataset encoded in the same way during training the loss equates to nan and the accuracy stays at 0. What am I missing?
", "votes": 0, "replies": [] }, { "id": 937123, "author_name": "Fateh Aliyev", "author_url": "", "post_date": "2020-07-20T18:29:03.060000", "content": "Would I be able to simply plug in the dataset for the train directories?
", "votes": 0, "replies": [] }, { "id": 937082, "author_name": "Fateh Aliyev", "author_url": "", "post_date": "2020-07-20T17:57:23.310000", "content": "Hi, slightly confused here, in build_tfrecords.py they use the dataset GLDv2. Is this the same thing as the dataset here? If not, how would I convert this to work with the dataset from this competition?
", "votes": 0, "replies": [ { "id": 937113, "author_name": "FP", "author_url": "", "post_date": "2020-07-20T18:20:18.467000", "content": "I believe you can manipulate the parameters you pass to the following:
\npython3 buildimagedataset.py \\
\n --traincsvpath=gldv2dataset/train/train.csv \\\n --traincleancsvpath=gldv2dataset/train/trainclean.csv \\
\n --traindirectory=gldv2dataset/train///*/ \\
\n --outputdirectory=gldv2dataset/tfrecord/ \\
\n --numshards=128 \\\n --generatetrainvalidationsplits \\
\n --validationsplitsize=0.2
You can change the parameters accordingly. For example, you can replace the train.csv with the files you specify.
", "votes": 0, "replies": [] } ] }, { "id": 937081, "author_name": "", "author_url": "", "post_date": "2020-07-20T17:56:47.203000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "937037": "Have you tried the following?\n\nhttps://github.com/tensorflow/models/tree/master/research/delf/delf/python/training#prepare-the-data-for-training", "937587": "Any luck with this?", "937256": "@pukkinming it seems to me that the landmark ids are encoded as an _int64_feature but when I try using my own dataset encoded in the same way during training the loss equates to nan and the accuracy stays at 0. What am I missing?", "937123": "Would I be able to simply plug in the dataset for the train directories?", "937082": "Hi, slightly confused here, in build_tfrecords.py they use the dataset GLDv2. Is this the same thing as the dataset here? If not, how would I convert this to work with the dataset from this competition?", "936971": "Hi, I am trying to create a TFRecord and I am slightly confused on how they should be formatted. Should they be images and then have probabilities for each number class (1-203K) or should they be formatted a different way? I have tried looking at previous challenges involving TPUs but I wasn't able to figure out how targets were presented? Please Help!", "937081": "" } }