{
  "id": 63095,
  "title": "tf records",
  "url": "/competitions/google-ai-open-images-object-detection-track/discussion/63095",
  "author_name": "",
  "post_date": "2018-08-11T23:33:20.029344400Z",
  "votes": 1,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Guys, how would you go by to convert the inference from tfrecords to csv files?</p>\n\n<p>So I used the tf object detection api to do some inference and after inference the predictions are outputted in tfrecords format. And I have been trying to convert it or parse it so I can get a csv file but no luck. Any help would be greatly appreciated.</p>",
  "messages": [
    {
      "id": "369095",
      "postDate": "08/11/2018 23:33:20",
      "content": "<p>Guys, how would you go by to convert the inference from tfrecords to csv files?</p>\n\n<p>So I used the tf object detection api to do some inference and after inference the predictions are outputted in tfrecords format. And I have been trying to convert it or parse it so I can get a csv file but no luck. Any help would be greatly appreciated.</p>",
      "rawMarkdown": "Guys, how would you go by to convert the inference from tfrecords to csv files?\n\nSo I used the tf object detection api to do some inference and after inference the predictions are outputted in tfrecords format. And I have been trying to convert it or parse it so I can get a csv file but no luck. Any help would be greatly appreciated.",
      "votes": null
    },
    {
      "id": "369098",
      "postDate": "08/11/2018 23:56:21",
      "content": "<p>I used the code in the offline evaluation script. Unfortunately it is on a machine that is off now. If you can't figure it out I'll boot it and send it to you somehow</p>",
      "rawMarkdown": "I used the code in the offline evaluation script. Unfortunately it is on a machine that is off now. If you can't figure it out I'll boot it and send it to you somehow",
      "votes": null
    },
    {
      "id": "369102",
      "postDate": "08/12/2018 00:21:08",
      "content": "<p>Thanks Moshel, will look for the evaluation script and convert</p>",
      "rawMarkdown": "Thanks Moshel, will look for the evaluation script and convert",
      "votes": null
    },
    {
      "id": "369395",
      "postDate": "08/13/2018 00:34:32",
      "content": "<p>def read_and_decode():</p>\n\n<pre><code>file_pattern = path + tfrecords_filename\nreader = tf.TFRecordReader\n\nkeys_to_features = {\n    'image/source_id': tf.FixedLenFeature((), tf.string),\n    'image/detection/bbox/ymin': tf.VarLenFeature([], tf.float32),\n    'image/detection/bbox/xmin': tf.VarLenFeature([], tf.float32),\n    'image/detection/bbox/ymax': tf.VarLenFeature([], tf.float32),\n    'image/detection/bbox/xmax': tf.VarLenFeature([], tf.float32),\n    'image/detection/label': tf.VarLenFeature([], tf.int64),\n    'image/detection/score': tf.VarLenFeature([], tf.float32)\n}\n\nfeatures = tf.parse_single_example(serialized_example_tensor, features=keys_to_features)\n\nslim_example_decoder = tf.contrib.slim.tfexample_decoder\n\nitems_to_handlers = {\n    'source_id': (slim_example_decoder.Tensor('image/source_id')),\n    'detection_boxes': (slim_example_decoder.BoundingBox(['ymin', 'xmin', 'ymax', 'xmax'], 'image/detection/bbox/')),\n    'detection_class_label': (slim_example_decoder.Tensor('image/detection/label')),\n    'detection_score': (slim_example_decoder.Tensor('image/detection/score'))\n}\n\ndecoder = slim_example_decoder.TFExampleDecoder(keys_to_features, items_to_handlers)\n\nreturn slim.dataset.Dataset(\n  data_sources=file_pattern,\n  reader=reader,\n  decoder=decoder,\n  num_samples=99999,\n  items_to_descriptions=_ITEMS_TO_DESCRIPTIONS)\n</code></pre>\n\n<p>slim = tf.contrib.slim\ndataset = read_and_decode()\nprovider = slim.dataset_data_provider.DatasetDataProvider(dataset, shuffle=False)</p>\n\n<p>[image, boxes, dlabels, score] = provider.get(['source_id', 'detection_boxes', 'detection_class_label',\n                                               'detection_score'])</p>\n\n<p>with tf.Session() as sess:\n    ini_op = tf.group(tf.global_variables_initializer(), tf.local_variables_initializer())\n    sess.run(ini_op)\n    coord = tf.train.Coordinator()\n    thread = tf.train.start_queue_runners(sess=sess,coord=coord)\n    for i in range(2):\n        cur_example_batch,cur_label_batch = sess.run([img_batch,label_batch])\n        print(cur_example_batch.shape)\n    coord.request_stop()\n    coord.join(thread)</p>",
      "rawMarkdown": "def read_and_decode():\n    \n    file_pattern = path + tfrecords_filename\n    reader = tf.TFRecordReader\n\n    keys_to_features = {\n        'image/source_id': tf.FixedLenFeature((), tf.string),\n        'image/detection/bbox/ymin': tf.VarLenFeature([], tf.float32),\n        'image/detection/bbox/xmin': tf.VarLenFeature([], tf.float32),\n        'image/detection/bbox/ymax': tf.VarLenFeature([], tf.float32),\n        'image/detection/bbox/xmax': tf.VarLenFeature([], tf.float32),\n        'image/detection/label': tf.VarLenFeature([], tf.int64),\n        'image/detection/score': tf.VarLenFeature([], tf.float32)\n    }\n\n    features = tf.parse_single_example(serialized_example_tensor, features=keys_to_features)\n\n    slim_example_decoder = tf.contrib.slim.tfexample_decoder\n\n    items_to_handlers = {\n        'source_id': (slim_example_decoder.Tensor('image/source_id')),\n        'detection_boxes': (slim_example_decoder.BoundingBox(['ymin', 'xmin', 'ymax', 'xmax'], 'image/detection/bbox/')),\n        'detection_class_label': (slim_example_decoder.Tensor('image/detection/label')),\n        'detection_score': (slim_example_decoder.Tensor('image/detection/score'))\n    }\n\n    decoder = slim_example_decoder.TFExampleDecoder(keys_to_features, items_to_handlers)\n\n    return slim.dataset.Dataset(\n      data_sources=file_pattern,\n      reader=reader,\n      decoder=decoder,\n      num_samples=99999,\n      items_to_descriptions=_ITEMS_TO_DESCRIPTIONS)\n\n\nslim = tf.contrib.slim\ndataset = read_and_decode()\nprovider = slim.dataset_data_provider.DatasetDataProvider(dataset, shuffle=False)\n\n[image, boxes, dlabels, score] = provider.get(['source_id', 'detection_boxes', 'detection_class_label',\n                                               'detection_score'])\n\n\nwith tf.Session() as sess:\n    ini_op = tf.group(tf.global_variables_initializer(), tf.local_variables_initializer())\n    sess.run(ini_op)\n    coord = tf.train.Coordinator()\n    thread = tf.train.start_queue_runners(sess=sess,coord=coord)\n    for i in range(2):\n        cur_example_batch,cur_label_batch = sess.run([img_batch,label_batch])\n        print(cur_example_batch.shape)\n    coord.request_stop()\n    coord.join(thread)",
      "votes": null
    },
    {
      "id": "369396",
      "postDate": "08/13/2018 00:36:20",
      "content": "<p>Hei @Moshel, this is kinda what I came up with, and I am getting serialization errors, how close is this to the code you had?</p>",
      "rawMarkdown": "Hei @Moshel, this is kinda what I came up with, and I am getting serialization errors, how close is this to the code you had?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 369098,
      "author_name": "moshel",
      "author_url": "",
      "post_date": "08/11/2018 23:56:21",
      "content": "<p>I used the code in the offline evaluation script. Unfortunately it is on a machine that is off now. If you can't figure it out I'll boot it and send it to you somehow</p>",
      "votes": null,
      "replies": [
        {
          "id": 369102,
          "author_name": "mukeshmithrakumar",
          "author_url": "",
          "post_date": "08/12/2018 00:21:08",
          "content": "<p>Thanks Moshel, will look for the evaluation script and convert</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 369395,
          "author_name": "mukeshmithrakumar",
          "author_url": "",
          "post_date": "08/13/2018 00:34:32",
          "content": "<p>def read_and_decode():</p>\n\n<pre><code>file_pattern = path + tfrecords_filename\nreader = tf.TFRecordReader\n\nkeys_to_features = {\n    'image/source_id': tf.FixedLenFeature((), tf.string),\n    'image/detection/bbox/ymin': tf.VarLenFeature([], tf.float32),\n    'image/detection/bbox/xmin': tf.VarLenFeature([], tf.float32),\n    'image/detection/bbox/ymax': tf.VarLenFeature([], tf.float32),\n    'image/detection/bbox/xmax': tf.VarLenFeature([], tf.float32),\n    'image/detection/label': tf.VarLenFeature([], tf.int64),\n    'image/detection/score': tf.VarLenFeature([], tf.float32)\n}\n\nfeatures = tf.parse_single_example(serialized_example_tensor, features=keys_to_features)\n\nslim_example_decoder = tf.contrib.slim.tfexample_decoder\n\nitems_to_handlers = {\n    'source_id': (slim_example_decoder.Tensor('image/source_id')),\n    'detection_boxes': (slim_example_decoder.BoundingBox(['ymin', 'xmin', 'ymax', 'xmax'], 'image/detection/bbox/')),\n    'detection_class_label': (slim_example_decoder.Tensor('image/detection/label')),\n    'detection_score': (slim_example_decoder.Tensor('image/detection/score'))\n}\n\ndecoder = slim_example_decoder.TFExampleDecoder(keys_to_features, items_to_handlers)\n\nreturn slim.dataset.Dataset(\n  data_sources=file_pattern,\n  reader=reader,\n  decoder=decoder,\n  num_samples=99999,\n  items_to_descriptions=_ITEMS_TO_DESCRIPTIONS)\n</code></pre>\n\n<p>slim = tf.contrib.slim\ndataset = read_and_decode()\nprovider = slim.dataset_data_provider.DatasetDataProvider(dataset, shuffle=False)</p>\n\n<p>[image, boxes, dlabels, score] = provider.get(['source_id', 'detection_boxes', 'detection_class_label',\n                                               'detection_score'])</p>\n\n<p>with tf.Session() as sess:\n    ini_op = tf.group(tf.global_variables_initializer(), tf.local_variables_initializer())\n    sess.run(ini_op)\n    coord = tf.train.Coordinator()\n    thread = tf.train.start_queue_runners(sess=sess,coord=coord)\n    for i in range(2):\n        cur_example_batch,cur_label_batch = sess.run([img_batch,label_batch])\n        print(cur_example_batch.shape)\n    coord.request_stop()\n    coord.join(thread)</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 369396,
          "author_name": "mukeshmithrakumar",
          "author_url": "",
          "post_date": "08/13/2018 00:36:20",
          "content": "<p>Hei @Moshel, this is kinda what I came up with, and I am getting serialization errors, how close is this to the code you had?</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "369095": "Guys, how would you go by to convert the inference from tfrecords to csv files?\n\nSo I used the tf object detection api to do some inference and after inference the predictions are outputted in tfrecords format. And I have been trying to convert it or parse it so I can get a csv file but no luck. Any help would be greatly appreciated.",
    "369098": "I used the code in the offline evaluation script. Unfortunately it is on a machine that is off now. If you can't figure it out I'll boot it and send it to you somehow",
    "369102": "Thanks Moshel, will look for the evaluation script and convert",
    "369395": "def read_and_decode():\n    \n    file_pattern = path + tfrecords_filename\n    reader = tf.TFRecordReader\n\n    keys_to_features = {\n        'image/source_id': tf.FixedLenFeature((), tf.string),\n        'image/detection/bbox/ymin': tf.VarLenFeature([], tf.float32),\n        'image/detection/bbox/xmin': tf.VarLenFeature([], tf.float32),\n        'image/detection/bbox/ymax': tf.VarLenFeature([], tf.float32),\n        'image/detection/bbox/xmax': tf.VarLenFeature([], tf.float32),\n        'image/detection/label': tf.VarLenFeature([], tf.int64),\n        'image/detection/score': tf.VarLenFeature([], tf.float32)\n    }\n\n    features = tf.parse_single_example(serialized_example_tensor, features=keys_to_features)\n\n    slim_example_decoder = tf.contrib.slim.tfexample_decoder\n\n    items_to_handlers = {\n        'source_id': (slim_example_decoder.Tensor('image/source_id')),\n        'detection_boxes': (slim_example_decoder.BoundingBox(['ymin', 'xmin', 'ymax', 'xmax'], 'image/detection/bbox/')),\n        'detection_class_label': (slim_example_decoder.Tensor('image/detection/label')),\n        'detection_score': (slim_example_decoder.Tensor('image/detection/score'))\n    }\n\n    decoder = slim_example_decoder.TFExampleDecoder(keys_to_features, items_to_handlers)\n\n    return slim.dataset.Dataset(\n      data_sources=file_pattern,\n      reader=reader,\n      decoder=decoder,\n      num_samples=99999,\n      items_to_descriptions=_ITEMS_TO_DESCRIPTIONS)\n\n\nslim = tf.contrib.slim\ndataset = read_and_decode()\nprovider = slim.dataset_data_provider.DatasetDataProvider(dataset, shuffle=False)\n\n[image, boxes, dlabels, score] = provider.get(['source_id', 'detection_boxes', 'detection_class_label',\n                                               'detection_score'])\n\n\nwith tf.Session() as sess:\n    ini_op = tf.group(tf.global_variables_initializer(), tf.local_variables_initializer())\n    sess.run(ini_op)\n    coord = tf.train.Coordinator()\n    thread = tf.train.start_queue_runners(sess=sess,coord=coord)\n    for i in range(2):\n        cur_example_batch,cur_label_batch = sess.run([img_batch,label_batch])\n        print(cur_example_batch.shape)\n    coord.request_stop()\n    coord.join(thread)",
    "369396": "Hei @Moshel, this is kinda what I came up with, and I am getting serialization errors, how close is this to the code you had?"
  },
  "source": "meta"
}