{
  "id": 217268,
  "title": "Please tell me about the data order of TFRecords.",
  "url": "/competitions/hpa-single-cell-image-classification/discussion/217268",
  "author_name": "K.Amano",
  "post_date": "2021-02-06T05:48:48.183000",
  "votes": 2,
  "comment_count": 3,
  "views": 0,
  "content": "<p>The data of TFRecords expected that 4 image data of R, G, B, Y are continuous for each \"image_name\", but it seems that is not the case.</p>\n<pre><code>train_files = tf.io.gfile.glob(TRAIN_TFREC + '/*.tfrec')\n...\ntrain_ds = load_dataset(train_files, ...\nimages_name, images, labels = next(iter(train_ds))\nprint(images_name)\n</code></pre>\n<p>The output is first three '4f73 …', followed by four 'a166 …' …</p>\n<pre><code>tf.Tensor(\n[b'4f737954-bbba-11e8-b2ba-ac1f6b6435d0_red'\n b'4f737954-bbba-11e8-b2ba-ac1f6b6435d0_green'\n b'4f737954-bbba-11e8-b2ba-ac1f6b6435d0_yellow'\n b'a166d11a-bbca-11e8-b2bc-ac1f6b6435d0_red'\n b'a166d11a-bbca-11e8-b2bc-ac1f6b6435d0_green'\n b'a166d11a-bbca-11e8-b2bc-ac1f6b6435d0_yellow'\n b'a166d11a-bbca-11e8-b2bc-ac1f6b6435d0_blue'\n b'd5076d80-bbbc-11e8-b2ba-ac1f6b6435d0_blue'\n...\n</code></pre>\n<p>Is the data order of TFRecords correct? Is it because I'm new to TFRecords?</p>",
  "messages": [
    {
      "id": 1188880,
      "postDate": "2021-02-06T15:00:55.573Z",
      "content": "<p>In the <strong><code>tf.data.TFRecordDataset</code></strong> call did you set the following <strong><code>num_parallel_calls=None</code></strong>?</p>\n<p>If not, they will be out of order (the tfrecord files), and some tfrecord files start in the middle of an image's 4-channels. See <strong><a href=\"https://www.kaggle.com/dschettler8845/hpa-xai-ig-tfrecords-tpu-training\" target=\"_blank\">my notebook here</a></strong> for more details on how I did it.</p>",
      "rawMarkdown": "In the **`tf.data.TFRecordDataset`** call did you set the following **`num_parallel_calls=None`**?\n\nIf not, they will be out of order (the tfrecord files), and some tfrecord files start in the middle of an image's 4-channels. See **[my notebook here](https://www.kaggle.com/dschettler8845/hpa-xai-ig-tfrecords-tpu-training)** for more details on how I did it.",
      "votes": 2,
      "replies": [
        {
          "id": 1188989,
          "postDate": "2021-02-06T16:24:20.917Z",
          "content": "<p>Hi Darien Schettler.</p>\n<p>Thank you for your teaching. <br>\nI had set <code>num_parallel_calls = AUTOTUNE</code>. In AUTOTUNE, it becomes parallel operation (asynchronous), and the order may change.<br>\nThank you very much.</p>",
          "rawMarkdown": "Hi Darien Schettler.\n\nThank you for your teaching. \nI had set `num_parallel_calls = AUTOTUNE`. In AUTOTUNE, it becomes parallel operation (asynchronous), and the order may change.\nThank you very much.",
          "votes": 3
        },
        {
          "id": 1189110,
          "postDate": "2021-02-06T18:10:13.943Z",
          "content": "<p>You’re welcome!</p>",
          "rawMarkdown": "You’re welcome!"
        }
      ]
    },
    {
      "id": 1188295,
      "postDate": "2021-02-06T05:48:48.183Z",
      "content": "<p>The data of TFRecords expected that 4 image data of R, G, B, Y are continuous for each \"image_name\", but it seems that is not the case.</p>\n<pre><code>train_files = tf.io.gfile.glob(TRAIN_TFREC + '/*.tfrec')\n...\ntrain_ds = load_dataset(train_files, ...\nimages_name, images, labels = next(iter(train_ds))\nprint(images_name)\n</code></pre>\n<p>The output is first three '4f73 …', followed by four 'a166 …' …</p>\n<pre><code>tf.Tensor(\n[b'4f737954-bbba-11e8-b2ba-ac1f6b6435d0_red'\n b'4f737954-bbba-11e8-b2ba-ac1f6b6435d0_green'\n b'4f737954-bbba-11e8-b2ba-ac1f6b6435d0_yellow'\n b'a166d11a-bbca-11e8-b2bc-ac1f6b6435d0_red'\n b'a166d11a-bbca-11e8-b2bc-ac1f6b6435d0_green'\n b'a166d11a-bbca-11e8-b2bc-ac1f6b6435d0_yellow'\n b'a166d11a-bbca-11e8-b2bc-ac1f6b6435d0_blue'\n b'd5076d80-bbbc-11e8-b2ba-ac1f6b6435d0_blue'\n...\n</code></pre>\n<p>Is the data order of TFRecords correct? Is it because I'm new to TFRecords?</p>",
      "rawMarkdown": "The data of TFRecords expected that 4 image data of R, G, B, Y are continuous for each \"image_name\", but it seems that is not the case.\n```\ntrain_files = tf.io.gfile.glob(TRAIN_TFREC + '/*.tfrec')\n...\ntrain_ds = load_dataset(train_files, ...\nimages_name, images, labels = next(iter(train_ds))\nprint(images_name)\n```\nThe output is first three '4f73 ...', followed by four 'a166 ...' ...\n```\ntf.Tensor(\n[b'4f737954-bbba-11e8-b2ba-ac1f6b6435d0_red'\n b'4f737954-bbba-11e8-b2ba-ac1f6b6435d0_green'\n b'4f737954-bbba-11e8-b2ba-ac1f6b6435d0_yellow'\n b'a166d11a-bbca-11e8-b2bc-ac1f6b6435d0_red'\n b'a166d11a-bbca-11e8-b2bc-ac1f6b6435d0_green'\n b'a166d11a-bbca-11e8-b2bc-ac1f6b6435d0_yellow'\n b'a166d11a-bbca-11e8-b2bc-ac1f6b6435d0_blue'\n b'd5076d80-bbbc-11e8-b2ba-ac1f6b6435d0_blue'\n...\n```\nIs the data order of TFRecords correct? Is it because I'm new to TFRecords?",
      "votes": 2
    }
  ],
  "comments": [
    {
      "id": 1188880,
      "author_name": "Darien Schettler",
      "author_url": "",
      "post_date": "2021-02-06T15:00:55.573000",
      "content": "<p>In the <strong><code>tf.data.TFRecordDataset</code></strong> call did you set the following <strong><code>num_parallel_calls=None</code></strong>?</p>\n<p>If not, they will be out of order (the tfrecord files), and some tfrecord files start in the middle of an image's 4-channels. See <strong><a href=\"https://www.kaggle.com/dschettler8845/hpa-xai-ig-tfrecords-tpu-training\" target=\"_blank\">my notebook here</a></strong> for more details on how I did it.</p>",
      "votes": 2,
      "replies": [
        {
          "id": 1188989,
          "author_name": "K.Amano",
          "author_url": "",
          "post_date": "2021-02-06T16:24:20.917000",
          "content": "<p>Hi Darien Schettler.</p>\n<p>Thank you for your teaching. <br>\nI had set <code>num_parallel_calls = AUTOTUNE</code>. In AUTOTUNE, it becomes parallel operation (asynchronous), and the order may change.<br>\nThank you very much.</p>",
          "votes": 3,
          "replies": []
        },
        {
          "id": 1189110,
          "author_name": "Darien Schettler",
          "author_url": "",
          "post_date": "2021-02-06T18:10:13.943000",
          "content": "<p>You’re welcome!</p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1188880": "In the **`tf.data.TFRecordDataset`** call did you set the following **`num_parallel_calls=None`**?\n\nIf not, they will be out of order (the tfrecord files), and some tfrecord files start in the middle of an image's 4-channels. See **[my notebook here](https://www.kaggle.com/dschettler8845/hpa-xai-ig-tfrecords-tpu-training)** for more details on how I did it.",
    "1188295": "The data of TFRecords expected that 4 image data of R, G, B, Y are continuous for each \"image_name\", but it seems that is not the case.\n```\ntrain_files = tf.io.gfile.glob(TRAIN_TFREC + '/*.tfrec')\n...\ntrain_ds = load_dataset(train_files, ...\nimages_name, images, labels = next(iter(train_ds))\nprint(images_name)\n```\nThe output is first three '4f73 ...', followed by four 'a166 ...' ...\n```\ntf.Tensor(\n[b'4f737954-bbba-11e8-b2ba-ac1f6b6435d0_red'\n b'4f737954-bbba-11e8-b2ba-ac1f6b6435d0_green'\n b'4f737954-bbba-11e8-b2ba-ac1f6b6435d0_yellow'\n b'a166d11a-bbca-11e8-b2bc-ac1f6b6435d0_red'\n b'a166d11a-bbca-11e8-b2bc-ac1f6b6435d0_green'\n b'a166d11a-bbca-11e8-b2bc-ac1f6b6435d0_yellow'\n b'a166d11a-bbca-11e8-b2bc-ac1f6b6435d0_blue'\n b'd5076d80-bbbc-11e8-b2ba-ac1f6b6435d0_blue'\n...\n```\nIs the data order of TFRecords correct? Is it because I'm new to TFRecords?"
  }
}