{
  "id": 241320,
  "title": "Use this to count number of files in TFRecords",
  "url": "/competitions/tpu-getting-started/discussion/241320",
  "author_name": "Bhargav ram",
  "post_date": "2021-05-24T04:58:47.374000",
  "votes": 0,
  "comment_count": 0,
  "views": null,
  "content": "<p><strong>The below code will give  the number of files in the TFRecords:</strong></p>\n<p><em>TRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train/</em>.tfrec')<br>\nVALIDATION_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/val/<em>.tfrec')</em></p>\n<p><em>TEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/test/</em>.tfrec') *</p>\n<p><em>tf.compat.v1.enable_eager_execution</em></p>\n<p><em>NUM_TRAINING_IMAGES = sum(1 for _ in tf.data.TFRecordDataset(TRAINING_FILENAMES))\nNUM_VALIDATION_IMAGES = sum(1 for _ in tf.data.TFRecordDataset(VALIDATION_FILENAMES))\nNUM_TEST_IMAGES = sum(1 for _ in tf.data.TFRecordDataset(TEST_FILENAMES))</em></p>\n<p><em>print('Dataset: {} training images, {} validation images, {} unlabeled test images'.format(NUM_TRAINING_IMAGES, NUM_VALIDATION_IMAGES, NUM_TEST_IMAGES))</em></p>",
  "messages": [
    {
      "id": 1320440,
      "postDate": "2021-05-24T04:58:47.373Z",
      "content": "<p><strong>The below code will give  the number of files in the TFRecords:</strong></p>\n<p><em>TRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train/</em>.tfrec')<br>\nVALIDATION_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/val/<em>.tfrec')</em></p>\n<p><em>TEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/test/</em>.tfrec') *</p>\n<p><em>tf.compat.v1.enable_eager_execution</em></p>\n<p><em>NUM_TRAINING_IMAGES = sum(1 for _ in tf.data.TFRecordDataset(TRAINING_FILENAMES))\nNUM_VALIDATION_IMAGES = sum(1 for _ in tf.data.TFRecordDataset(VALIDATION_FILENAMES))\nNUM_TEST_IMAGES = sum(1 for _ in tf.data.TFRecordDataset(TEST_FILENAMES))</em></p>\n<p><em>print('Dataset: {} training images, {} validation images, {} unlabeled test images'.format(NUM_TRAINING_IMAGES, NUM_VALIDATION_IMAGES, NUM_TEST_IMAGES))</em></p>",
      "rawMarkdown": "**The below code will give  the number of files in the TFRecords:**\n\n\n*TRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train/*.tfrec')\nVALIDATION_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/val/*.tfrec')*\n\n*TEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/test/*.tfrec') *\n\n*tf.compat.v1.enable_eager_execution*\n\n*NUM_TRAINING_IMAGES = sum(1 for _ in tf.data.TFRecordDataset(TRAINING_FILENAMES))\nNUM_VALIDATION_IMAGES = sum(1 for _ in tf.data.TFRecordDataset(VALIDATION_FILENAMES))\nNUM_TEST_IMAGES = sum(1 for _ in tf.data.TFRecordDataset(TEST_FILENAMES))*\n\n*print('Dataset: {} training images, {} validation images, {} unlabeled test images'.format(NUM_TRAINING_IMAGES, NUM_VALIDATION_IMAGES, NUM_TEST_IMAGES))*"
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "1320440": "**The below code will give  the number of files in the TFRecords:**\n\n\n*TRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train/*.tfrec')\nVALIDATION_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/val/*.tfrec')*\n\n*TEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/test/*.tfrec') *\n\n*tf.compat.v1.enable_eager_execution*\n\n*NUM_TRAINING_IMAGES = sum(1 for _ in tf.data.TFRecordDataset(TRAINING_FILENAMES))\nNUM_VALIDATION_IMAGES = sum(1 for _ in tf.data.TFRecordDataset(VALIDATION_FILENAMES))\nNUM_TEST_IMAGES = sum(1 for _ in tf.data.TFRecordDataset(TEST_FILENAMES))*\n\n*print('Dataset: {} training images, {} validation images, {} unlabeled test images'.format(NUM_TRAINING_IMAGES, NUM_VALIDATION_IMAGES, NUM_TEST_IMAGES))*"
  }
}