{
  "id": 142113,
  "title": "Data Pipeline with TensorFlow it to simple",
  "url": "/competitions/flower-classification-with-tpus/discussion/142113",
  "author_name": "",
  "post_date": "2020-04-09T00:56:06.383302100Z",
  "votes": null,
  "comment_count": 1,
  "views": 0,
  "content": "<p>The tf.data API enables you to build complex input pipelines\nit use the concept of extract, transform, load (ETL)</p>\n\n<p>`</p>\n\n<h1>**Extract **the data set</h1>\n\n<p>dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO) </p>\n\n<h1>**Transform **and preprocess</h1>\n\n<p>dataset = dataset.map(data_augment, num_parallel_calls=AUTO)</p>\n\n<h1>the training dataset must repeat for several epochs</h1>\n\n<p>dataset = dataset.repeat() </p>\n\n<h1>shuffle to get a random mixture of sample</h1>\n\n<p>dataset = dataset.shuffle(1777)</p>\n\n<h1>**Load **theme Put them in bundles(batch)</h1>\n\n<p>dataset = dataset.batch(BATCH_SIZE)</p>\n\n<h1>prefetch next batch while training (autotune prefetch buffer size)</h1>\n\n<p>dataset = dataset.prefetch(AUTO) </p>\n\n<h1>prefetch next batch while training (autotune prefetch buffer size)</h1>\n\n<p>`\nsome code was ignored for the purpose of simplicity</p>",
  "messages": [
    {
      "id": "801948",
      "postDate": "04/09/2020 00:56:06",
      "content": "<p>The tf.data API enables you to build complex input pipelines\nit use the concept of extract, transform, load (ETL)</p>\n\n<p>`</p>\n\n<h1>**Extract **the data set</h1>\n\n<p>dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO) </p>\n\n<h1>**Transform **and preprocess</h1>\n\n<p>dataset = dataset.map(data_augment, num_parallel_calls=AUTO)</p>\n\n<h1>the training dataset must repeat for several epochs</h1>\n\n<p>dataset = dataset.repeat() </p>\n\n<h1>shuffle to get a random mixture of sample</h1>\n\n<p>dataset = dataset.shuffle(1777)</p>\n\n<h1>**Load **theme Put them in bundles(batch)</h1>\n\n<p>dataset = dataset.batch(BATCH_SIZE)</p>\n\n<h1>prefetch next batch while training (autotune prefetch buffer size)</h1>\n\n<p>dataset = dataset.prefetch(AUTO) </p>\n\n<h1>prefetch next batch while training (autotune prefetch buffer size)</h1>\n\n<p>`\nsome code was ignored for the purpose of simplicity</p>",
      "rawMarkdown": "The tf.data API enables you to build complex input pipelines\nit use the concept of extract, transform, load (ETL)\n\n`\n#**Extract **the data set\ndataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO) \n\n#**Transform **and preprocess\ndataset = dataset.map(data_augment, num_parallel_calls=AUTO)\n# the training dataset must repeat for several epochs\ndataset = dataset.repeat() \n# shuffle to get a random mixture of sample\ndataset = dataset.shuffle(1777)\n\n#**Load **theme Put them in bundles(batch)\ndataset = dataset.batch(BATCH_SIZE)\n# prefetch next batch while training (autotune prefetch buffer size)\ndataset = dataset.prefetch(AUTO) \n# prefetch next batch while training (autotune prefetch buffer size)\n`\nsome code was ignored for the purpose of simplicity",
      "votes": null
    },
    {
      "id": "802035",
      "postDate": "04/09/2020 04:37:47",
      "content": "<p>Thanks for this. I really find it to be so easy to train. And TPUs are awsome!</p>",
      "rawMarkdown": "Thanks for this. I really find it to be so easy to train. And TPUs are awsome!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 802035,
      "author_name": "parmarsuraj99",
      "author_url": "",
      "post_date": "04/09/2020 04:37:47",
      "content": "<p>Thanks for this. I really find it to be so easy to train. And TPUs are awsome!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "801948": "The tf.data API enables you to build complex input pipelines\nit use the concept of extract, transform, load (ETL)\n\n`\n#**Extract **the data set\ndataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO) \n\n#**Transform **and preprocess\ndataset = dataset.map(data_augment, num_parallel_calls=AUTO)\n# the training dataset must repeat for several epochs\ndataset = dataset.repeat() \n# shuffle to get a random mixture of sample\ndataset = dataset.shuffle(1777)\n\n#**Load **theme Put them in bundles(batch)\ndataset = dataset.batch(BATCH_SIZE)\n# prefetch next batch while training (autotune prefetch buffer size)\ndataset = dataset.prefetch(AUTO) \n# prefetch next batch while training (autotune prefetch buffer size)\n`\nsome code was ignored for the purpose of simplicity",
    "802035": "Thanks for this. I really find it to be so easy to train. And TPUs are awsome!"
  },
  "source": "meta"
}