{
  "id": 184040,
  "title": "Can we train on TPUs without making a TFRecord?",
  "url": "/competitions/landmark-recognition-2020/discussion/184040",
  "author_name": "",
  "post_date": "2020-09-19T05:59:44.065584200Z",
  "votes": null,
  "comment_count": 1,
  "views": 0,
  "content": "<p>Hey all,</p>\n<p>I have read many of the top leaderboard guys are training on Kaggle/colab TPUs. So are you guys doing it by making tfrecords first and reading the tfrecords during training. Or is there any simpler way to just read from original dataset.</p>\n<p>I tried this <br>\nimage = tf.io.read_file(image_path)<br>\nwhere the image path is some like this<br>\ngs://kds-484b5c4cd446908689d73867f667c13a266bf9c43e20a29dadfd5617/train/2/5/c/25c9dfc7ea69838d.jpg</p>\n<p>But the pipeline is very slow. Is there a better way? Those who are training on TPUs please help.</p>",
  "messages": [
    {
      "id": "1016625",
      "postDate": "09/19/2020 05:59:44",
      "content": "<p>Hey all,</p>\n<p>I have read many of the top leaderboard guys are training on Kaggle/colab TPUs. So are you guys doing it by making tfrecords first and reading the tfrecords during training. Or is there any simpler way to just read from original dataset.</p>\n<p>I tried this <br>\nimage = tf.io.read_file(image_path)<br>\nwhere the image path is some like this<br>\ngs://kds-484b5c4cd446908689d73867f667c13a266bf9c43e20a29dadfd5617/train/2/5/c/25c9dfc7ea69838d.jpg</p>\n<p>But the pipeline is very slow. Is there a better way? Those who are training on TPUs please help.</p>",
      "rawMarkdown": "Hey all,\n\nI have read many of the top leaderboard guys are training on Kaggle/colab TPUs. So are you guys doing it by making tfrecords first and reading the tfrecords during training. Or is there any simpler way to just read from original dataset.\n\nI tried this \nimage = tf.io.read_file(image_path)\nwhere the image path is some like this\ngs://kds-484b5c4cd446908689d73867f667c13a266bf9c43e20a29dadfd5617/train/2/5/c/25c9dfc7ea69838d.jpg\n\nBut the pipeline is very slow. Is there a better way? Those who are training on TPUs please help.",
      "votes": null
    },
    {
      "id": "1018157",
      "postDate": "09/19/2020 13:29:02",
      "content": "<p>Of course you can train on TPU without tfrecords, especially if input data can be fast delivered to model. But I think in image processing it is worth spend time and prepare data in tfrecords, you will save so much time in the future</p>",
      "rawMarkdown": "Of course you can train on TPU without tfrecords, especially if input data can be fast delivered to model. But I think in image processing it is worth spend time and prepare data in tfrecords, you will save so much time in the future",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1018157,
      "author_name": "aybatov",
      "author_url": "",
      "post_date": "09/19/2020 13:29:02",
      "content": "<p>Of course you can train on TPU without tfrecords, especially if input data can be fast delivered to model. But I think in image processing it is worth spend time and prepare data in tfrecords, you will save so much time in the future</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1016625": "Hey all,\n\nI have read many of the top leaderboard guys are training on Kaggle/colab TPUs. So are you guys doing it by making tfrecords first and reading the tfrecords during training. Or is there any simpler way to just read from original dataset.\n\nI tried this \nimage = tf.io.read_file(image_path)\nwhere the image path is some like this\ngs://kds-484b5c4cd446908689d73867f667c13a266bf9c43e20a29dadfd5617/train/2/5/c/25c9dfc7ea69838d.jpg\n\nBut the pipeline is very slow. Is there a better way? Those who are training on TPUs please help.",
    "1018157": "Of course you can train on TPU without tfrecords, especially if input data can be fast delivered to model. But I think in image processing it is worth spend time and prepare data in tfrecords, you will save so much time in the future"
  },
  "source": "meta"
}