{
  "id": 186560,
  "title": "How to use tfrecords in pytorch-xla ?",
  "url": "/competitions/lyft-motion-prediction-autonomous-vehicles/discussion/186560",
  "author_name": "",
  "post_date": "2020-09-24T22:43:51.738408600Z",
  "votes": null,
  "comment_count": 1,
  "views": 0,
  "content": "<p>I used tfrecords with tensorfow but i am not able to find the solution with pytorch-xla. Any reference and code samples ?</p>",
  "messages": [
    {
      "id": "1025942",
      "postDate": "09/24/2020 22:43:51",
      "content": "<p>I used tfrecords with tensorfow but i am not able to find the solution with pytorch-xla. Any reference and code samples ?</p>",
      "rawMarkdown": "I used tfrecords with tensorfow but i am not able to find the solution with pytorch-xla. Any reference and code samples ?",
      "votes": null
    },
    {
      "id": "1026318",
      "postDate": "09/25/2020 08:16:08",
      "content": "<p>There's <a href=\"https://github.com/pytorch/xla/blob/master/torch_xla/utils/tf_record_reader.py\" target=\"_blank\"><code>torch_xla.utils.TFRecordReader</code></a>.<br>\nBut you generally don't use tfrecords with pytorch_xla so there's not many examples. Unlike with tensorflow, in pytorch_xla you can't load straight from Google Cloud Storage so you don't get the same advantages (in TF the TPU reads directly from GCS and decodes the tfrecords). The usual approach is to use a standard PyTorch dataset like you would for GPU.</p>",
      "rawMarkdown": "There's [`torch_xla.utils.TFRecordReader`](https://github.com/pytorch/xla/blob/master/torch_xla/utils/tf_record_reader.py).\nBut you generally don't use tfrecords with pytorch_xla so there's not many examples. Unlike with tensorflow, in pytorch_xla you can't load straight from Google Cloud Storage so you don't get the same advantages (in TF the TPU reads directly from GCS and decodes the tfrecords). The usual approach is to use a standard PyTorch dataset like you would for GPU.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1026318,
      "author_name": "thomasbrandon",
      "author_url": "",
      "post_date": "09/25/2020 08:16:08",
      "content": "<p>There's <a href=\"https://github.com/pytorch/xla/blob/master/torch_xla/utils/tf_record_reader.py\" target=\"_blank\"><code>torch_xla.utils.TFRecordReader</code></a>.<br>\nBut you generally don't use tfrecords with pytorch_xla so there's not many examples. Unlike with tensorflow, in pytorch_xla you can't load straight from Google Cloud Storage so you don't get the same advantages (in TF the TPU reads directly from GCS and decodes the tfrecords). The usual approach is to use a standard PyTorch dataset like you would for GPU.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1025942": "I used tfrecords with tensorfow but i am not able to find the solution with pytorch-xla. Any reference and code samples ?",
    "1026318": "There's [`torch_xla.utils.TFRecordReader`](https://github.com/pytorch/xla/blob/master/torch_xla/utils/tf_record_reader.py).\nBut you generally don't use tfrecords with pytorch_xla so there's not many examples. Unlike with tensorflow, in pytorch_xla you can't load straight from Google Cloud Storage so you don't get the same advantages (in TF the TPU reads directly from GCS and decodes the tfrecords). The usual approach is to use a standard PyTorch dataset like you would for GPU."
  },
  "source": "meta"
}