{
  "id": 214844,
  "title": "Fast pipeline resources for TFRecords+TPUs+Pytorch",
  "url": "/competitions/hpa-single-cell-image-classification/discussion/214844",
  "author_name": "JohnM",
  "post_date": "2021-01-27T20:12:34.547000",
  "votes": 9,
  "comment_count": 5,
  "views": 0,
  "content": "<p>Here are some resources I found while exploring the use of Pytorch with TPUs and TFRecords. Edit: XLA can do it all, see comment below.</p>\n<ul>\n<li><a href=\"https://docs.nvidia.com/deeplearning/dali/master-user-guide/docs/examples/general/data_loading/dataloading_tfrecord.html\" target=\"_blank\">Nvidia DALI</a>  to load TFRecords for PyTorch. </li>\n<li><a href=\"https://github.com/vahidk/tfrecord\" target=\"_blank\">TFRecord Reader</a> also</li>\n<li><a href=\"https://pytorch.org/xla/release/1.7/index.html\" target=\"_blank\">PyTorch XLA</a> to run on a TPU. </li>\n</ul>\n<p>There are also many notebooks on Kaggle though I haven't seen one end-to-end. They may help address any Kaggle-specific issues.</p>",
  "messages": [
    {
      "id": 1173349,
      "postDate": "2021-01-27T20:12:34.547Z",
      "content": "<p>Here are some resources I found while exploring the use of Pytorch with TPUs and TFRecords. Edit: XLA can do it all, see comment below.</p>\n<ul>\n<li><a href=\"https://docs.nvidia.com/deeplearning/dali/master-user-guide/docs/examples/general/data_loading/dataloading_tfrecord.html\" target=\"_blank\">Nvidia DALI</a>  to load TFRecords for PyTorch. </li>\n<li><a href=\"https://github.com/vahidk/tfrecord\" target=\"_blank\">TFRecord Reader</a> also</li>\n<li><a href=\"https://pytorch.org/xla/release/1.7/index.html\" target=\"_blank\">PyTorch XLA</a> to run on a TPU. </li>\n</ul>\n<p>There are also many notebooks on Kaggle though I haven't seen one end-to-end. They may help address any Kaggle-specific issues.</p>",
      "rawMarkdown": "Here are some resources I found while exploring the use of Pytorch with TPUs and TFRecords. Edit: XLA can do it all, see comment below.\n\n - [Nvidia DALI](https://docs.nvidia.com/deeplearning/dali/master-user-guide/docs/examples/general/data_loading/dataloading_tfrecord.html)  to load TFRecords for PyTorch. \n - [TFRecord Reader](https://github.com/vahidk/tfrecord) also\n - [PyTorch XLA](https://pytorch.org/xla/release/1.7/index.html) to run on a TPU. \n\nThere are also many notebooks on Kaggle though I haven't seen one end-to-end. They may help address any Kaggle-specific issues.\n",
      "votes": 9
    },
    {
      "id": 1175063,
      "postDate": "2021-01-28T22:02:49.720Z",
      "content": "<p>PyTorch XLA comes with a TfRecordReader already:<br>\n<a href=\"https://pytorch.org/xla/master/#module-torch_xla.utils.tf_record_reader\" target=\"_blank\">https://pytorch.org/xla/master/#module-torch_xla.utils.tf_record_reader</a></p>",
      "rawMarkdown": "PyTorch XLA comes with a TfRecordReader already:\nhttps://pytorch.org/xla/master/#module-torch_xla.utils.tf_record_reader",
      "votes": 3,
      "replies": [
        {
          "id": 1175217,
          "postDate": "2021-01-29T02:38:22.353Z",
          "content": "<p>Awesome - XLA will do it! <br>\nKagglers, here is a most excellent notebook from <em>ilovescience</em> showing XLA and TPUs: <a href=\"https://www.kaggle.com/tanlikesmath/cassava-pytorch-xla-tpu-starter-training\" target=\"_blank\">https://www.kaggle.com/tanlikesmath/cassava-pytorch-xla-tpu-starter-training</a></p>",
          "rawMarkdown": "Awesome - XLA will do it! \nKagglers, here is a most excellent notebook from *ilovescience* showing XLA and TPUs: https://www.kaggle.com/tanlikesmath/cassava-pytorch-xla-tpu-starter-training"
        }
      ]
    },
    {
      "id": 1173571,
      "postDate": "2021-01-28T01:03:49.857Z",
      "content": "<p>Some months ago, using PyTorch with TPUs was slower than TF 2.x, is that true now? </p>",
      "rawMarkdown": "Some months ago, using PyTorch with TPUs was slower than TF 2.x, is that true now? ",
      "votes": 2,
      "replies": [
        {
          "id": 1173707,
          "postDate": "2021-01-28T04:31:12.933Z",
          "content": "<p>Did you find any recent comparisons? It would not surprise me. TFRecords, TF, and TPUs are all designed by Google to work together.</p>",
          "rawMarkdown": "Did you find any recent comparisons? It would not surprise me. TFRecords, TF, and TPUs are all designed by Google to work together."
        }
      ]
    },
    {
      "id": 1180034,
      "postDate": "2021-02-01T03:52:56.697Z",
      "content": "<p>I'm a fresh man and I want to load the data in TPU. <br>\nAnd do you know what the feathre_dict is when I decode the tfrec file?</p>",
      "rawMarkdown": "I'm a fresh man and I want to load the data in TPU. \nAnd do you know what the feathre_dict is when I decode the tfrec file?"
    }
  ],
  "comments": [
    {
      "id": 1175063,
      "author_name": "ilovescience",
      "author_url": "",
      "post_date": "2021-01-28T22:02:49.720000",
      "content": "<p>PyTorch XLA comes with a TfRecordReader already:<br>\n<a href=\"https://pytorch.org/xla/master/#module-torch_xla.utils.tf_record_reader\" target=\"_blank\">https://pytorch.org/xla/master/#module-torch_xla.utils.tf_record_reader</a></p>",
      "votes": 3,
      "replies": [
        {
          "id": 1175217,
          "author_name": "JohnM",
          "author_url": "",
          "post_date": "2021-01-29T02:38:22.353000",
          "content": "<p>Awesome - XLA will do it! <br>\nKagglers, here is a most excellent notebook from <em>ilovescience</em> showing XLA and TPUs: <a href=\"https://www.kaggle.com/tanlikesmath/cassava-pytorch-xla-tpu-starter-training\" target=\"_blank\">https://www.kaggle.com/tanlikesmath/cassava-pytorch-xla-tpu-starter-training</a></p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1173571,
      "author_name": "Hiram Coria 🧬",
      "author_url": "",
      "post_date": "2021-01-28T01:03:49.857000",
      "content": "<p>Some months ago, using PyTorch with TPUs was slower than TF 2.x, is that true now? </p>",
      "votes": 2,
      "replies": [
        {
          "id": 1173707,
          "author_name": "JohnM",
          "author_url": "",
          "post_date": "2021-01-28T04:31:12.933000",
          "content": "<p>Did you find any recent comparisons? It would not surprise me. TFRecords, TF, and TPUs are all designed by Google to work together.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1180034,
      "author_name": "Yifan Wu",
      "author_url": "",
      "post_date": "2021-02-01T03:52:56.697000",
      "content": "<p>I'm a fresh man and I want to load the data in TPU. <br>\nAnd do you know what the feathre_dict is when I decode the tfrec file?</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1173349": "Here are some resources I found while exploring the use of Pytorch with TPUs and TFRecords. Edit: XLA can do it all, see comment below.\n\n - [Nvidia DALI](https://docs.nvidia.com/deeplearning/dali/master-user-guide/docs/examples/general/data_loading/dataloading_tfrecord.html)  to load TFRecords for PyTorch. \n - [TFRecord Reader](https://github.com/vahidk/tfrecord) also\n - [PyTorch XLA](https://pytorch.org/xla/release/1.7/index.html) to run on a TPU. \n\nThere are also many notebooks on Kaggle though I haven't seen one end-to-end. They may help address any Kaggle-specific issues.\n",
    "1175063": "PyTorch XLA comes with a TfRecordReader already:\nhttps://pytorch.org/xla/master/#module-torch_xla.utils.tf_record_reader",
    "1173571": "Some months ago, using PyTorch with TPUs was slower than TF 2.x, is that true now? ",
    "1180034": "I'm a fresh man and I want to load the data in TPU. \nAnd do you know what the feathre_dict is when I decode the tfrec file?"
  }
}