{
  "id": 394740,
  "title": "Saving and Loading IceCube Data as TFRecord",
  "url": "/competitions/icecube-neutrinos-in-deep-ice/discussion/394740",
  "author_name": "",
  "post_date": "2023-03-14T17:36:18.445333500Z",
  "votes": 3,
  "comment_count": 2,
  "views": 0,
  "content": "<p>As I have been working on this competition it has been super useful to have a TFRecord dataset to store all of the event data because the data is loaded on demand which lets me work with much more data than can fit in memory. I thought I would share my code for creating and loading such TFRecords.</p>\n<p>See <a href=\"https://www.kaggle.com/datasets/tatelarkin/icecube-tfrecord\" target=\"_blank\">https://www.kaggle.com/datasets/tatelarkin/icecube-tfrecord</a> for 10 pre-created TFRecords corresponding to 10 batches, and see <a href=\"https://www.kaggle.com/code/tatelarkin/saving-and-loading-icecube-data-as-tfrecord\" target=\"_blank\">https://www.kaggle.com/code/tatelarkin/saving-and-loading-icecube-data-as-tfrecord</a> for the code.</p>\n<p>I hope this helps people working with Tensorflow models, or people who want a faster data pipeline 😊</p>",
  "messages": [
    {
      "id": "2181686",
      "postDate": "03/14/2023 17:36:18",
      "content": "<p>As I have been working on this competition it has been super useful to have a TFRecord dataset to store all of the event data because the data is loaded on demand which lets me work with much more data than can fit in memory. I thought I would share my code for creating and loading such TFRecords.</p>\n<p>See <a href=\"https://www.kaggle.com/datasets/tatelarkin/icecube-tfrecord\" target=\"_blank\">https://www.kaggle.com/datasets/tatelarkin/icecube-tfrecord</a> for 10 pre-created TFRecords corresponding to 10 batches, and see <a href=\"https://www.kaggle.com/code/tatelarkin/saving-and-loading-icecube-data-as-tfrecord\" target=\"_blank\">https://www.kaggle.com/code/tatelarkin/saving-and-loading-icecube-data-as-tfrecord</a> for the code.</p>\n<p>I hope this helps people working with Tensorflow models, or people who want a faster data pipeline 😊</p>",
      "rawMarkdown": "As I have been working on this competition it has been super useful to have a TFRecord dataset to store all of the event data because the data is loaded on demand which lets me work with much more data than can fit in memory. I thought I would share my code for creating and loading such TFRecords.\n\nSee https://www.kaggle.com/datasets/tatelarkin/icecube-tfrecord for 10 pre-created TFRecords corresponding to 10 batches, and see https://www.kaggle.com/code/tatelarkin/saving-and-loading-icecube-data-as-tfrecord for the code.\n\nI hope this helps people working with Tensorflow models, or people who want a faster data pipeline 😊",
      "votes": null
    },
    {
      "id": "2182101",
      "postDate": "03/15/2023 00:12:59",
      "content": "<p>404 error for your first link.</p>",
      "rawMarkdown": "404 error for your first link.",
      "votes": null
    },
    {
      "id": "2182107",
      "postDate": "03/15/2023 00:16:56",
      "content": "<p>Thank you for letting me know. Its fixed!</p>",
      "rawMarkdown": "Thank you for letting me know. Its fixed!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2182101,
      "author_name": "pcjimmmy",
      "author_url": "",
      "post_date": "03/15/2023 00:12:59",
      "content": "<p>404 error for your first link.</p>",
      "votes": null,
      "replies": [
        {
          "id": 2182107,
          "author_name": "tatelarkin",
          "author_url": "",
          "post_date": "03/15/2023 00:16:56",
          "content": "<p>Thank you for letting me know. Its fixed!</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2181686": "As I have been working on this competition it has been super useful to have a TFRecord dataset to store all of the event data because the data is loaded on demand which lets me work with much more data than can fit in memory. I thought I would share my code for creating and loading such TFRecords.\n\nSee https://www.kaggle.com/datasets/tatelarkin/icecube-tfrecord for 10 pre-created TFRecords corresponding to 10 batches, and see https://www.kaggle.com/code/tatelarkin/saving-and-loading-icecube-data-as-tfrecord for the code.\n\nI hope this helps people working with Tensorflow models, or people who want a faster data pipeline 😊",
    "2182101": "404 error for your first link.",
    "2182107": "Thank you for letting me know. Its fixed!"
  },
  "source": "meta"
}