{
  "id": 162957,
  "title": "TPU Memory Problem",
  "url": "/competitions/alaska2-image-steganalysis/discussion/162957",
  "author_name": "",
  "post_date": "2020-06-30T15:16:32.408951600Z",
  "votes": 2,
  "comment_count": 5,
  "views": 0,
  "content": "<p>Hi All,</p>\n\n<p>Since it's necessary to cache data in TPU while training to achieve maximum speed, but the total data size is 30G, while TPUv2 has 8G memory, and TPUv3 has 16 memory, how did you guys solve the memory issue if using TPU?</p>\n\n<p>Thanks!</p>",
  "messages": [
    {
      "id": "909413",
      "postDate": "06/30/2020 15:16:32",
      "content": "<p>Hi All,</p>\n\n<p>Since it's necessary to cache data in TPU while training to achieve maximum speed, but the total data size is 30G, while TPUv2 has 8G memory, and TPUv3 has 16 memory, how did you guys solve the memory issue if using TPU?</p>\n\n<p>Thanks!</p>",
      "rawMarkdown": "Hi All,\n\nSince it's necessary to cache data in TPU while training to achieve maximum speed, but the total data size is 30G, while TPUv2 has 8G memory, and TPUv3 has 16 memory, how did you guys solve the memory issue if using TPU?\n\nThanks!",
      "votes": null
    },
    {
      "id": "909452",
      "postDate": "06/30/2020 15:46:51",
      "content": "<p>Check <a href=\"https://codelabs.developers.google.com/codelabs/keras-flowers-tpu/#4\">here</a></p>",
      "rawMarkdown": "Check [here](https://codelabs.developers.google.com/codelabs/keras-flowers-tpu/#4)",
      "votes": null
    },
    {
      "id": "909473",
      "postDate": "06/30/2020 16:02:26",
      "content": "<p><a href=\"/wuliaokaola\">@wuliaokaola</a> Thank you! I also saw people mentioning TFRecord. Does that bring the same speed as cache?</p>",
      "rawMarkdown": "wuliaokaola Thank you! I also saw people mentioning TFRecord. Does that bring the same speed as cache?",
      "votes": null
    },
    {
      "id": "909484",
      "postDate": "06/30/2020 16:08:51",
      "content": "<p>By using TFRecord, you can train one epoch in about 20 mins. </p>",
      "rawMarkdown": "By using TFRecord, you can train one epoch in about 20 mins.",
      "votes": null
    },
    {
      "id": "910609",
      "postDate": "07/01/2020 08:49:57",
      "content": "<p>True, but how would you get proper TF record with mixed kind of images? The question is TF records holds more than  200 images at the same time. So, how would we decide which image ID to be placed in which record and so on? </p>",
      "rawMarkdown": "True, but how would you get proper TF record with mixed kind of images? The question is TF records holds more than  200 images at the same time. So, how would we decide which image ID to be placed in which record and so on?",
      "votes": null
    },
    {
      "id": "915503",
      "postDate": "07/04/2020 19:31:27",
      "content": "<p>you can cache the bits of the image instead of the decoded 512x512x3 array.</p>",
      "rawMarkdown": "you can cache the bits of the image instead of the decoded 512x512x3 array.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 909452,
      "author_name": "wuliaokaola",
      "author_url": "",
      "post_date": "06/30/2020 15:46:51",
      "content": "<p>Check <a href=\"https://codelabs.developers.google.com/codelabs/keras-flowers-tpu/#4\">here</a></p>",
      "votes": null,
      "replies": [
        {
          "id": 909473,
          "author_name": "tonychenxyz",
          "author_url": "",
          "post_date": "06/30/2020 16:02:26",
          "content": "<p><a href=\"/wuliaokaola\">@wuliaokaola</a> Thank you! I also saw people mentioning TFRecord. Does that bring the same speed as cache?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 909484,
          "author_name": "wuliaokaola",
          "author_url": "",
          "post_date": "06/30/2020 16:08:51",
          "content": "<p>By using TFRecord, you can train one epoch in about 20 mins. </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 910609,
          "author_name": "urvishp80",
          "author_url": "",
          "post_date": "07/01/2020 08:49:57",
          "content": "<p>True, but how would you get proper TF record with mixed kind of images? The question is TF records holds more than  200 images at the same time. So, how would we decide which image ID to be placed in which record and so on? </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 915503,
      "author_name": "hooong",
      "author_url": "",
      "post_date": "07/04/2020 19:31:27",
      "content": "<p>you can cache the bits of the image instead of the decoded 512x512x3 array.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "909413": "Hi All,\n\nSince it's necessary to cache data in TPU while training to achieve maximum speed, but the total data size is 30G, while TPUv2 has 8G memory, and TPUv3 has 16 memory, how did you guys solve the memory issue if using TPU?\n\nThanks!",
    "909452": "Check [here](https://codelabs.developers.google.com/codelabs/keras-flowers-tpu/#4)",
    "909473": "wuliaokaola Thank you! I also saw people mentioning TFRecord. Does that bring the same speed as cache?",
    "909484": "By using TFRecord, you can train one epoch in about 20 mins.",
    "910609": "True, but how would you get proper TF record with mixed kind of images? The question is TF records holds more than  200 images at the same time. So, how would we decide which image ID to be placed in which record and so on?",
    "915503": "you can cache the bits of the image instead of the decoded 512x512x3 array."
  },
  "source": "meta"
}