{
  "id": 139287,
  "title": "DataLossError",
  "url": "/competitions/flower-classification-with-tpus/discussion/139287",
  "author_name": "",
  "post_date": "2020-03-28T06:49:59.667996300Z",
  "votes": 1,
  "comment_count": 3,
  "views": 0,
  "content": "<p>I have created a tfrec dataset which the data is divided by different fold, which you can see in my personal dataset section.\nBut after I change the dataset into this new created one, I am not able to train the model.\nOnce I train the model, the notebook give me following error. Did anyone meet same issue as mine?</p>\n\n<p>Thanks in advances\n```</p>\n\n<p>DataLossError: {{function_node __inference_distributed_function_296927}} truncated record at 796638\n     [[{{node MultiDeviceIteratorGetNextFromShard}}]]\n     [[RemoteCall]]\n     [[IteratorGetNextAsOptional]]\n```</p>\n\n<h2>Update</h2>\n\n<p>Please refer the comments for solution of my case.</p>",
  "messages": [
    {
      "id": "788931",
      "postDate": "03/28/2020 06:49:59",
      "content": "<p>I have created a tfrec dataset which the data is divided by different fold, which you can see in my personal dataset section.\nBut after I change the dataset into this new created one, I am not able to train the model.\nOnce I train the model, the notebook give me following error. Did anyone meet same issue as mine?</p>\n\n<p>Thanks in advances\n```</p>\n\n<p>DataLossError: {{function_node __inference_distributed_function_296927}} truncated record at 796638\n     [[{{node MultiDeviceIteratorGetNextFromShard}}]]\n     [[RemoteCall]]\n     [[IteratorGetNextAsOptional]]\n```</p>\n\n<h2>Update</h2>\n\n<p>Please refer the comments for solution of my case.</p>",
      "rawMarkdown": "I have created a tfrec dataset which the data is divided by different fold, which you can see in my personal dataset section.\nBut after I change the dataset into this new created one, I am not able to train the model.\nOnce I train the model, the notebook give me following error. Did anyone meet same issue as mine?\n\nThanks in advances\n```\n\nDataLossError: {{function_node __inference_distributed_function_296927}} truncated record at 796638\n\t [[{{node MultiDeviceIteratorGetNextFromShard}}]]\n\t [[RemoteCall]]\n\t [[IteratorGetNextAsOptional]]\n```\n\nUpdate\n-------------------------------------------------------------------------------\nPlease refer the comments for solution of my case.",
      "votes": null
    },
    {
      "id": "791961",
      "postDate": "03/30/2020 20:01:28",
      "content": "<p>The error say \"truncated record\". This looks like a file integrity problem.</p>",
      "rawMarkdown": "The error say \"truncated record\". This looks like a file integrity problem.",
      "votes": null
    },
    {
      "id": "792259",
      "postDate": "03/31/2020 03:20:57",
      "content": "<p>Yes, I think you are right. I also found a related information about <a href=\"https://www.tensorflow.org/api_docs/python/tf/errors/DataLossError?hl=zh-TW\">DataLossError</a>. But the weird part is, I already created the customize tfrec files before, and it worked just fine. So I'm still searching the differences between last time and this time. </p>",
      "rawMarkdown": "Yes, I think you are right. I also found a related information about [DataLossError](https://www.tensorflow.org/api_docs/python/tf/errors/DataLossError?hl=zh-TW). But the weird part is, I already created the customize tfrec files before, and it worked just fine. So I'm still searching the differences between last time and this time.",
      "votes": null
    },
    {
      "id": "793699",
      "postDate": "04/01/2020 08:04:16",
      "content": "<p>The problem is finally solved! \nI used to create several tfrecord writers for different fold of data. Each fold with one writer.\nAlthough I don't understand why this will cause the DataLossError.. But after I change to use one writer for all the data, the problem is solved.\nThe change is pretty much like:</p>\n\n<p>Old version:\n```\nrecord_filenames = [(\"val%d_%.3d.tfrec\" % (fold, record_file_num)) for fold, record_file_num in zip(range(5), record_file_nums)]</p>\n\n<p>writers = [tf.io.TFRecordWriter(target_path + record_filename) for record_filename in record_filenames]\n```</p>\n\n<p>New version:\n<code>\nfor fold in range(5):\n        record_filename = \"val%d_%.3d.tfrec\" %(fold, record_file_num)\n        writer = tf.io.TFRecordWriter(target_path + record_filename)\n</code></p>",
      "rawMarkdown": "The problem is finally solved! \nI used to create several tfrecord writers for different fold of data. Each fold with one writer.\nAlthough I don't understand why this will cause the DataLossError.. But after I change to use one writer for all the data, the problem is solved.\nThe change is pretty much like:\n\n\nOld version:\n```\nrecord_filenames = [(\"val%d_%.3d.tfrec\" % (fold, record_file_num)) for fold, record_file_num in zip(range(5), record_file_nums)]\n\nwriters = [tf.io.TFRecordWriter(target_path + record_filename) for record_filename in record_filenames]\n```\n\n\nNew version:\n```\nfor fold in range(5):\n        record_filename = \"val%d_%.3d.tfrec\" %(fold, record_file_num)\n        writer = tf.io.TFRecordWriter(target_path + record_filename)\n```",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 791961,
      "author_name": "mgorner",
      "author_url": "",
      "post_date": "03/30/2020 20:01:28",
      "content": "<p>The error say \"truncated record\". This looks like a file integrity problem.</p>",
      "votes": null,
      "replies": [
        {
          "id": 792259,
          "author_name": "xiejialun",
          "author_url": "",
          "post_date": "03/31/2020 03:20:57",
          "content": "<p>Yes, I think you are right. I also found a related information about <a href=\"https://www.tensorflow.org/api_docs/python/tf/errors/DataLossError?hl=zh-TW\">DataLossError</a>. But the weird part is, I already created the customize tfrec files before, and it worked just fine. So I'm still searching the differences between last time and this time. </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 793699,
          "author_name": "xiejialun",
          "author_url": "",
          "post_date": "04/01/2020 08:04:16",
          "content": "<p>The problem is finally solved! \nI used to create several tfrecord writers for different fold of data. Each fold with one writer.\nAlthough I don't understand why this will cause the DataLossError.. But after I change to use one writer for all the data, the problem is solved.\nThe change is pretty much like:</p>\n\n<p>Old version:\n```\nrecord_filenames = [(\"val%d_%.3d.tfrec\" % (fold, record_file_num)) for fold, record_file_num in zip(range(5), record_file_nums)]</p>\n\n<p>writers = [tf.io.TFRecordWriter(target_path + record_filename) for record_filename in record_filenames]\n```</p>\n\n<p>New version:\n<code>\nfor fold in range(5):\n        record_filename = \"val%d_%.3d.tfrec\" %(fold, record_file_num)\n        writer = tf.io.TFRecordWriter(target_path + record_filename)\n</code></p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "788931": "I have created a tfrec dataset which the data is divided by different fold, which you can see in my personal dataset section.\nBut after I change the dataset into this new created one, I am not able to train the model.\nOnce I train the model, the notebook give me following error. Did anyone meet same issue as mine?\n\nThanks in advances\n```\n\nDataLossError: {{function_node __inference_distributed_function_296927}} truncated record at 796638\n\t [[{{node MultiDeviceIteratorGetNextFromShard}}]]\n\t [[RemoteCall]]\n\t [[IteratorGetNextAsOptional]]\n```\n\nUpdate\n-------------------------------------------------------------------------------\nPlease refer the comments for solution of my case.",
    "791961": "The error say \"truncated record\". This looks like a file integrity problem.",
    "792259": "Yes, I think you are right. I also found a related information about [DataLossError](https://www.tensorflow.org/api_docs/python/tf/errors/DataLossError?hl=zh-TW). But the weird part is, I already created the customize tfrec files before, and it worked just fine. So I'm still searching the differences between last time and this time.",
    "793699": "The problem is finally solved! \nI used to create several tfrecord writers for different fold of data. Each fold with one writer.\nAlthough I don't understand why this will cause the DataLossError.. But after I change to use one writer for all the data, the problem is solved.\nThe change is pretty much like:\n\n\nOld version:\n```\nrecord_filenames = [(\"val%d_%.3d.tfrec\" % (fold, record_file_num)) for fold, record_file_num in zip(range(5), record_file_nums)]\n\nwriters = [tf.io.TFRecordWriter(target_path + record_filename) for record_filename in record_filenames]\n```\n\n\nNew version:\n```\nfor fold in range(5):\n        record_filename = \"val%d_%.3d.tfrec\" %(fold, record_file_num)\n        writer = tf.io.TFRecordWriter(target_path + record_filename)\n```"
  },
  "source": "meta"
}