{
  "id": 207342,
  "title": "why errors such as \"RandomShuffleQueue '_1_shuffle_batch/random_shuffle_queue' is closed and has insufficient elements\" always occurred",
  "url": "/competitions/cassava-leaf-disease-classification/discussion/207342",
  "author_name": "",
  "post_date": "2020-12-29T09:40:41.450329Z",
  "votes": null,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I try to load tfrecord files by using the following code:</p>\n<p>` <br>\nimport os<br>\nimport sys<br>\nimport logging<br>\nimport random<br>\nimport json<br>\nimport numpy as np<br>\nimport tensorflow as tf<br>\nimport cv2<br>\nimport PIL<br>\nimport pathlib<br>\nfrom pathlib import Path<br>\nfrom PIL import Image<br>\nimport gflags<br>\nfrom gflags import *</p>\n<p>def read_and_decode(filename_queue):<br>\n    reader = tf.TFRecordReader()<br>\n    _, example = reader.read(filename_queue)</p>\n<pre><code>feature_dict = {}\nfeature_dict[\"image\"] = tf.FixedLenFeature([], tf.string)\nfeature_dict[\"image_name\"] = tf.FixedLenFeature([], tf.string)\nfeature_dict[\"target\"] = tf.FixedLenFeature([], tf.int64)\n\nparsed_example = tf.parse_single_example(example, features=feature_dict)\n\nimage = tf.decode_raw(parsed_example[\"image\"], tf.float32)\nimage = tf.reshape(image, [192, 192, 3])\nlabel = tf.cast(parsed_example[\"target\"], tf.int32)\nname = tf.cast(parsed_example[\"image_name\"], tf.string)\nreturn image, name, label\n</code></pre>\n<p>def gen_batch(paths, examples_num, batch_size, shuffle):<br>\n    filename_queue = tf.train.string_input_producer(paths, num_epochs=2)<br>\n    print(filename_queue.size())<br>\n    image, name, label = read_and_decode(filename_queue)<br>\n    min_fraction_of_examples_in_queue = 0.4<br>\n    min_queue_examples = int(min_fraction_of_examples_in_queue * examples_num)<br>\n    min_queue_examples = 500<br>\n    num_process_threads = 4<br>\n    images, names, labels = tf.train.shuffle_batch([image, name, label],\\<br>\n                batch_size=batch_size,\\<br>\n                num_threads=num_process_threads,\\<br>\n                #capacity=min_queue_examples + batch_size * 3,\\<br>\n                capacity=1000,\\<br>\n                min_after_dequeue=100)<br>\n    return images, names, labels</p>\n<p>def main():<br>\n    root = \"/opt/program/services/project/cassava-leaf-disease-classification/\"</p>\n<pre><code># train images num\nimage_path = Path('%strain_images/' % root)\nimage_count = len(list(image_path.glob('*.jpg')))\n\npaths = ['0.tfrecords', '1.tfrecords', '2.tfrecords']\nimages, names, labels = gen_batch(paths, image_count, 1, True)\n\nmodel_path = \"%s/model/leaf-disease.model\" % root\nsaver = tf.train.Saver()\nwith tf.Session() as sess:\n    sess.run(tf.global_variables_initializer())\n    sess.run(tf.local_variables_initializer())\n\n    coord = tf.train.Coordinator()\n    threads = tf.train.start_queue_runners(coord=coord)\n\n    for i in range(3):\n        print(\"i = %d\" % i)\n        image_list= sess.run([images])\n        if i % 100 == 0:\n            print('step: {}, loss: {}'.format(i, train_loss))\n            saver.save(sess, model_path, global_step=i)\n    coord.request_stop()\n    coord.join(threads)\n</code></pre>\n<p>if <strong>name</strong> == '<strong>main</strong>':<br>\n    main()`</p>\n<p><strong>Note: '0.tfrecords', '1.tfrecords', '2.tfrecords' locate in same path with my code</strong></p>\n<p>but when runing,  error occured as following:</p>\n<p><strong>OutOfRangeError (see above for traceback): RandomShuffleQueue '_1_shuffle_batch/random_shuffle_queue' is closed and has insufficient elements (requested 1, current size 0) [[Node: shuffle_batch = QueueDequeueManyV2[component_types=[DT_FLOAT, DT_STRING, DT_INT32], timeout_ms=-1, _device=\"/job:localhost/replica:0/task:0/device:CPU:0\"](shuffle_batch/random_shuffle_queue, shuffle_batch/n)]]</strong></p>\n<p>I have try to check my code Carefully，but can't location the problem.</p>",
  "messages": [
    {
      "id": "1130749",
      "postDate": "12/29/2020 09:40:41",
      "content": "<p>I try to load tfrecord files by using the following code:</p>\n<p>` <br>\nimport os<br>\nimport sys<br>\nimport logging<br>\nimport random<br>\nimport json<br>\nimport numpy as np<br>\nimport tensorflow as tf<br>\nimport cv2<br>\nimport PIL<br>\nimport pathlib<br>\nfrom pathlib import Path<br>\nfrom PIL import Image<br>\nimport gflags<br>\nfrom gflags import *</p>\n<p>def read_and_decode(filename_queue):<br>\n    reader = tf.TFRecordReader()<br>\n    _, example = reader.read(filename_queue)</p>\n<pre><code>feature_dict = {}\nfeature_dict[\"image\"] = tf.FixedLenFeature([], tf.string)\nfeature_dict[\"image_name\"] = tf.FixedLenFeature([], tf.string)\nfeature_dict[\"target\"] = tf.FixedLenFeature([], tf.int64)\n\nparsed_example = tf.parse_single_example(example, features=feature_dict)\n\nimage = tf.decode_raw(parsed_example[\"image\"], tf.float32)\nimage = tf.reshape(image, [192, 192, 3])\nlabel = tf.cast(parsed_example[\"target\"], tf.int32)\nname = tf.cast(parsed_example[\"image_name\"], tf.string)\nreturn image, name, label\n</code></pre>\n<p>def gen_batch(paths, examples_num, batch_size, shuffle):<br>\n    filename_queue = tf.train.string_input_producer(paths, num_epochs=2)<br>\n    print(filename_queue.size())<br>\n    image, name, label = read_and_decode(filename_queue)<br>\n    min_fraction_of_examples_in_queue = 0.4<br>\n    min_queue_examples = int(min_fraction_of_examples_in_queue * examples_num)<br>\n    min_queue_examples = 500<br>\n    num_process_threads = 4<br>\n    images, names, labels = tf.train.shuffle_batch([image, name, label],\\<br>\n                batch_size=batch_size,\\<br>\n                num_threads=num_process_threads,\\<br>\n                #capacity=min_queue_examples + batch_size * 3,\\<br>\n                capacity=1000,\\<br>\n                min_after_dequeue=100)<br>\n    return images, names, labels</p>\n<p>def main():<br>\n    root = \"/opt/program/services/project/cassava-leaf-disease-classification/\"</p>\n<pre><code># train images num\nimage_path = Path('%strain_images/' % root)\nimage_count = len(list(image_path.glob('*.jpg')))\n\npaths = ['0.tfrecords', '1.tfrecords', '2.tfrecords']\nimages, names, labels = gen_batch(paths, image_count, 1, True)\n\nmodel_path = \"%s/model/leaf-disease.model\" % root\nsaver = tf.train.Saver()\nwith tf.Session() as sess:\n    sess.run(tf.global_variables_initializer())\n    sess.run(tf.local_variables_initializer())\n\n    coord = tf.train.Coordinator()\n    threads = tf.train.start_queue_runners(coord=coord)\n\n    for i in range(3):\n        print(\"i = %d\" % i)\n        image_list= sess.run([images])\n        if i % 100 == 0:\n            print('step: {}, loss: {}'.format(i, train_loss))\n            saver.save(sess, model_path, global_step=i)\n    coord.request_stop()\n    coord.join(threads)\n</code></pre>\n<p>if <strong>name</strong> == '<strong>main</strong>':<br>\n    main()`</p>\n<p><strong>Note: '0.tfrecords', '1.tfrecords', '2.tfrecords' locate in same path with my code</strong></p>\n<p>but when runing,  error occured as following:</p>\n<p><strong>OutOfRangeError (see above for traceback): RandomShuffleQueue '_1_shuffle_batch/random_shuffle_queue' is closed and has insufficient elements (requested 1, current size 0) [[Node: shuffle_batch = QueueDequeueManyV2[component_types=[DT_FLOAT, DT_STRING, DT_INT32], timeout_ms=-1, _device=\"/job:localhost/replica:0/task:0/device:CPU:0\"](shuffle_batch/random_shuffle_queue, shuffle_batch/n)]]</strong></p>\n<p>I have try to check my code Carefully，but can't location the problem.</p>",
      "rawMarkdown": "I try to load tfrecord files by using the following code:\n\n` \nimport os\nimport sys\nimport logging\nimport random\nimport json\nimport numpy as np\nimport tensorflow as tf\nimport cv2\nimport PIL\nimport pathlib\nfrom pathlib import Path\nfrom PIL import Image\nimport gflags\nfrom gflags import *\n\ndef read_and_decode(filename_queue):\n    reader = tf.TFRecordReader()\n    _, example = reader.read(filename_queue)\n\n    feature_dict = {}\n    feature_dict[\"image\"] = tf.FixedLenFeature([], tf.string)\n    feature_dict[\"image_name\"] = tf.FixedLenFeature([], tf.string)\n    feature_dict[\"target\"] = tf.FixedLenFeature([], tf.int64)\n\n    parsed_example = tf.parse_single_example(example, features=feature_dict)\n\n    image = tf.decode_raw(parsed_example[\"image\"], tf.float32)\n    image = tf.reshape(image, [192, 192, 3])\n    label = tf.cast(parsed_example[\"target\"], tf.int32)\n    name = tf.cast(parsed_example[\"image_name\"], tf.string)\n    return image, name, label\n\ndef gen_batch(paths, examples_num, batch_size, shuffle):\n    filename_queue = tf.train.string_input_producer(paths, num_epochs=2)\n    print(filename_queue.size())\n    image, name, label = read_and_decode(filename_queue)\n    min_fraction_of_examples_in_queue = 0.4\n    min_queue_examples = int(min_fraction_of_examples_in_queue * examples_num)\n    min_queue_examples = 500\n    num_process_threads = 4\n    images, names, labels = tf.train.shuffle_batch([image, name, label],\\\n                batch_size=batch_size,\\\n                num_threads=num_process_threads,\\\n                #capacity=min_queue_examples + batch_size * 3,\\\n                capacity=1000,\\\n                min_after_dequeue=100)\n    return images, names, labels\n\ndef main():\n    root = \"/opt/program/services/project/cassava-leaf-disease-classification/\"\n\n    # train images num\n    image_path = Path('%strain_images/' % root)\n    image_count = len(list(image_path.glob('*.jpg')))\n\n    paths = ['0.tfrecords', '1.tfrecords', '2.tfrecords']\n    images, names, labels = gen_batch(paths, image_count, 1, True)\n\n    model_path = \"%s/model/leaf-disease.model\" % root\n    saver = tf.train.Saver()\n    with tf.Session() as sess:\n        sess.run(tf.global_variables_initializer())\n        sess.run(tf.local_variables_initializer())\n\n        coord = tf.train.Coordinator()\n        threads = tf.train.start_queue_runners(coord=coord)\n\n        for i in range(3):\n            print(\"i = %d\" % i)\n            image_list= sess.run([images])\n            if i % 100 == 0:\n                print('step: {}, loss: {}'.format(i, train_loss))\n                saver.save(sess, model_path, global_step=i)\n        coord.request_stop()\n        coord.join(threads)\n\nif __name__ == '__main__':\n    main()`\n\n**Note: '0.tfrecords', '1.tfrecords', '2.tfrecords' locate in same path with my code**\n\nbut when runing,  error occured as following:\n\n**OutOfRangeError (see above for traceback): RandomShuffleQueue '_1_shuffle_batch/random_shuffle_queue' is closed and has insufficient elements (requested 1, current size 0) [[Node: shuffle_batch = QueueDequeueManyV2[component_types=[DT_FLOAT, DT_STRING, DT_INT32], timeout_ms=-1, _device=\"/job:localhost/replica:0/task:0/device:CPU:0\"](shuffle_batch/random_shuffle_queue, shuffle_batch/n)]]**\n\nI have try to check my code Carefully，but can't location the problem.",
      "votes": null
    },
    {
      "id": "1131855",
      "postDate": "12/30/2020 03:15:46",
      "content": "<p>Emmmm，I notice that you may set the wrong path for your tfrecord files,so the number is 0 and shuffle() doesn't work.<br>\nYou can try adding ‘/train_tfrecords' to your absolute path : )</p>",
      "rawMarkdown": "Emmmm，I notice that you may set the wrong path for your tfrecord files,so the number is 0 and shuffle() doesn't work.\nYou can try adding ‘/train_tfrecords' to your absolute path : )",
      "votes": null
    },
    {
      "id": "1132042",
      "postDate": "12/30/2020 06:15:48",
      "content": "<p><a href=\"https://www.kaggle.com/xzy777\" target=\"_blank\">@xzy777</a> thank you a lot, but i have try to use \"['/0.tfrecords', '/2.tfrecords', '/3.tfrecords']\"  as paths, same error occured too</p>",
      "rawMarkdown": "xzy777 thank you a lot, but i have try to use \"['/0.tfrecords', '/2.tfrecords', '/3.tfrecords']\"  as paths, same error occured too",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1131855,
      "author_name": "xzy777",
      "author_url": "",
      "post_date": "12/30/2020 03:15:46",
      "content": "<p>Emmmm，I notice that you may set the wrong path for your tfrecord files,so the number is 0 and shuffle() doesn't work.<br>\nYou can try adding ‘/train_tfrecords' to your absolute path : )</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1132042,
      "author_name": "xiaolangjun",
      "author_url": "",
      "post_date": "12/30/2020 06:15:48",
      "content": "<p><a href=\"https://www.kaggle.com/xzy777\" target=\"_blank\">@xzy777</a> thank you a lot, but i have try to use \"['/0.tfrecords', '/2.tfrecords', '/3.tfrecords']\"  as paths, same error occured too</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1130749": "I try to load tfrecord files by using the following code:\n\n` \nimport os\nimport sys\nimport logging\nimport random\nimport json\nimport numpy as np\nimport tensorflow as tf\nimport cv2\nimport PIL\nimport pathlib\nfrom pathlib import Path\nfrom PIL import Image\nimport gflags\nfrom gflags import *\n\ndef read_and_decode(filename_queue):\n    reader = tf.TFRecordReader()\n    _, example = reader.read(filename_queue)\n\n    feature_dict = {}\n    feature_dict[\"image\"] = tf.FixedLenFeature([], tf.string)\n    feature_dict[\"image_name\"] = tf.FixedLenFeature([], tf.string)\n    feature_dict[\"target\"] = tf.FixedLenFeature([], tf.int64)\n\n    parsed_example = tf.parse_single_example(example, features=feature_dict)\n\n    image = tf.decode_raw(parsed_example[\"image\"], tf.float32)\n    image = tf.reshape(image, [192, 192, 3])\n    label = tf.cast(parsed_example[\"target\"], tf.int32)\n    name = tf.cast(parsed_example[\"image_name\"], tf.string)\n    return image, name, label\n\ndef gen_batch(paths, examples_num, batch_size, shuffle):\n    filename_queue = tf.train.string_input_producer(paths, num_epochs=2)\n    print(filename_queue.size())\n    image, name, label = read_and_decode(filename_queue)\n    min_fraction_of_examples_in_queue = 0.4\n    min_queue_examples = int(min_fraction_of_examples_in_queue * examples_num)\n    min_queue_examples = 500\n    num_process_threads = 4\n    images, names, labels = tf.train.shuffle_batch([image, name, label],\\\n                batch_size=batch_size,\\\n                num_threads=num_process_threads,\\\n                #capacity=min_queue_examples + batch_size * 3,\\\n                capacity=1000,\\\n                min_after_dequeue=100)\n    return images, names, labels\n\ndef main():\n    root = \"/opt/program/services/project/cassava-leaf-disease-classification/\"\n\n    # train images num\n    image_path = Path('%strain_images/' % root)\n    image_count = len(list(image_path.glob('*.jpg')))\n\n    paths = ['0.tfrecords', '1.tfrecords', '2.tfrecords']\n    images, names, labels = gen_batch(paths, image_count, 1, True)\n\n    model_path = \"%s/model/leaf-disease.model\" % root\n    saver = tf.train.Saver()\n    with tf.Session() as sess:\n        sess.run(tf.global_variables_initializer())\n        sess.run(tf.local_variables_initializer())\n\n        coord = tf.train.Coordinator()\n        threads = tf.train.start_queue_runners(coord=coord)\n\n        for i in range(3):\n            print(\"i = %d\" % i)\n            image_list= sess.run([images])\n            if i % 100 == 0:\n                print('step: {}, loss: {}'.format(i, train_loss))\n                saver.save(sess, model_path, global_step=i)\n        coord.request_stop()\n        coord.join(threads)\n\nif __name__ == '__main__':\n    main()`\n\n**Note: '0.tfrecords', '1.tfrecords', '2.tfrecords' locate in same path with my code**\n\nbut when runing,  error occured as following:\n\n**OutOfRangeError (see above for traceback): RandomShuffleQueue '_1_shuffle_batch/random_shuffle_queue' is closed and has insufficient elements (requested 1, current size 0) [[Node: shuffle_batch = QueueDequeueManyV2[component_types=[DT_FLOAT, DT_STRING, DT_INT32], timeout_ms=-1, _device=\"/job:localhost/replica:0/task:0/device:CPU:0\"](shuffle_batch/random_shuffle_queue, shuffle_batch/n)]]**\n\nI have try to check my code Carefully，but can't location the problem.",
    "1131855": "Emmmm，I notice that you may set the wrong path for your tfrecord files,so the number is 0 and shuffle() doesn't work.\nYou can try adding ‘/train_tfrecords' to your absolute path : )",
    "1132042": "xzy777 thank you a lot, but i have try to use \"['/0.tfrecords', '/2.tfrecords', '/3.tfrecords']\"  as paths, same error occured too"
  },
  "source": "meta"
}