Using the Keras Image Generator in Kaggle Kernels seems to now have issues with some of the images. Is this a dataset issue or a Kernels issue?
\n\nhttps://www.kaggle.com/kmader/baseline-u-net-model-part-1?scriptVersionId=4896157 \nhttps://www.kaggle.com/kmader/transfer-learning-for-boat-or-no-boat\n```\nWARNING:tensorflow:Variable *= will be deprecated. Use variable.assign_mul if you want assignment to the variable value or 'x = x * y' if you want a new python Tensor object.\nEpoch 1/3
\n\nOSError Traceback (most recent call last)\n/opt/conda/lib/python3.6/site-packages/skimage/io/_plugins/pil_plugin.py in pil_to_ndarray(im, dtype, img_num)\n 52 # this will raise an IOError if the file is not readable\n---> 53 im.getdata()[0]\n 54 except IOError as e:
\n\n/opt/conda/lib/python3.6/site-packages/PIL/Image.py in getdata(self, band)\n 1186 \n-> 1187 self.load()\n 1188 if band is not None:
\n\n/opt/conda/lib/python3.6/site-packages/PIL/ImageFile.py in load(self)\n 230 raise IOError(\"image file is truncated \"\n--> 231 \"(%d bytes not processed)\" % len(b))\n 232
\n\nOSError: image file is truncated (55 bytes not processed)
\n\nDuring handling of the above exception, another exception occurred:
\n\nValueError Traceback (most recent call last)\n in ()\n 6 validation_data=(valid_x, valid_y),\n 7 callbacks=callbacks_list,\n----> 8 workers=1 # the generator is not very thread safe\n 9 )]
\n\n/opt/conda/lib/python3.6/site-packages/Keras-2.1.5-py3.6.egg/keras/legacy/interfaces.py in wrapper(*args, **kwargs)\n 89 warnings.warn('Update your ' + object_name +\n 90 ' call to the Keras 2 API: ' + signature, stacklevel=2)\n---> 91 return func(*args, **kwargs)\n 92 wrapper._original_function = func\n 93 return wrapper
/opt/conda/lib/python3.6/site-packages/Keras-2.1.5-py3.6.egg/keras/engine/training.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)\n 2193 batch_index = 0\n```
", "messages": [ { "id": "365736", "postDate": "08/03/2018 09:32:53", "content": "Using the Keras Image Generator in Kaggle Kernels seems to now have issues with some of the images. Is this a dataset issue or a Kernels issue?
\n\nhttps://www.kaggle.com/kmader/baseline-u-net-model-part-1?scriptVersionId=4896157 \nhttps://www.kaggle.com/kmader/transfer-learning-for-boat-or-no-boat\n```\nWARNING:tensorflow:Variable *= will be deprecated. Use variable.assign_mul if you want assignment to the variable value or 'x = x * y' if you want a new python Tensor object.\nEpoch 1/3
\n\nOSError Traceback (most recent call last)\n/opt/conda/lib/python3.6/site-packages/skimage/io/_plugins/pil_plugin.py in pil_to_ndarray(im, dtype, img_num)\n 52 # this will raise an IOError if the file is not readable\n---> 53 im.getdata()[0]\n 54 except IOError as e:
\n\n/opt/conda/lib/python3.6/site-packages/PIL/Image.py in getdata(self, band)\n 1186 \n-> 1187 self.load()\n 1188 if band is not None:
\n\n/opt/conda/lib/python3.6/site-packages/PIL/ImageFile.py in load(self)\n 230 raise IOError(\"image file is truncated \"\n--> 231 \"(%d bytes not processed)\" % len(b))\n 232
\n\nOSError: image file is truncated (55 bytes not processed)
\n\nDuring handling of the above exception, another exception occurred:
\n\nValueError Traceback (most recent call last)\n in ()\n 6 validation_data=(valid_x, valid_y),\n 7 callbacks=callbacks_list,\n----> 8 workers=1 # the generator is not very thread safe\n 9 )]
\n\n/opt/conda/lib/python3.6/site-packages/Keras-2.1.5-py3.6.egg/keras/legacy/interfaces.py in wrapper(*args, **kwargs)\n 89 warnings.warn('Update your ' + object_name +\n 90 ' call to the Keras 2 API: ' + signature, stacklevel=2)\n---> 91 return func(*args, **kwargs)\n 92 wrapper._original_function = func\n 93 return wrapper
/opt/conda/lib/python3.6/site-packages/Keras-2.1.5-py3.6.egg/keras/engine/training.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)\n 2193 batch_index = 0\n```
", "rawMarkdown": "Using the Keras Image Generator in Kaggle Kernels seems to now have issues with some of the images. Is this a dataset issue or a Kernels issue?\n\nhttps://www.kaggle.com/kmader/baseline-u-net-model-part-1?scriptVersionId=4896157 \nhttps://www.kaggle.com/kmader/transfer-learning-for-boat-or-no-boat\n```\nWARNING:tensorflow:Variable *= will be deprecated. Use variable.assign_mul if you want assignment to the variable value or 'x = x * y' if you want a new python Tensor object.\nEpoch 1/3\n132/150 [=========================>....] - ETA: 40s - loss: -0.0718 - dice_coef: 0.0721 - binary_accuracy: 0.9971 - true_positive_rate: 1.8234e-04\n---------------------------------------------------------------------------\nOSError Traceback (most recent call last)\n/opt/conda/lib/python3.6/site-packages/skimage/io/_plugins/pil_plugin.py in pil_to_ndarray(im, dtype, img_num)\n 52 # this will raise an IOError if the file is not readable\n---> 53 im.getdata()[0]\n 54 except IOError as e:\n\n/opt/conda/lib/python3.6/site-packages/PIL/Image.py in getdata(self, band)\n 1186 \n-> 1187 self.load()\n 1188 if band is not None:\n\n/opt/conda/lib/python3.6/site-packages/PIL/ImageFile.py in load(self)\n 230 raise IOError(\"image file is truncated \"\n--> 231 \"(%d bytes not processed)\" % len(b))\n 232 \n\nOSError: image file is truncated (55 bytes not processed)\n\nDuring handling of the above exception, another exception occurred:\n\nValueError Traceback (most recent call last)", "votes": null }, { "id": "365838", "postDate": "08/03/2018 13:53:53", "content": "Hi Kevin, \nI'm having the same problem. It doesn't look like problem with Keras, I think it's just some corrupted file that PIL isn't able to read. It would be nice to post ids of such images here. Currently I'm searching for the bad images. I'll update you when I find something.
", "rawMarkdown": "Hi Kevin, \nI'm having the same problem. It doesn't look like problem with Keras, I think it's just some corrupted file that PIL isn't able to read. It would be nice to post ids of such images here. Currently I'm searching for the bad images. I'll update you when I find something.", "votes": null }, { "id": "365843", "postDate": "08/03/2018 14:04:29", "content": "I added this block to remove files smaller than 50kb (using os.stat) and it seems to work now
\n\nunique_img_ids['file_size_kb'] = unique_img_ids['ImageId'].map(lambda c_img_id: \n os.stat(os.path.join(train_image_dir, \n c_img_id)).st_size/1024)
unique_img_ids = unique_img_ids[unique_img_ids['file_size_kb']>50] # keep only 50kb files\n
see kernel https://www.kaggle.com/kmader/baseline-u-net-model-part-1
", "rawMarkdown": "I added this block to remove files smaller than 50kb (using os.stat) and it seems to work now \n\n```unique_img_ids['file_size_kb'] = unique_img_ids['ImageId'].map(lambda c_img_id: \n os.stat(os.path.join(train_image_dir, \n c_img_id)).st_size/1024)```\n\n```unique_img_ids = unique_img_ids[unique_img_ids['file_size_kb']>50] # keep only 50kb files\n```\n\nsee kernel https://www.kaggle.com/kmader/baseline-u-net-model-part-1", "votes": null }, { "id": "366746", "postDate": "08/06/2018 12:00:39", "content": "This is the corrupted file: 6384c3e78.jpg. Others are OK.
I haven't hit this issue and I was following your part 1 kernel kevin (and purposefully didn't filter out files based on image size). I was running locally on my own machine with the dataset downloaded so Maybe it's a problem with the Kaggle version of the dataset?
", "rawMarkdown": "I haven't hit this issue and I was following your part 1 kernel kevin (and purposefully didn't filter out files based on image size). I was running locally on my own machine with the dataset downloaded so Maybe it's a problem with the Kaggle version of the dataset?", "votes": null }, { "id": "367118", "postDate": "08/07/2018 06:01:30", "content": "I've downloaded dataset through Kaggle api yesterday and 6384c3e78.jpg is broken too.
i had the same problem; i just remove 6384c3e78.jpg from the dataset
", "rawMarkdown": "i had the same problem; i just remove 6384c3e78.jpg from the dataset", "votes": null }, { "id": "367248", "postDate": "08/07/2018 11:23:07", "content": "the image file 6384c3e78.jpg is truncated; i think whether the issue occurred due to how you read the pic; cv2.imread/plt.imread is ok, skimage.data.imread fail;
", "rawMarkdown": "the image file 6384c3e78.jpg is truncated; i think whether the issue occurred due to how you read the pic; cv2.imread/plt.imread is ok, skimage.data.imread fail;", "votes": null }, { "id": "406306", "postDate": "10/19/2018 02:57:04", "content": "The image can be loaded by setting PIL.ImageFile.LOAD_TRUNCATED_IMAGES = True at the beginning
", "rawMarkdown": "The image can be loaded by setting PIL.ImageFile.LOAD_TRUNCATED_IMAGES = True at the beginning", "votes": null }, { "id": "415468", "postDate": "11/05/2018 07:25:22", "content": "Thank you this helped resolve a similar issue I had in a different dataset!
", "rawMarkdown": "Thank you this helped resolve a similar issue I had in a different dataset!", "votes": null }, { "id": "445141", "postDate": "12/25/2018 18:21:44", "content": "I want to understand what truncated means in the case of a jpg?
", "rawMarkdown": "I want to understand what truncated means in the case of a jpg?", "votes": null } ], "comments": [ { "id": 365838, "author_name": "dzem90", "author_url": "", "post_date": "08/03/2018 13:53:53", "content": "Hi Kevin, \nI'm having the same problem. It doesn't look like problem with Keras, I think it's just some corrupted file that PIL isn't able to read. It would be nice to post ids of such images here. Currently I'm searching for the bad images. I'll update you when I find something.
", "votes": null, "replies": [ { "id": 365843, "author_name": "kmader", "author_url": "", "post_date": "08/03/2018 14:04:29", "content": "I added this block to remove files smaller than 50kb (using os.stat) and it seems to work now
\n\nunique_img_ids['file_size_kb'] = unique_img_ids['ImageId'].map(lambda c_img_id: \n os.stat(os.path.join(train_image_dir, \n c_img_id)).st_size/1024)
unique_img_ids = unique_img_ids[unique_img_ids['file_size_kb']>50] # keep only 50kb files\n
see kernel https://www.kaggle.com/kmader/baseline-u-net-model-part-1
", "votes": null, "replies": [] } ] }, { "id": 366746, "author_name": "dzem90", "author_url": "", "post_date": "08/06/2018 12:00:39", "content": "This is the corrupted file: 6384c3e78.jpg. Others are OK.
I've downloaded dataset through Kaggle api yesterday and 6384c3e78.jpg is broken too.
I haven't hit this issue and I was following your part 1 kernel kevin (and purposefully didn't filter out files based on image size). I was running locally on my own machine with the dataset downloaded so Maybe it's a problem with the Kaggle version of the dataset?
", "votes": null, "replies": [] }, { "id": 367243, "author_name": "wang10803", "author_url": "", "post_date": "08/07/2018 11:14:36", "content": "i had the same problem; i just remove 6384c3e78.jpg from the dataset
", "votes": null, "replies": [] }, { "id": 367248, "author_name": "wang10803", "author_url": "", "post_date": "08/07/2018 11:23:07", "content": "the image file 6384c3e78.jpg is truncated; i think whether the issue occurred due to how you read the pic; cv2.imread/plt.imread is ok, skimage.data.imread fail;
", "votes": null, "replies": [ { "id": 445141, "author_name": "manishrai", "author_url": "", "post_date": "12/25/2018 18:21:44", "content": "I want to understand what truncated means in the case of a jpg?
", "votes": null, "replies": [] } ] }, { "id": 406306, "author_name": "lyurip", "author_url": "", "post_date": "10/19/2018 02:57:04", "content": "The image can be loaded by setting PIL.ImageFile.LOAD_TRUNCATED_IMAGES = True at the beginning
", "votes": null, "replies": [] }, { "id": 415468, "author_name": "mfekadu", "author_url": "", "post_date": "11/05/2018 07:25:22", "content": "Thank you this helped resolve a similar issue I had in a different dataset!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "365736": "Using the Keras Image Generator in Kaggle Kernels seems to now have issues with some of the images. Is this a dataset issue or a Kernels issue?\n\nhttps://www.kaggle.com/kmader/baseline-u-net-model-part-1?scriptVersionId=4896157 \nhttps://www.kaggle.com/kmader/transfer-learning-for-boat-or-no-boat\n```\nWARNING:tensorflow:Variable *= will be deprecated. Use variable.assign_mul if you want assignment to the variable value or 'x = x * y' if you want a new python Tensor object.\nEpoch 1/3\n132/150 [=========================>....] - ETA: 40s - loss: -0.0718 - dice_coef: 0.0721 - binary_accuracy: 0.9971 - true_positive_rate: 1.8234e-04\n---------------------------------------------------------------------------\nOSError Traceback (most recent call last)\n/opt/conda/lib/python3.6/site-packages/skimage/io/_plugins/pil_plugin.py in pil_to_ndarray(im, dtype, img_num)\n 52 # this will raise an IOError if the file is not readable\n---> 53 im.getdata()[0]\n 54 except IOError as e:\n\n/opt/conda/lib/python3.6/site-packages/PIL/Image.py in getdata(self, band)\n 1186 \n-> 1187 self.load()\n 1188 if band is not None:\n\n/opt/conda/lib/python3.6/site-packages/PIL/ImageFile.py in load(self)\n 230 raise IOError(\"image file is truncated \"\n--> 231 \"(%d bytes not processed)\" % len(b))\n 232 \n\nOSError: image file is truncated (55 bytes not processed)\n\nDuring handling of the above exception, another exception occurred:\n\nValueError Traceback (most recent call last)", "365838": "Hi Kevin, \nI'm having the same problem. It doesn't look like problem with Keras, I think it's just some corrupted file that PIL isn't able to read. It would be nice to post ids of such images here. Currently I'm searching for the bad images. I'll update you when I find something.", "365843": "I added this block to remove files smaller than 50kb (using os.stat) and it seems to work now \n\n```unique_img_ids['file_size_kb'] = unique_img_ids['ImageId'].map(lambda c_img_id: \n os.stat(os.path.join(train_image_dir, \n c_img_id)).st_size/1024)```\n\n```unique_img_ids = unique_img_ids[unique_img_ids['file_size_kb']>50] # keep only 50kb files\n```\n\nsee kernel https://www.kaggle.com/kmader/baseline-u-net-model-part-1", "366746": "This is the corrupted file: `6384c3e78.jpg`. Others are OK.", "367022": "I haven't hit this issue and I was following your part 1 kernel kevin (and purposefully didn't filter out files based on image size). I was running locally on my own machine with the dataset downloaded so Maybe it's a problem with the Kaggle version of the dataset?", "367118": "I've downloaded dataset through Kaggle api yesterday and `6384c3e78.jpg` is broken too.", "367243": "i had the same problem; i just remove 6384c3e78.jpg from the dataset", "367248": "the image file 6384c3e78.jpg is truncated; i think whether the issue occurred due to how you read the pic; cv2.imread/plt.imread is ok, skimage.data.imread fail;", "406306": "The image can be loaded by setting PIL.ImageFile.LOAD_TRUNCATED_IMAGES = True at the beginning", "415468": "Thank you this helped resolve a similar issue I had in a different dataset!", "445141": "I want to understand what truncated means in the case of a jpg?" }, "source": "meta" }