{
  "id": 48463,
  "title": "Please help: Error during JPEG compression",
  "url": "/competitions/sp-society-camera-model-identification/discussion/48463",
  "author_name": "",
  "post_date": "2018-01-28T07:13:03.833780800Z",
  "votes": null,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I have been getting these errors while doing 'JPEG' compression on my training set for image manipulations - </p>\n\n<p> in image_aug(image, index)\n     21         img = Image.fromarray(image)\n     22         img.save('quality-90.jpg', \"JPEG\", quality=90)\n---&gt; 23         res_image = mpimg.imread('quality-90.jpg')\n     24     return res_image</p>\n\n<p>/home/janardhan/anaconda3/envs/dl/lib/python3.5/site-packages/matplotlib/image.py in imread(fname, format)\n   1276 \n   1277     if ext not in handlers:\n-&gt; 1278         im = pilread(fname)\n   1279         if im is None:\n   1280             raise ValueError('Only know how to handle extensions: %s; '</p>\n\n<p>/home/janardhan/anaconda3/envs/dl/lib/python3.5/site-packages/matplotlib/image.py in pilread(fname)\n   1255             return None\n   1256         with Image.open(fname) as image:\n-&gt; 1257             return pil_to_array(image)\n   1258 \n   1259     handlers = {'png': _png.read_png, }</p>\n\n<p>/home/janardhan/anaconda3/envs/dl/lib/python3.5/site-packages/matplotlib/image.py in pil_to_array(pilImage)\n   1381         # return MxNx3 RGB array\n   1382         im = pilImage  # no need to RGB images\n-&gt; 1383         x = toarray(im)\n   1384         x.shape = im.size[1], im.size[0], 3\n   1385         return x</p>\n\n<p>/home/janardhan/anaconda3/envs/dl/lib/python3.5/site-packages/matplotlib/image.py in toarray(im, dtype)\n   1364         # Pillow wants us to use \"tobytes\"\n   1365         if hasattr(im, 'tobytes'):\n-&gt; 1366             x_str = im.tobytes('raw', im.mode)\n   1367         else:\n   1368             x_str = im.tostring('raw', im.mode)</p>\n\n<p>/home/janardhan/anaconda3/envs/dl/lib/python3.5/site-packages/PIL/Image.py in tobytes(self, encoder_name, *args)\n    687         info, mode, size, palette, data = state\n    688         self.info = info\n--&gt; 689         self.mode = mode\n    690         self.size = size\n    691         self.im = core.new(mode, size)</p>\n\n<p>/home/janardhan/anaconda3/envs/dl/lib/python3.5/site-packages/PIL/ImageFile.py in load(self)\n    229                             if LOAD_TRUNCATED_IMAGES:\n    230                                 break\n--&gt; 231                             else:\n    232                                 raise IOError(\"image file is truncated \"\n    233                                               \"(%d bytes not processed)\" % len(b))</p>\n\n<p>OSError: image file is truncated (11 bytes not processed)</p>\n\n<p>This is my code for doing JPEG compression of quality 90 - \nimg = Image.fromarray(image)\nimg.save('quality-90.jpg', \"JPEG\", quality=90)\nres_image = mpimg.imread('quality-90.jpg')</p>\n\n<p>I have been stuck on this error or the past 1 day, Can you please help me regarding this?\nThank you.</p>",
  "messages": [
    {
      "id": "275078",
      "postDate": "01/28/2018 07:13:03",
      "content": "<p>I have been getting these errors while doing 'JPEG' compression on my training set for image manipulations - </p>\n\n<p> in image_aug(image, index)\n     21         img = Image.fromarray(image)\n     22         img.save('quality-90.jpg', \"JPEG\", quality=90)\n---&gt; 23         res_image = mpimg.imread('quality-90.jpg')\n     24     return res_image</p>\n\n<p>/home/janardhan/anaconda3/envs/dl/lib/python3.5/site-packages/matplotlib/image.py in imread(fname, format)\n   1276 \n   1277     if ext not in handlers:\n-&gt; 1278         im = pilread(fname)\n   1279         if im is None:\n   1280             raise ValueError('Only know how to handle extensions: %s; '</p>\n\n<p>/home/janardhan/anaconda3/envs/dl/lib/python3.5/site-packages/matplotlib/image.py in pilread(fname)\n   1255             return None\n   1256         with Image.open(fname) as image:\n-&gt; 1257             return pil_to_array(image)\n   1258 \n   1259     handlers = {'png': _png.read_png, }</p>\n\n<p>/home/janardhan/anaconda3/envs/dl/lib/python3.5/site-packages/matplotlib/image.py in pil_to_array(pilImage)\n   1381         # return MxNx3 RGB array\n   1382         im = pilImage  # no need to RGB images\n-&gt; 1383         x = toarray(im)\n   1384         x.shape = im.size[1], im.size[0], 3\n   1385         return x</p>\n\n<p>/home/janardhan/anaconda3/envs/dl/lib/python3.5/site-packages/matplotlib/image.py in toarray(im, dtype)\n   1364         # Pillow wants us to use \"tobytes\"\n   1365         if hasattr(im, 'tobytes'):\n-&gt; 1366             x_str = im.tobytes('raw', im.mode)\n   1367         else:\n   1368             x_str = im.tostring('raw', im.mode)</p>\n\n<p>/home/janardhan/anaconda3/envs/dl/lib/python3.5/site-packages/PIL/Image.py in tobytes(self, encoder_name, *args)\n    687         info, mode, size, palette, data = state\n    688         self.info = info\n--&gt; 689         self.mode = mode\n    690         self.size = size\n    691         self.im = core.new(mode, size)</p>\n\n<p>/home/janardhan/anaconda3/envs/dl/lib/python3.5/site-packages/PIL/ImageFile.py in load(self)\n    229                             if LOAD_TRUNCATED_IMAGES:\n    230                                 break\n--&gt; 231                             else:\n    232                                 raise IOError(\"image file is truncated \"\n    233                                               \"(%d bytes not processed)\" % len(b))</p>\n\n<p>OSError: image file is truncated (11 bytes not processed)</p>\n\n<p>This is my code for doing JPEG compression of quality 90 - \nimg = Image.fromarray(image)\nimg.save('quality-90.jpg', \"JPEG\", quality=90)\nres_image = mpimg.imread('quality-90.jpg')</p>\n\n<p>I have been stuck on this error or the past 1 day, Can you please help me regarding this?\nThank you.</p>",
      "rawMarkdown": "I have been getting these errors while doing 'JPEG' compression on my training set for image manipulations -",
      "votes": null
    },
    {
      "id": "275208",
      "postDate": "01/28/2018 13:38:44",
      "content": "<p>Have you tried this?:</p>\n\n<pre><code>from PIL import ImageFile\nImageFile.LOAD_TRUNCATED_IMAGES = True\n</code></pre>\n\n<p><a href=\"https://stackoverflow.com/a/23575424/3822261\">https://stackoverflow.com/a/23575424/3822261</a></p>\n\n<p>Disclosure: I haven't tried this, nor have I tried JPEG compression yet.</p>",
      "rawMarkdown": "Have you tried this?:\n\n    from PIL import ImageFile\n    ImageFile.LOAD_TRUNCATED_IMAGES = True\n\nhttps://stackoverflow.com/a/23575424/3822261\n\nDisclosure: I haven't tried this, nor have I tried JPEG compression yet.",
      "votes": null
    },
    {
      "id": "275210",
      "postDate": "01/28/2018 13:50:33",
      "content": "<p>yes, I have tried this. But the issue is solved with another change(though not sure what is the problem). I am using the same image_manipulation function for both training and validation, now i have separated the  functions for training and validation, and it is working now. It looks like while executing keras  model.fit_generator, both the validation_generator and train_generator are running parallelly, which is causing the issue - both validation and train generators are writing to the same file - 'quality-90.jpg'.</p>",
      "rawMarkdown": "yes, I have tried this. But the issue is solved with another change(though not sure what is the problem). I am using the same image_manipulation function for both training and validation, now i have separated the  functions for training and validation, and it is working now. It looks like while executing keras  model.fit_generator, both the validation_generator and train_generator are running parallelly, which is causing the issue - both validation and train generators are writing to the same file - 'quality-90.jpg'.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 275208,
      "author_name": "kleinsmith",
      "author_url": "",
      "post_date": "01/28/2018 13:38:44",
      "content": "<p>Have you tried this?:</p>\n\n<pre><code>from PIL import ImageFile\nImageFile.LOAD_TRUNCATED_IMAGES = True\n</code></pre>\n\n<p><a href=\"https://stackoverflow.com/a/23575424/3822261\">https://stackoverflow.com/a/23575424/3822261</a></p>\n\n<p>Disclosure: I haven't tried this, nor have I tried JPEG compression yet.</p>",
      "votes": null,
      "replies": [
        {
          "id": 275210,
          "author_name": "janardhanadapa",
          "author_url": "",
          "post_date": "01/28/2018 13:50:33",
          "content": "<p>yes, I have tried this. But the issue is solved with another change(though not sure what is the problem). I am using the same image_manipulation function for both training and validation, now i have separated the  functions for training and validation, and it is working now. It looks like while executing keras  model.fit_generator, both the validation_generator and train_generator are running parallelly, which is causing the issue - both validation and train generators are writing to the same file - 'quality-90.jpg'.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "275078": "I have been getting these errors while doing 'JPEG' compression on my training set for image manipulations -",
    "275208": "Have you tried this?:\n\n    from PIL import ImageFile\n    ImageFile.LOAD_TRUNCATED_IMAGES = True\n\nhttps://stackoverflow.com/a/23575424/3822261\n\nDisclosure: I haven't tried this, nor have I tried JPEG compression yet.",
    "275210": "yes, I have tried this. But the issue is solved with another change(though not sure what is the problem). I am using the same image_manipulation function for both training and validation, now i have separated the  functions for training and validation, and it is working now. It looks like while executing keras  model.fit_generator, both the validation_generator and train_generator are running parallelly, which is causing the issue - both validation and train generators are writing to the same file - 'quality-90.jpg'."
  },
  "source": "meta"
}