{
  "id": 38629,
  "title": "how to speed up image processing for training/testing",
  "url": "/competitions/carvana-image-masking-challenge/discussion/38629",
  "author_name": "kirk",
  "post_date": "2017-08-27T20:11:32.700000",
  "votes": 3,
  "comment_count": 0,
  "views": 0,
  "content": "<p>I've seen in the forums some people compare <code>opencv vs pillow</code>. I find it an unfair comparison to be honest. Why you might ask? Well, pillow in case you didn't know has a little brother or big whatever suits you better. It's called <code>pillow-simd</code>, <code>pip install -U --force-reinstall pillow-simd</code> which is inherently designed for multiple data to run inside a single instruction. I think that would be a more fair comparison between the two. </p>\n\n<p>But in all fairness even though they are great libraries still there is a lot of room for improvement here that could speed up the processing part by orders of magnitude. If you have a multi-core machine which most of you do by using either <code>opencv</code> or <code>pillow-simd</code> you still are not fully utilizing all your processing power.</p>\n\n<p>Again you might ask why? Don't worry I've been asking these questions myself lately a lot. You see the problem is with python which uses <a href=\"https://wiki.python.org/moin/GlobalInterpreterLock\">GIL</a>. It simply blocks python from utilizing multiple process on your machine. For that reason I've created a kernel demonstrating how to harness all your processing power for the greater good <a href=\"https://www.kaggle.com/kirk86/multicore-image-processing-for-training-prediction\">here</a>. What I did is that I took <code>Peter Giannakopoulos</code> starter code and made a few changes so that it speeds up the image processing by orders of magnitude on a multi-core machine.</p>\n\n<p>Keras provides an <code>ImageDataGenerator</code> which is very handy but is darn slow. When I tested it on 3200 images with a batch size of 32 it took 60.73 min. while utilizing multiprocessing I was able to increase prediction speed and finish all 100K images at approximately the same time.</p>\n\n<p>Cheers!</p>",
  "messages": [
    {
      "id": 216728,
      "postDate": "2017-08-27T20:11:32.700Z",
      "content": "<p>I've seen in the forums some people compare <code>opencv vs pillow</code>. I find it an unfair comparison to be honest. Why you might ask? Well, pillow in case you didn't know has a little brother or big whatever suits you better. It's called <code>pillow-simd</code>, <code>pip install -U --force-reinstall pillow-simd</code> which is inherently designed for multiple data to run inside a single instruction. I think that would be a more fair comparison between the two. </p>\n\n<p>But in all fairness even though they are great libraries still there is a lot of room for improvement here that could speed up the processing part by orders of magnitude. If you have a multi-core machine which most of you do by using either <code>opencv</code> or <code>pillow-simd</code> you still are not fully utilizing all your processing power.</p>\n\n<p>Again you might ask why? Don't worry I've been asking these questions myself lately a lot. You see the problem is with python which uses <a href=\"https://wiki.python.org/moin/GlobalInterpreterLock\">GIL</a>. It simply blocks python from utilizing multiple process on your machine. For that reason I've created a kernel demonstrating how to harness all your processing power for the greater good <a href=\"https://www.kaggle.com/kirk86/multicore-image-processing-for-training-prediction\">here</a>. What I did is that I took <code>Peter Giannakopoulos</code> starter code and made a few changes so that it speeds up the image processing by orders of magnitude on a multi-core machine.</p>\n\n<p>Keras provides an <code>ImageDataGenerator</code> which is very handy but is darn slow. When I tested it on 3200 images with a batch size of 32 it took 60.73 min. while utilizing multiprocessing I was able to increase prediction speed and finish all 100K images at approximately the same time.</p>\n\n<p>Cheers!</p>",
      "rawMarkdown": "I've seen in the forums some people compare `opencv vs pillow`. I find it an unfair comparison to be honest. Why you might ask? Well, pillow in case you didn't know has a little brother or big whatever suits you better. It's called `pillow-simd`, `pip install -U --force-reinstall pillow-simd` which is inherently designed for multiple data to run inside a single instruction. I think that would be a more fair comparison between the two. \n\nBut in all fairness even though they are great libraries still there is a lot of room for improvement here that could speed up the processing part by orders of magnitude. If you have a multi-core machine which most of you do by using either `opencv` or `pillow-simd` you still are not fully utilizing all your processing power.\n\nAgain you might ask why? Don't worry I've been asking these questions myself lately a lot. You see the problem is with python which uses [GIL][1]. It simply blocks python from utilizing multiple process on your machine. For that reason I've created a kernel demonstrating how to harness all your processing power for the greater good [here][2]. What I did is that I took `Peter Giannakopoulos` starter code and made a few changes so that it speeds up the image processing by orders of magnitude on a multi-core machine.\n\nKeras provides an `ImageDataGenerator` which is very handy but is darn slow. When I tested it on 3200 images with a batch size of 32 it took 60.73 min. while utilizing multiprocessing I was able to increase prediction speed and finish all 100K images at approximately the same time.\n\nCheers!\n\n\n  [1]: https://wiki.python.org/moin/GlobalInterpreterLock\n  [2]: https://www.kaggle.com/kirk86/multicore-image-processing-for-training-prediction",
      "votes": 3
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "216728": "I've seen in the forums some people compare `opencv vs pillow`. I find it an unfair comparison to be honest. Why you might ask? Well, pillow in case you didn't know has a little brother or big whatever suits you better. It's called `pillow-simd`, `pip install -U --force-reinstall pillow-simd` which is inherently designed for multiple data to run inside a single instruction. I think that would be a more fair comparison between the two. \n\nBut in all fairness even though they are great libraries still there is a lot of room for improvement here that could speed up the processing part by orders of magnitude. If you have a multi-core machine which most of you do by using either `opencv` or `pillow-simd` you still are not fully utilizing all your processing power.\n\nAgain you might ask why? Don't worry I've been asking these questions myself lately a lot. You see the problem is with python which uses [GIL][1]. It simply blocks python from utilizing multiple process on your machine. For that reason I've created a kernel demonstrating how to harness all your processing power for the greater good [here][2]. What I did is that I took `Peter Giannakopoulos` starter code and made a few changes so that it speeds up the image processing by orders of magnitude on a multi-core machine.\n\nKeras provides an `ImageDataGenerator` which is very handy but is darn slow. When I tested it on 3200 images with a batch size of 32 it took 60.73 min. while utilizing multiprocessing I was able to increase prediction speed and finish all 100K images at approximately the same time.\n\nCheers!\n\n\n  [1]: https://wiki.python.org/moin/GlobalInterpreterLock\n  [2]: https://www.kaggle.com/kirk86/multicore-image-processing-for-training-prediction"
  }
}