{
  "id": 20842,
  "title": "Hardware/software",
  "url": "/competitions/state-farm-distracted-driver-detection/discussion/20842",
  "author_name": "Vladimir Iglovikov",
  "post_date": "2016-05-11T00:05:48.923000",
  "votes": 4,
  "comment_count": 0,
  "views": 346,
  "content": "<p>Hello,</p>\n\n<p>These are probably lame questions, that are asked all the time, but still:</p>\n\n<ol>\n<li>What hardware are you using for computations for this competition? </li>\n<li>How to make training faster? </li>\n</ol>\n\n<p>For example for 1:\nI am using GTX 980 Ti that I bought to be able to train  neural networks. It takes 10 minutes per epoch for VGG16 with batch size 32 using keras with Theano backend. (I tried TensorFlow backend, it is much slower)</p>\n\n<p>For example, for 2:\nif you use Theano and add allow_gc=False to your .theanorc your GPU will consume more memory, but it will save time on reallocation memory =&gt; faster computation.</p>\n\n<p>Or this benchmark <a href=\"https://github.com/soumith/convnet-benchmarks\">NN packages</a> implies that neon works faster than Caffe and theano.</p>\n\n<p>I understand that almost noone would care if training(or predicting) process will become 5% faster, but if one may get it almost for free, by choosing proper batch size, or adding extra parameter in some config I would try to go for it.</p>\n\n<p>Or, this question: Did anyone try to overclock GPU in linux to add these extra 5% to performance?</p>\n\n<p>Vladimir</p>",
  "messages": [
    {
      "id": 119503,
      "postDate": "2016-05-11T00:05:48.923Z",
      "content": "<p>Hello,</p>\n\n<p>These are probably lame questions, that are asked all the time, but still:</p>\n\n<ol>\n<li>What hardware are you using for computations for this competition? </li>\n<li>How to make training faster? </li>\n</ol>\n\n<p>For example for 1:\nI am using GTX 980 Ti that I bought to be able to train  neural networks. It takes 10 minutes per epoch for VGG16 with batch size 32 using keras with Theano backend. (I tried TensorFlow backend, it is much slower)</p>\n\n<p>For example, for 2:\nif you use Theano and add allow_gc=False to your .theanorc your GPU will consume more memory, but it will save time on reallocation memory =&gt; faster computation.</p>\n\n<p>Or this benchmark <a href=\"https://github.com/soumith/convnet-benchmarks\">NN packages</a> implies that neon works faster than Caffe and theano.</p>\n\n<p>I understand that almost noone would care if training(or predicting) process will become 5% faster, but if one may get it almost for free, by choosing proper batch size, or adding extra parameter in some config I would try to go for it.</p>\n\n<p>Or, this question: Did anyone try to overclock GPU in linux to add these extra 5% to performance?</p>\n\n<p>Vladimir</p>",
      "rawMarkdown": "Hello,\r\n\r\nThese are probably lame questions, that are asked all the time, but still:\r\n\r\n 1. What hardware are you using for computations for this competition? \r\n 2. How to make training faster? \r\n\r\nFor example for 1:\r\nI am using GTX 980 Ti that I bought to be able to train  neural networks. It takes 10 minutes per epoch for VGG16 with batch size 32 using keras with Theano backend. (I tried TensorFlow backend, it is much slower)\r\n\r\nFor example, for 2:\r\nif you use Theano and add allow_gc=False to your .theanorc your GPU will consume more memory, but it will save time on reallocation memory => faster computation.\r\n\r\nOr this benchmark [NN packages][1] implies that neon works faster than Caffe and theano.\r\n\r\nI understand that almost noone would care if training(or predicting) process will become 5% faster, but if one may get it almost for free, by choosing proper batch size, or adding extra parameter in some config I would try to go for it.\r\n\r\nOr, this question: Did anyone try to overclock GPU in linux to add these extra 5% to performance?\r\n\r\nVladimir\r\n\r\n\r\n  [1]: https://github.com/soumith/convnet-benchmarks",
      "votes": 4
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "119503": "Hello,\r\n\r\nThese are probably lame questions, that are asked all the time, but still:\r\n\r\n 1. What hardware are you using for computations for this competition? \r\n 2. How to make training faster? \r\n\r\nFor example for 1:\r\nI am using GTX 980 Ti that I bought to be able to train  neural networks. It takes 10 minutes per epoch for VGG16 with batch size 32 using keras with Theano backend. (I tried TensorFlow backend, it is much slower)\r\n\r\nFor example, for 2:\r\nif you use Theano and add allow_gc=False to your .theanorc your GPU will consume more memory, but it will save time on reallocation memory => faster computation.\r\n\r\nOr this benchmark [NN packages][1] implies that neon works faster than Caffe and theano.\r\n\r\nI understand that almost noone would care if training(or predicting) process will become 5% faster, but if one may get it almost for free, by choosing proper batch size, or adding extra parameter in some config I would try to go for it.\r\n\r\nOr, this question: Did anyone try to overclock GPU in linux to add these extra 5% to performance?\r\n\r\nVladimir\r\n\r\n\r\n  [1]: https://github.com/soumith/convnet-benchmarks"
  }
}