{
  "id": 19129,
  "title": "NVIDIA: CUDA works in SLI ? ",
  "url": "/competitions/second-annual-data-science-bowl/discussion/19129",
  "author_name": "",
  "post_date": "2016-02-22T21:36:05.890Z",
  "votes": null,
  "comment_count": 4,
  "views": 4984,
  "content": "<p>Hi, I have not found the answer in the CUDA Zone, not even a consensus in the forums</p>\n\n<p>NVIDIA CUDA toolkit works with two or more GPUs in SLI ? </p>\n\n<p>If works, what improvement I gain ? </p>",
  "messages": [
    {
      "id": "109051",
      "postDate": "02/22/2016 21:36:05",
      "content": "<p>Hi, I have not found the answer in the CUDA Zone, not even a consensus in the forums</p>\n\n<p>NVIDIA CUDA toolkit works with two or more GPUs in SLI ? </p>\n\n<p>If works, what improvement I gain ? </p>",
      "rawMarkdown": "Hi, I have not found the answer in the CUDA Zone, not even a consensus in the forums\r\n\r\n\r\nNVIDIA CUDA toolkit works with two or more GPUs in SLI ? \r\n\r\nIf works, what improvement I gain ?",
      "votes": null
    },
    {
      "id": "109053",
      "postDate": "02/22/2016 21:53:18",
      "content": "<p>Hi Alvaro.  SLI is not used within CUDA programming, it is a technology related to the use of GPUs for graphics.  Am I right in assuming you are hoping to speed up deep neural network training using multiple GPUs?  If so, there are a number of deep learning frameworks that support multi-GPU training of a single model.  This can happen because each GPU is individually addressable within a CUDA application, so workload can be distributed across them.  For example, the version of Caffe that powers the <a href=\"https://developer.nvidia.com/digits\">NVIDIA DIGITS</a> deep learning interface supports training a single model on multiple GPUs within a single compute node.  </p>\n\n<p>Disclosure: I work for NVIDIA.</p>",
      "rawMarkdown": "Hi Alvaro.  SLI is not used within CUDA programming, it is a technology related to the use of GPUs for graphics.  Am I right in assuming you are hoping to speed up deep neural network training using multiple GPUs?  If so, there are a number of deep learning frameworks that support multi-GPU training of a single model.  This can happen because each GPU is individually addressable within a CUDA application, so workload can be distributed across them.  For example, the version of Caffe that powers the [NVIDIA DIGITS][1] deep learning interface supports training a single model on multiple GPUs within a single compute node.  \r\n\r\nDisclosure: I work for NVIDIA.\r\n\r\n  [1]: https://developer.nvidia.com/digits",
      "votes": null
    },
    {
      "id": "109067",
      "postDate": "02/22/2016 23:57:27",
      "content": "<p>Hi Senecaur thanks for the answer.</p>\n\n<p>Yes, not only to train a neural neutwork but some image and data processing,  I'm trying to reduce the processing time in half. :D </p>",
      "rawMarkdown": "Hi Senecaur thanks for the answer.\r\n\r\nYes, not only to train a neural neutwork but some image and data processing,  I'm trying to reduce the processing time in half. :D",
      "votes": null
    },
    {
      "id": "109608",
      "postDate": "02/28/2016 15:46:47",
      "content": "<p>Even when not used in SLI, having two GPUs might help if you are training multiple models (for bagging with different seeds) or variations of the nnet structure. For instance, if you are using the Theano backend, you can simply run two parallel processes with device=gpu0 and device=gpu1 -- this basically halves my processing time...</p>",
      "rawMarkdown": "Even when not used in SLI, having two GPUs might help if you are training multiple models (for bagging with different seeds) or variations of the nnet structure. For instance, if you are using the Theano backend, you can simply run two parallel processes with device=gpu0 and device=gpu1 -- this basically halves my processing time...",
      "votes": null
    },
    {
      "id": "109634",
      "postDate": "02/28/2016 23:19:57",
      "content": "<p>I see, and the NVIDIA CUDA toolkit have some tools para debug some memory issues. It's good.</p>\n\n<p>I still need to learn in NVIDIA courses. :O</p>",
      "rawMarkdown": "I see, and the NVIDIA CUDA toolkit have some tools para debug some memory issues. It's good.\r\n\r\nI still need to learn in NVIDIA courses. :O",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 109053,
      "author_name": "senecaur",
      "author_url": "",
      "post_date": "02/22/2016 21:53:18",
      "content": "<p>Hi Alvaro.  SLI is not used within CUDA programming, it is a technology related to the use of GPUs for graphics.  Am I right in assuming you are hoping to speed up deep neural network training using multiple GPUs?  If so, there are a number of deep learning frameworks that support multi-GPU training of a single model.  This can happen because each GPU is individually addressable within a CUDA application, so workload can be distributed across them.  For example, the version of Caffe that powers the <a href=\"https://developer.nvidia.com/digits\">NVIDIA DIGITS</a> deep learning interface supports training a single model on multiple GPUs within a single compute node.  </p>\n\n<p>Disclosure: I work for NVIDIA.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 109067,
      "author_name": "alvaroosvaldo",
      "author_url": "",
      "post_date": "02/22/2016 23:57:27",
      "content": "<p>Hi Senecaur thanks for the answer.</p>\n\n<p>Yes, not only to train a neural neutwork but some image and data processing,  I'm trying to reduce the processing time in half. :D </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 109608,
      "author_name": "yilihome",
      "author_url": "",
      "post_date": "02/28/2016 15:46:47",
      "content": "<p>Even when not used in SLI, having two GPUs might help if you are training multiple models (for bagging with different seeds) or variations of the nnet structure. For instance, if you are using the Theano backend, you can simply run two parallel processes with device=gpu0 and device=gpu1 -- this basically halves my processing time...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 109634,
      "author_name": "alvaroosvaldo",
      "author_url": "",
      "post_date": "02/28/2016 23:19:57",
      "content": "<p>I see, and the NVIDIA CUDA toolkit have some tools para debug some memory issues. It's good.</p>\n\n<p>I still need to learn in NVIDIA courses. :O</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "109051": "Hi, I have not found the answer in the CUDA Zone, not even a consensus in the forums\r\n\r\n\r\nNVIDIA CUDA toolkit works with two or more GPUs in SLI ? \r\n\r\nIf works, what improvement I gain ?",
    "109053": "Hi Alvaro.  SLI is not used within CUDA programming, it is a technology related to the use of GPUs for graphics.  Am I right in assuming you are hoping to speed up deep neural network training using multiple GPUs?  If so, there are a number of deep learning frameworks that support multi-GPU training of a single model.  This can happen because each GPU is individually addressable within a CUDA application, so workload can be distributed across them.  For example, the version of Caffe that powers the [NVIDIA DIGITS][1] deep learning interface supports training a single model on multiple GPUs within a single compute node.  \r\n\r\nDisclosure: I work for NVIDIA.\r\n\r\n  [1]: https://developer.nvidia.com/digits",
    "109067": "Hi Senecaur thanks for the answer.\r\n\r\nYes, not only to train a neural neutwork but some image and data processing,  I'm trying to reduce the processing time in half. :D",
    "109608": "Even when not used in SLI, having two GPUs might help if you are training multiple models (for bagging with different seeds) or variations of the nnet structure. For instance, if you are using the Theano backend, you can simply run two parallel processes with device=gpu0 and device=gpu1 -- this basically halves my processing time...",
    "109634": "I see, and the NVIDIA CUDA toolkit have some tools para debug some memory issues. It's good.\r\n\r\nI still need to learn in NVIDIA courses. :O"
  },
  "source": "meta"
}