{
  "id": 40092,
  "title": "GPUs are hot at the eleventh hour!",
  "url": "/competitions/carvana-image-masking-challenge/discussion/40092",
  "author_name": "",
  "post_date": "2017-09-27T16:57:35.670710800Z",
  "votes": 5,
  "comment_count": 12,
  "views": 0,
  "content": "<p>There's still time! What are you guys doing in the final hours?</p>\n\n<p>Running inference like crazy here! </p>\n\n<p><img src=\"https://pbs.twimg.com/media/DKuK2FAW4AACBBR.jpg\" alt=\"8 GPUs\" title=\"\">\n<img src=\"https://pbs.twimg.com/media/DKuK4OMWkAA5a7k.jpg\" alt=\"2 GPUs\" title=\"\"></p>",
  "messages": [
    {
      "id": "224785",
      "postDate": "09/27/2017 16:57:35",
      "content": "<p>There's still time! What are you guys doing in the final hours?</p>\n\n<p>Running inference like crazy here! </p>\n\n<p><img src=\"https://pbs.twimg.com/media/DKuK2FAW4AACBBR.jpg\" alt=\"8 GPUs\" title=\"\">\n<img src=\"https://pbs.twimg.com/media/DKuK4OMWkAA5a7k.jpg\" alt=\"2 GPUs\" title=\"\"></p>",
      "rawMarkdown": "There's still time! What are you guys doing in the final hours?\n\nRunning inference like crazy here! \n\n![8 GPUs][1]\n![2 GPUs][2]\n\n\n\n\n  [1]: https://pbs.twimg.com/media/DKuK2FAW4AACBBR.jpg\n  [2]: https://pbs.twimg.com/media/DKuK4OMWkAA5a7k.jpg",
      "votes": null
    },
    {
      "id": "224788",
      "postDate": "09/27/2017 17:01:43",
      "content": "<p>you will never take the maximum from your GPUs if your X-server is still running ;)</p>",
      "rawMarkdown": "you will never take the maximum from your GPUs if your X-server is still running ;)",
      "votes": null
    },
    {
      "id": "224791",
      "postDate": "09/27/2017 17:04:24",
      "content": "<p>Thanks! I'll investigate how to use the onboard HDMI for X-Server, etc. instead of nvidia. If you have any pointers feel free to share...</p>",
      "rawMarkdown": "Thanks! I'll investigate how to use the onboard HDMI for X-Server, etc. instead of nvidia. If you have any pointers feel free to share...",
      "votes": null
    },
    {
      "id": "224792",
      "postDate": "09/27/2017 17:11:15",
      "content": "<p>It was mostly a joke, but there is also a real point that X Server takes resources from your GPU, so it may run a bit slower. At least in my case it was always a bit helpful to shut Xorg down. Especially in case of a single GPU only</p>",
      "rawMarkdown": "It was mostly a joke, but there is also a real point that X Server takes resources from your GPU, so it may run a bit slower. At least in my case it was always a bit helpful to shut Xorg down. Especially in case of a single GPU only",
      "votes": null
    },
    {
      "id": "224794",
      "postDate": "09/27/2017 17:16:18",
      "content": "<p>Not a joke at all, at least not in terms of performance (X server should be idle) but memory wise it takes ~300 Mbs... and incidentally I had to use a smaller batch size in that GPU vs. the other so there's a clear and tangible benefit of running the DL GPUs in headless mode.</p>",
      "rawMarkdown": "Not a joke at all, at least not in terms of performance (X server should be idle) but memory wise it takes ~300 Mbs... and incidentally I had to use a smaller batch size in that GPU vs. the other so there's a clear and tangible benefit of running the DL GPUs in headless mode.",
      "votes": null
    },
    {
      "id": "224800",
      "postDate": "09/27/2017 17:30:31",
      "content": "<p>Plan for tomorrow:\n<a href=\"https://devtalk.nvidia.com/default/topic/991849/-solved-run-cuda-on-dedicated-nvidia-gpu-while-connecting-monitors-to-intel-hd-graphics-is-this-possible-/\">https://devtalk.nvidia.com/default/topic/991849/-solved-run-cuda-on-dedicated-nvidia-gpu-while-connecting-monitors-to-intel-hd-graphics-is-this-possible-/</a></p>\n\n<p>:-)</p>",
      "rawMarkdown": "Plan for tomorrow:\nhttps://devtalk.nvidia.com/default/topic/991849/-solved-run-cuda-on-dedicated-nvidia-gpu-while-connecting-monitors-to-intel-hd-graphics-is-this-possible-/\n\n:-)",
      "votes": null
    },
    {
      "id": "224808",
      "postDate": "09/27/2017 17:56:17",
      "content": "<p>You have 8 gpus???? :O</p>",
      "rawMarkdown": "You have 8 gpus???? :O",
      "votes": null
    },
    {
      "id": "224810",
      "postDate": "09/27/2017 18:00:11",
      "content": "<p>Nah. Just 2 x 1080 Tis.... the other 8 GPUs are K80s from a cloud server. K80s are super-slow. \nIn this project (and others I've run) 1 x 1080 Ti is ~20% faster than 4 x K80s.</p>",
      "rawMarkdown": "Nah. Just 2 x 1080 Tis.... the other 8 GPUs are K80s from a cloud server. K80s are super-slow. \nIn this project (and others I've run) 1 x 1080 Ti is ~20% faster than 4 x K80s.",
      "votes": null
    },
    {
      "id": "224839",
      "postDate": "09/27/2017 19:01:20",
      "content": "<p>Cool story.  1 x 1080 Ti (10.6 TFlops SPFP) is faster than 4 x K80s (8.74 x4 TFlops SPFP) ???</p>",
      "rawMarkdown": "Cool story.  1 x 1080 Ti (10.6 TFlops SPFP) is faster than 4 x K80s (8.74 x4 TFlops SPFP) ???",
      "votes": null
    },
    {
      "id": "224849",
      "postDate": "09/27/2017 19:16:46",
      "content": "<p>That's my experience and to be honest I am very surprised... but I admit I have not spent a lot of time tuning the K80 setup. </p>\n\n<p>Some benchmarks: <a href=\"https://github.com/szilard/benchm-dl/blob/master/keras_backend.md\">https://github.com/szilard/benchm-dl/blob/master/keras_backend.md</a></p>",
      "rawMarkdown": "That's my experience and to be honest I am very surprised... but I admit I have not spent a lot of time tuning the K80 setup. \n\nSome benchmarks: https://github.com/szilard/benchm-dl/blob/master/keras_backend.md",
      "votes": null
    },
    {
      "id": "224857",
      "postDate": "09/27/2017 19:31:03",
      "content": "<p>Maybe there was bottleneck in CPU?</p>",
      "rawMarkdown": "Maybe there was bottleneck in CPU?",
      "votes": null
    },
    {
      "id": "224860",
      "postDate": "09/27/2017 19:35:02",
      "content": "<p>The 8 x K80s had 16 core CPU and the 2 x 1080 Ti had 4 core CPU. In both cases <code>nvidia-smi</code> reports sustained ~99% GPU usage. but now that I think about it I could have increased batch size on the K80s. It's too late now...</p>",
      "rawMarkdown": "The 8 x K80s had 16 core CPU and the 2 x 1080 Ti had 4 core CPU. In both cases `nvidia-smi` reports sustained ~99% GPU usage. but now that I think about it I could have increased batch size on the K80s. It's too late now...",
      "votes": null
    },
    {
      "id": "224900",
      "postDate": "09/27/2017 22:06:49",
      "content": "<p>Running inference on a p2.16  because I forgot to set checkpoints D:\n<img src=\"http://s2.quickmeme.com/img/38/382f90313fde253278147683a666aa27c79a4cdaa57791c383b1ad1f01d4bd26.jpg\" alt=\"whyaws\" title=\"\"></p>",
      "rawMarkdown": "Running inference on a p2.16  because I forgot to set checkpoints D:\n![whyaws][1]\n\n\n  [1]: http://s2.quickmeme.com/img/38/382f90313fde253278147683a666aa27c79a4cdaa57791c383b1ad1f01d4bd26.jpg",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 224788,
      "author_name": "truepk",
      "author_url": "",
      "post_date": "09/27/2017 17:01:43",
      "content": "<p>you will never take the maximum from your GPUs if your X-server is still running ;)</p>",
      "votes": null,
      "replies": [
        {
          "id": 224791,
          "author_name": "antorsae",
          "author_url": "",
          "post_date": "09/27/2017 17:04:24",
          "content": "<p>Thanks! I'll investigate how to use the onboard HDMI for X-Server, etc. instead of nvidia. If you have any pointers feel free to share...</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 224792,
          "author_name": "truepk",
          "author_url": "",
          "post_date": "09/27/2017 17:11:15",
          "content": "<p>It was mostly a joke, but there is also a real point that X Server takes resources from your GPU, so it may run a bit slower. At least in my case it was always a bit helpful to shut Xorg down. Especially in case of a single GPU only</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 224794,
          "author_name": "antorsae",
          "author_url": "",
          "post_date": "09/27/2017 17:16:18",
          "content": "<p>Not a joke at all, at least not in terms of performance (X server should be idle) but memory wise it takes ~300 Mbs... and incidentally I had to use a smaller batch size in that GPU vs. the other so there's a clear and tangible benefit of running the DL GPUs in headless mode.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 224800,
          "author_name": "antorsae",
          "author_url": "",
          "post_date": "09/27/2017 17:30:31",
          "content": "<p>Plan for tomorrow:\n<a href=\"https://devtalk.nvidia.com/default/topic/991849/-solved-run-cuda-on-dedicated-nvidia-gpu-while-connecting-monitors-to-intel-hd-graphics-is-this-possible-/\">https://devtalk.nvidia.com/default/topic/991849/-solved-run-cuda-on-dedicated-nvidia-gpu-while-connecting-monitors-to-intel-hd-graphics-is-this-possible-/</a></p>\n\n<p>:-)</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 224808,
      "author_name": "ironbar",
      "author_url": "",
      "post_date": "09/27/2017 17:56:17",
      "content": "<p>You have 8 gpus???? :O</p>",
      "votes": null,
      "replies": [
        {
          "id": 224810,
          "author_name": "antorsae",
          "author_url": "",
          "post_date": "09/27/2017 18:00:11",
          "content": "<p>Nah. Just 2 x 1080 Tis.... the other 8 GPUs are K80s from a cloud server. K80s are super-slow. \nIn this project (and others I've run) 1 x 1080 Ti is ~20% faster than 4 x K80s.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 224839,
          "author_name": "asanakoev",
          "author_url": "",
          "post_date": "09/27/2017 19:01:20",
          "content": "<p>Cool story.  1 x 1080 Ti (10.6 TFlops SPFP) is faster than 4 x K80s (8.74 x4 TFlops SPFP) ???</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 224849,
          "author_name": "antorsae",
          "author_url": "",
          "post_date": "09/27/2017 19:16:46",
          "content": "<p>That's my experience and to be honest I am very surprised... but I admit I have not spent a lot of time tuning the K80 setup. </p>\n\n<p>Some benchmarks: <a href=\"https://github.com/szilard/benchm-dl/blob/master/keras_backend.md\">https://github.com/szilard/benchm-dl/blob/master/keras_backend.md</a></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 224857,
          "author_name": "ceperaang",
          "author_url": "",
          "post_date": "09/27/2017 19:31:03",
          "content": "<p>Maybe there was bottleneck in CPU?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 224860,
          "author_name": "antorsae",
          "author_url": "",
          "post_date": "09/27/2017 19:35:02",
          "content": "<p>The 8 x K80s had 16 core CPU and the 2 x 1080 Ti had 4 core CPU. In both cases <code>nvidia-smi</code> reports sustained ~99% GPU usage. but now that I think about it I could have increased batch size on the K80s. It's too late now...</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 224900,
      "author_name": "stevenknguyen",
      "author_url": "",
      "post_date": "09/27/2017 22:06:49",
      "content": "<p>Running inference on a p2.16  because I forgot to set checkpoints D:\n<img src=\"http://s2.quickmeme.com/img/38/382f90313fde253278147683a666aa27c79a4cdaa57791c383b1ad1f01d4bd26.jpg\" alt=\"whyaws\" title=\"\"></p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "224785": "There's still time! What are you guys doing in the final hours?\n\nRunning inference like crazy here! \n\n![8 GPUs][1]\n![2 GPUs][2]\n\n\n\n\n  [1]: https://pbs.twimg.com/media/DKuK2FAW4AACBBR.jpg\n  [2]: https://pbs.twimg.com/media/DKuK4OMWkAA5a7k.jpg",
    "224788": "you will never take the maximum from your GPUs if your X-server is still running ;)",
    "224791": "Thanks! I'll investigate how to use the onboard HDMI for X-Server, etc. instead of nvidia. If you have any pointers feel free to share...",
    "224792": "It was mostly a joke, but there is also a real point that X Server takes resources from your GPU, so it may run a bit slower. At least in my case it was always a bit helpful to shut Xorg down. Especially in case of a single GPU only",
    "224794": "Not a joke at all, at least not in terms of performance (X server should be idle) but memory wise it takes ~300 Mbs... and incidentally I had to use a smaller batch size in that GPU vs. the other so there's a clear and tangible benefit of running the DL GPUs in headless mode.",
    "224800": "Plan for tomorrow:\nhttps://devtalk.nvidia.com/default/topic/991849/-solved-run-cuda-on-dedicated-nvidia-gpu-while-connecting-monitors-to-intel-hd-graphics-is-this-possible-/\n\n:-)",
    "224808": "You have 8 gpus???? :O",
    "224810": "Nah. Just 2 x 1080 Tis.... the other 8 GPUs are K80s from a cloud server. K80s are super-slow. \nIn this project (and others I've run) 1 x 1080 Ti is ~20% faster than 4 x K80s.",
    "224839": "Cool story.  1 x 1080 Ti (10.6 TFlops SPFP) is faster than 4 x K80s (8.74 x4 TFlops SPFP) ???",
    "224849": "That's my experience and to be honest I am very surprised... but I admit I have not spent a lot of time tuning the K80 setup. \n\nSome benchmarks: https://github.com/szilard/benchm-dl/blob/master/keras_backend.md",
    "224857": "Maybe there was bottleneck in CPU?",
    "224860": "The 8 x K80s had 16 core CPU and the 2 x 1080 Ti had 4 core CPU. In both cases `nvidia-smi` reports sustained ~99% GPU usage. but now that I think about it I could have increased batch size on the K80s. It's too late now...",
    "224900": "Running inference on a p2.16  because I forgot to set checkpoints D:\n![whyaws][1]\n\n\n  [1]: http://s2.quickmeme.com/img/38/382f90313fde253278147683a666aa27c79a4cdaa57791c383b1ad1f01d4bd26.jpg"
  },
  "source": "meta"
}