{
  "id": 196840,
  "title": "LGBM model taking more time to train on GPU than CPU",
  "url": "/competitions/riiid-test-answer-prediction/discussion/196840",
  "author_name": "Popeyes_Queen",
  "post_date": "2020-11-13T02:37:00.689000",
  "votes": 7,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I recently found that it takes more time to train a LGBM model on GPU for each round than it does on CPU under the same setting. </p>\n<p>It just seems counterintuitive to me as I expected GPU to do calculation faster.</p>\n<p>Also on GPU, the model trained for more rounds before early stopping. My guess is that it has to do with random initialization of weights, and LGBM on GPU setting just happens to converge slower.</p>\n<p>I'm quite new to ML and Kaggle environment. Could anyone be kind enough to share some thoughts?</p>",
  "messages": [
    {
      "id": 1076887,
      "postDate": "2020-11-13T02:37:00.690Z",
      "content": "<p>I recently found that it takes more time to train a LGBM model on GPU for each round than it does on CPU under the same setting. </p>\n<p>It just seems counterintuitive to me as I expected GPU to do calculation faster.</p>\n<p>Also on GPU, the model trained for more rounds before early stopping. My guess is that it has to do with random initialization of weights, and LGBM on GPU setting just happens to converge slower.</p>\n<p>I'm quite new to ML and Kaggle environment. Could anyone be kind enough to share some thoughts?</p>",
      "rawMarkdown": "I recently found that it takes more time to train a LGBM model on GPU for each round than it does on CPU under the same setting. \n\nIt just seems counterintuitive to me as I expected GPU to do calculation faster.\n\nAlso on GPU, the model trained for more rounds before early stopping. My guess is that it has to do with random initialization of weights, and LGBM on GPU setting just happens to converge slower.\n\nI'm quite new to ML and Kaggle environment. Could anyone be kind enough to share some thoughts?",
      "votes": 7
    },
    {
      "id": 1077172,
      "postDate": "2020-11-13T09:41:49.057Z",
      "content": "<p>I am not so surprised to be honest… There are some caveats on using GPU with lightgbm, see:   <br>\n<a href=\"https://lightgbm.readthedocs.io/en/latest/GPU-Performance.html#how-to-achieve-good-speedup-on-gpu\" target=\"_blank\">https://lightgbm.readthedocs.io/en/latest/GPU-Performance.html#how-to-achieve-good-speedup-on-gpu</a></p>\n<p>I personally never managed to get a speedup on GPU compared to to my 8 cores machine using lightgbm.</p>",
      "rawMarkdown": "I am not so surprised to be honest... There are some caveats on using GPU with lightgbm, see:   \nhttps://lightgbm.readthedocs.io/en/latest/GPU-Performance.html#how-to-achieve-good-speedup-on-gpu\n\nI personally never managed to get a speedup on GPU compared to to my 8 cores machine using lightgbm."
    },
    {
      "id": 1577011,
      "postDate": "2021-11-09T17:09:11.177Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 1077172,
      "author_name": "bluetrain",
      "author_url": "",
      "post_date": "2020-11-13T09:41:49.057000",
      "content": "<p>I am not so surprised to be honest… There are some caveats on using GPU with lightgbm, see:   <br>\n<a href=\"https://lightgbm.readthedocs.io/en/latest/GPU-Performance.html#how-to-achieve-good-speedup-on-gpu\" target=\"_blank\">https://lightgbm.readthedocs.io/en/latest/GPU-Performance.html#how-to-achieve-good-speedup-on-gpu</a></p>\n<p>I personally never managed to get a speedup on GPU compared to to my 8 cores machine using lightgbm.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1577011,
      "author_name": "",
      "author_url": "",
      "post_date": "2021-11-09T17:09:11.177000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1076887": "I recently found that it takes more time to train a LGBM model on GPU for each round than it does on CPU under the same setting. \n\nIt just seems counterintuitive to me as I expected GPU to do calculation faster.\n\nAlso on GPU, the model trained for more rounds before early stopping. My guess is that it has to do with random initialization of weights, and LGBM on GPU setting just happens to converge slower.\n\nI'm quite new to ML and Kaggle environment. Could anyone be kind enough to share some thoughts?",
    "1077172": "I am not so surprised to be honest... There are some caveats on using GPU with lightgbm, see:   \nhttps://lightgbm.readthedocs.io/en/latest/GPU-Performance.html#how-to-achieve-good-speedup-on-gpu\n\nI personally never managed to get a speedup on GPU compared to to my 8 cores machine using lightgbm.",
    "1577011": ""
  }
}