--Update--
\nTraining speed comes back in this week and GPU spec is almost no different. I don't know why, but problem goes away…
I run a kaggle notebook which I made before, and found xgboost inference with GPU becomes very slow and result got worse despite setting seed, comparing to previous run (a week ago). I'm sure that condition is totally same because I just fork the notebook and run it.
\nMy guess about inference speed is different spec gpu might be allocated every week, similar to google colab GPU. But this is just my guess. Anyone knows why?
", "messages": [ { "id": "1102818", "postDate": "12/05/2020 10:53:19", "content": "--Update--
\nTraining speed comes back in this week and GPU spec is almost no different. I don't know why, but problem goes away…
I run a kaggle notebook which I made before, and found xgboost inference with GPU becomes very slow and result got worse despite setting seed, comparing to previous run (a week ago). I'm sure that condition is totally same because I just fork the notebook and run it.
\nMy guess about inference speed is different spec gpu might be allocated every week, similar to google colab GPU. But this is just my guess. Anyone knows why?
", "rawMarkdown": "Update--\nTraining speed comes back in this week and GPU spec is almost no different. I don't know why, but problem goes away...\n\n--------\nI run a kaggle notebook which I made before, and found xgboost inference with GPU becomes very slow and result got worse despite setting seed, comparing to previous run (a week ago). I'm sure that condition is totally same because I just fork the notebook and run it.\n\nMy guess about inference speed is different spec gpu might be allocated every week, similar to google colab GPU. But this is just my guess. Anyone knows why?", "votes": null }, { "id": "1102858", "postDate": "12/05/2020 12:02:39", "content": "Kaggle introduced a dynamic weekly GPU quota allocation since some time.
", "rawMarkdown": "Kaggle introduced a dynamic weekly GPU quota allocation since some time.", "votes": null }, { "id": "1102875", "postDate": "12/05/2020 12:33:43", "content": "@stecasasso
\nYour \"quota allocation\" means different spec GPU? (I know hours of quota is different every week)
\nIf you know related page, I'd like to know it
I posted a similiar query sometime back when I noticed my exact same code having different run times (and as a result I was have timeouts on my notebooks)
\nI was told that have only a single spec GPU. Not sure where the variability comes from, but it is probably a temporary thing. What %age variation are you observing ?
", "rawMarkdown": "I posted a similiar query sometime back when I noticed my exact same code having different run times (and as a result I was have timeouts on my notebooks)\n\nI was told that have only a single spec GPU. Not sure where the variability comes from, but it is probably a temporary thing. What %age variation are you observing ?", "votes": null }, { "id": "1103525", "postDate": "12/06/2020 02:21:46", "content": "@watzisname
\nIt took only about 1min to train in previous time, while this time took about 2h. This difference becomes heavy burden for me. It is questionable for GPU to work, but GPU time is properly decreasing and cudf works so GPU seems in use. Despite temporary, it is fatal to wait next week..
Sorry, I think I misunderstood your question. No, I am not aware of different type of GPU. I guess one can print the device name at runtime to check
", "rawMarkdown": "Sorry, I think I misunderstood your question. No, I am not aware of different type of GPU. I guess one can print the device name at runtime to check", "votes": null }, { "id": "1103848", "postDate": "12/06/2020 11:12:34", "content": "@stecasasso
\nOK. I haven't checked device name, so I'll try to do $ nvidia-smi. Thank you!
you can also do:
\n\nimport torch; torch.cuda.get_device_name()\n
Thanks, I'll also try it
\n===update===
\nI'll leave it
Tesla P100-PCIE-16GB
\nSun Dec 6 14:17:21 2020
\n+-----------------------------------------------------------------------------+
\n| NVIDIA-SMI 450.51.06 Driver Version: 450.51.06 CUDA Version: 11.0 |
\n|-------------------------------+----------------------+----------------------+
\n| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
\n| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
\n| | | MIG M. |
\n|===============================+======================+======================|
\n| 0 Tesla P100-PCIE… Off | 00000000:00:04.0 Off | 0 |
\n| N/A 36C P0 26W / 250W | 2MiB / 16280MiB | 0% Default |
\n| | | N/A |
\n+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
\n| Processes: |
\n| GPU GI CI PID Type Process name GPU Memory |
\n| ID ID Usage |
\n|=============================================================================|
\n| No running processes found |
\n+-----------------------------------------------------------------------------+
Training speed comes back in this week and GPU spec is almost no different. I don't know why, but problem goes away.
\nTesla P100-PCIE-16GB
\nSat Dec 12 03:23:07 2020
\n+-----------------------------------------------------------------------------+
\n| NVIDIA-SMI 450.51.06 Driver Version: 450.51.06 CUDA Version: 11.0 |
\n|-------------------------------+----------------------+----------------------+
\n| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
\n| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
\n| | | MIG M. |
\n|===============================+======================+======================|
\n| 0 Tesla P100-PCIE… Off | 00000000:00:04.0 Off | 0 |
\n| N/A 38C P0 28W / 250W | 2MiB / 16280MiB | 0% Default |
\n| | | N/A |
\n+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
\n| Processes: |
\n| GPU GI CI PID Type Process name GPU Memory |
\n| ID ID Usage |
\n|=============================================================================|
\n| No running processes found |
\n+-----------------------------------------------------------------------------+
Kaggle introduced a dynamic weekly GPU quota allocation since some time.
", "votes": null, "replies": [ { "id": 1102875, "author_name": "ant3ng", "author_url": "", "post_date": "12/05/2020 12:33:43", "content": "@stecasasso
\nYour \"quota allocation\" means different spec GPU? (I know hours of quota is different every week)
\nIf you know related page, I'd like to know it
Sorry, I think I misunderstood your question. No, I am not aware of different type of GPU. I guess one can print the device name at runtime to check
", "votes": null, "replies": [] }, { "id": 1103848, "author_name": "ant3ng", "author_url": "", "post_date": "12/06/2020 11:12:34", "content": "@stecasasso
\nOK. I haven't checked device name, so I'll try to do $ nvidia-smi. Thank you!
you can also do:
\n\nimport torch; torch.cuda.get_device_name()\n
Thanks, I'll also try it
\n===update===
\nI'll leave it
Tesla P100-PCIE-16GB
\nSun Dec 6 14:17:21 2020
\n+-----------------------------------------------------------------------------+
\n| NVIDIA-SMI 450.51.06 Driver Version: 450.51.06 CUDA Version: 11.0 |
\n|-------------------------------+----------------------+----------------------+
\n| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
\n| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
\n| | | MIG M. |
\n|===============================+======================+======================|
\n| 0 Tesla P100-PCIE… Off | 00000000:00:04.0 Off | 0 |
\n| N/A 36C P0 26W / 250W | 2MiB / 16280MiB | 0% Default |
\n| | | N/A |
\n+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
\n| Processes: |
\n| GPU GI CI PID Type Process name GPU Memory |
\n| ID ID Usage |
\n|=============================================================================|
\n| No running processes found |
\n+-----------------------------------------------------------------------------+
Training speed comes back in this week and GPU spec is almost no different. I don't know why, but problem goes away.
\nTesla P100-PCIE-16GB
\nSat Dec 12 03:23:07 2020
\n+-----------------------------------------------------------------------------+
\n| NVIDIA-SMI 450.51.06 Driver Version: 450.51.06 CUDA Version: 11.0 |
\n|-------------------------------+----------------------+----------------------+
\n| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
\n| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
\n| | | MIG M. |
\n|===============================+======================+======================|
\n| 0 Tesla P100-PCIE… Off | 00000000:00:04.0 Off | 0 |
\n| N/A 38C P0 28W / 250W | 2MiB / 16280MiB | 0% Default |
\n| | | N/A |
\n+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
\n| Processes: |
\n| GPU GI CI PID Type Process name GPU Memory |
\n| ID ID Usage |
\n|=============================================================================|
\n| No running processes found |
\n+-----------------------------------------------------------------------------+
I posted a similiar query sometime back when I noticed my exact same code having different run times (and as a result I was have timeouts on my notebooks)
\nI was told that have only a single spec GPU. Not sure where the variability comes from, but it is probably a temporary thing. What %age variation are you observing ?
", "votes": null, "replies": [ { "id": 1103525, "author_name": "ant3ng", "author_url": "", "post_date": "12/06/2020 02:21:46", "content": "@watzisname
\nIt took only about 1min to train in previous time, while this time took about 2h. This difference becomes heavy burden for me. It is questionable for GPU to work, but GPU time is properly decreasing and cudf works so GPU seems in use. Despite temporary, it is fatal to wait next week..