My model trained 1 epoch in 500 seconds on kaggle and 4000 secocnds on colab
\n\nEdit: 1st epoch is 4000s but 2nd and 3rd and after are 500s??
", "messages": [ { "id": "674822", "postDate": "11/17/2019 06:06:05", "content": "My model trained 1 epoch in 500 seconds on kaggle and 4000 secocnds on colab
\n\nEdit: 1st epoch is 4000s but 2nd and 3rd and after are 500s??
", "rawMarkdown": "My model trained 1 epoch in 500 seconds on kaggle and 4000 secocnds on colab\n\nEdit: 1st epoch is 4000s but 2nd and 3rd and after are 500s??", "votes": null }, { "id": "674823", "postDate": "11/17/2019 06:07:45", "content": "because kaggle uses a powerful gpu where colab uses a slow gpu for training your model
", "rawMarkdown": "because kaggle uses a powerful gpu where colab uses a slow gpu for training your model", "votes": null }, { "id": "674824", "postDate": "11/17/2019 06:09:55", "content": "check with !nvidia-smi, to check what GPU you got from Colab and Kaggle @cweed28
Kaggle's GPU is Tesla P100, and in colab, you may get Tesla P100 sometimes, but most of the time will be K80. However, the training time shouldn't have this much difference. Are you sure all parameters are the same?
", "rawMarkdown": "Kaggle's GPU is Tesla P100, and in colab, you may get Tesla P100 sometimes, but most of the time will be K80. However, the training time shouldn't have this much difference. Are you sure all parameters are the same?", "votes": null }, { "id": "674828", "postDate": "11/17/2019 06:13:03", "content": "Sun Nov 17 06:12:35 2019
\n+-----------------------------------------------------------------------------+\n| NVIDIA-SMI 430.50 Driver Version: 418.67 CUDA Version: 10.1 |\n|-------------------------------+----------------------+----------------------+\n| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |\n| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |\n|===============================+======================+======================|\n| 0 Tesla P100-PCIE... Off | 00000000:00:04.0 Off | 0 |\n| N/A 46C P0 33W / 250W | 265MiB / 16280MiB | 0% Default |\n+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+\n| Processes: GPU Memory |\n| GPU PID Type Process name Usage |\n|=============================================================================|\n+-----------------------------------------------------------------------------+
\n\nIt's very strange the first epoch it was 4000 seconds but the second epoch was 500 seconds. I don't know what is going on.
", "rawMarkdown": "Sun Nov 17 06:12:35 2019 \n+-----------------------------------------------------------------------------+\n| NVIDIA-SMI 430.50 Driver Version: 418.67 CUDA Version: 10.1 |\n|-------------------------------+----------------------+----------------------+\n| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |\n| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |\n|===============================+======================+======================|\n| 0 Tesla P100-PCIE... Off | 00000000:00:04.0 Off | 0 |\n| N/A 46C P0 33W / 250W | 265MiB / 16280MiB | 0% Default |\n+-------------------------------+----------------------+----------------------+\n \n+-----------------------------------------------------------------------------+\n| Processes: GPU Memory |\n| GPU PID Type Process name Usage |\n|=============================================================================|\n+-----------------------------------------------------------------------------+\n\nIt's very strange the first epoch it was 4000 seconds but the second epoch was 500 seconds. I don't know what is going on.", "votes": null }, { "id": "674831", "postDate": "11/17/2019 06:14:21", "content": "And do kaggle and colab both use p100?
", "rawMarkdown": "And do kaggle and colab both use p100?", "votes": null }, { "id": "674832", "postDate": "11/17/2019 06:14:33", "content": "You already getting P100..so there must be something different of your code
", "rawMarkdown": "You already getting P100..so there must be something different of your code", "votes": null }, { "id": "674835", "postDate": "11/17/2019 06:16:14", "content": "First epoch takes time sometimes
", "rawMarkdown": "First epoch takes time sometimes", "votes": null }, { "id": "674838", "postDate": "11/17/2019 06:17:38", "content": "He need to turn on the GPU to see the above information. So he must already turned the GPU on.
", "rawMarkdown": "He need to turn on the GPU to see the above information. So he must already turned the GPU on.", "votes": null }, { "id": "674900", "postDate": "11/17/2019 08:37:57", "content": "from your other comment, why is first epoch not slow on kaggle then?
", "rawMarkdown": "from your other comment, why is first epoch not slow on kaggle then?", "votes": null }, { "id": "674909", "postDate": "11/17/2019 09:13:46", "content": "Dear @cweed28 \ni used rishab's kernel like this one : https://www.kaggle.com/rishabhiitbhu/unet-starter-kernel-pytorch-lb-0-88 \nin google colab and i noticed first epoch took long time to finish then i asked rishab why 1st epoch is taking long time? he didn't said me the reason but just said 1st epoch takes long time.\ncolab is terrible mate,i recommend you paperspace\nin steel competition the model that i could train for 10-15 epochs in kaggle kernel i couldn't train that same model for than 5 epochs in colab,it is terribly slow
", "rawMarkdown": "Dear @cweed28 \ni used rishab's kernel like this one : https://www.kaggle.com/rishabhiitbhu/unet-starter-kernel-pytorch-lb-0-88 \nin google colab and i noticed first epoch took long time to finish then i asked rishab why 1st epoch is taking long time? he didn't said me the reason but just said 1st epoch takes long time.\ncolab is terrible mate,i recommend you paperspace\nin steel competition the model that i could train for 10-15 epochs in kaggle kernel i couldn't train that same model for than 5 epochs in colab,it is terribly slow", "votes": null } ], "comments": [ { "id": 674823, "author_name": "mobassir", "author_url": "", "post_date": "11/17/2019 06:07:45", "content": "because kaggle uses a powerful gpu where colab uses a slow gpu for training your model
", "votes": null, "replies": [ { "id": 674900, "author_name": "cweed28", "author_url": "", "post_date": "11/17/2019 08:37:57", "content": "from your other comment, why is first epoch not slow on kaggle then?
", "votes": null, "replies": [] }, { "id": 674909, "author_name": "mobassir", "author_url": "", "post_date": "11/17/2019 09:13:46", "content": "Dear @cweed28 \ni used rishab's kernel like this one : https://www.kaggle.com/rishabhiitbhu/unet-starter-kernel-pytorch-lb-0-88 \nin google colab and i noticed first epoch took long time to finish then i asked rishab why 1st epoch is taking long time? he didn't said me the reason but just said 1st epoch takes long time.\ncolab is terrible mate,i recommend you paperspace\nin steel competition the model that i could train for 10-15 epochs in kaggle kernel i couldn't train that same model for than 5 epochs in colab,it is terribly slow
", "votes": null, "replies": [] } ] }, { "id": 674824, "author_name": "seshurajup", "author_url": "", "post_date": "11/17/2019 06:09:55", "content": "check with !nvidia-smi, to check what GPU you got from Colab and Kaggle @cweed28
Sun Nov 17 06:12:35 2019
\n+-----------------------------------------------------------------------------+\n| NVIDIA-SMI 430.50 Driver Version: 418.67 CUDA Version: 10.1 |\n|-------------------------------+----------------------+----------------------+\n| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |\n| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |\n|===============================+======================+======================|\n| 0 Tesla P100-PCIE... Off | 00000000:00:04.0 Off | 0 |\n| N/A 46C P0 33W / 250W | 265MiB / 16280MiB | 0% Default |\n+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+\n| Processes: GPU Memory |\n| GPU PID Type Process name Usage |\n|=============================================================================|\n+-----------------------------------------------------------------------------+
\n\nIt's very strange the first epoch it was 4000 seconds but the second epoch was 500 seconds. I don't know what is going on.
", "votes": null, "replies": [] }, { "id": 674831, "author_name": "cweed28", "author_url": "", "post_date": "11/17/2019 06:14:21", "content": "And do kaggle and colab both use p100?
", "votes": null, "replies": [] }, { "id": 674832, "author_name": "xiejialun", "author_url": "", "post_date": "11/17/2019 06:14:33", "content": "You already getting P100..so there must be something different of your code
", "votes": null, "replies": [] }, { "id": 674835, "author_name": "mobassir", "author_url": "", "post_date": "11/17/2019 06:16:14", "content": "First epoch takes time sometimes
", "votes": null, "replies": [] }, { "id": 674838, "author_name": "xiejialun", "author_url": "", "post_date": "11/17/2019 06:17:38", "content": "He need to turn on the GPU to see the above information. So he must already turned the GPU on.
", "votes": null, "replies": [] } ] }, { "id": 674825, "author_name": "xiejialun", "author_url": "", "post_date": "11/17/2019 06:12:21", "content": "Kaggle's GPU is Tesla P100, and in colab, you may get Tesla P100 sometimes, but most of the time will be K80. However, the training time shouldn't have this much difference. Are you sure all parameters are the same?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "674822": "My model trained 1 epoch in 500 seconds on kaggle and 4000 secocnds on colab\n\nEdit: 1st epoch is 4000s but 2nd and 3rd and after are 500s??", "674823": "because kaggle uses a powerful gpu where colab uses a slow gpu for training your model", "674824": "check with `!nvidia-smi`, to check what GPU you got from Colab and Kaggle @cweed28", "674825": "Kaggle's GPU is Tesla P100, and in colab, you may get Tesla P100 sometimes, but most of the time will be K80. However, the training time shouldn't have this much difference. Are you sure all parameters are the same?", "674828": "Sun Nov 17 06:12:35 2019 \n+-----------------------------------------------------------------------------+\n| NVIDIA-SMI 430.50 Driver Version: 418.67 CUDA Version: 10.1 |\n|-------------------------------+----------------------+----------------------+\n| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |\n| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |\n|===============================+======================+======================|\n| 0 Tesla P100-PCIE... Off | 00000000:00:04.0 Off | 0 |\n| N/A 46C P0 33W / 250W | 265MiB / 16280MiB | 0% Default |\n+-------------------------------+----------------------+----------------------+\n \n+-----------------------------------------------------------------------------+\n| Processes: GPU Memory |\n| GPU PID Type Process name Usage |\n|=============================================================================|\n+-----------------------------------------------------------------------------+\n\nIt's very strange the first epoch it was 4000 seconds but the second epoch was 500 seconds. I don't know what is going on.", "674831": "And do kaggle and colab both use p100?", "674832": "You already getting P100..so there must be something different of your code", "674835": "First epoch takes time sometimes", "674838": "He need to turn on the GPU to see the above information. So he must already turned the GPU on.", "674900": "from your other comment, why is first epoch not slow on kaggle then?", "674909": "Dear @cweed28 \ni used rishab's kernel like this one : https://www.kaggle.com/rishabhiitbhu/unet-starter-kernel-pytorch-lb-0-88 \nin google colab and i noticed first epoch took long time to finish then i asked rishab why 1st epoch is taking long time? he didn't said me the reason but just said 1st epoch takes long time.\ncolab is terrible mate,i recommend you paperspace\nin steel competition the model that i could train for 10-15 epochs in kaggle kernel i couldn't train that same model for than 5 epochs in colab,it is terribly slow" }, "source": "meta" }