WIth PyTorch with jpeg image take 30min on 1 epoch.\nWIth Tensorflow with TFRecord only 6min each epoch.\nAnyone know why?
", "messages": [ { "id": "924424", "postDate": "07/11/2020 12:18:52", "content": "WIth PyTorch with jpeg image take 30min on 1 epoch.\nWIth Tensorflow with TFRecord only 6min each epoch.\nAnyone know why?
", "rawMarkdown": "WIth PyTorch with jpeg image take 30min on 1 epoch.\nWIth Tensorflow with TFRecord only 6min each epoch.\nAnyone know why?", "votes": null }, { "id": "924446", "postDate": "07/11/2020 12:29:46", "content": "When you open a jpeg file your compute has to decode it. But I think that TFRecord-Files are faster to open because you don´t need to decode it. That´s why they are much bigger than the jpg files
", "rawMarkdown": "When you open a jpeg file your compute has to decode it. But I think that TFRecord-Files are faster to open because you don´t need to decode it. That´s why they are much bigger than the jpg files", "votes": null }, { "id": "924469", "postDate": "07/11/2020 12:49:36", "content": "Nice. Can we create something for Pytorch like TFRecord for everyone. That's cool
", "rawMarkdown": "Nice. Can we create something for Pytorch like TFRecord for everyone. That's cool", "votes": null }, { "id": "924637", "postDate": "07/11/2020 14:52:59", "content": "are you comparing performance between 2 tpu models?
", "rawMarkdown": "are you comparing performance between 2 tpu models?", "votes": null }, { "id": "924784", "postDate": "07/11/2020 16:19:57", "content": "yeah. 2 model with the same config
", "rawMarkdown": "yeah. 2 model with the same config", "votes": null }, { "id": "924861", "postDate": "07/11/2020 17:00:55", "content": "@doanquanvietnamca because pytorch tpu is very very slow which is using v2 tpu where tf tpu is super fast which is using v3.8 and have more memory so can do fast calculation(pytorch xla team is working hard to make torch xla fast like tf tpu,,wait for their next release which is expected to come before upcoming September)
", "rawMarkdown": "doanquanvietnamca because pytorch tpu is very very slow which is using v2 tpu where tf tpu is super fast which is using v3.8 and have more memory so can do fast calculation(pytorch xla team is working hard to make torch xla fast like tf tpu,,wait for their next release which is expected to come before upcoming September)", "votes": null }, { "id": "924867", "postDate": "07/11/2020 17:03:40", "content": "The jpeg inside the TFRecord is encoded in the same way as a JPEG file on disk. Both my JPEG Kaggle datasets and TFRecord Kaggle datasets are the same size. So there must be another reason why one is slower.
", "rawMarkdown": "The jpeg inside the TFRecord is encoded in the same way as a JPEG file on disk. Both my JPEG Kaggle datasets and TFRecord Kaggle datasets are the same size. So there must be another reason why one is slower.", "votes": null }, { "id": "924877", "postDate": "07/11/2020 17:11:10", "content": "I use tpu v3.8 on colab Pro. Don't think that XLA slower than TF
", "rawMarkdown": "I use tpu v3.8 on colab Pro. Don't think that XLA slower than TF", "votes": null }, { "id": "924883", "postDate": "07/11/2020 17:12:34", "content": "I use pytorch lighting TPU. Do you think it slowly than TF
", "rawMarkdown": "I use pytorch lighting TPU. Do you think it slowly than TF", "votes": null }, { "id": "924884", "postDate": "07/11/2020 17:12:45", "content": "it's showing v3.8 in colab pro but pytorch xla will use v2\nif you don't believe just search on google then,good day
", "rawMarkdown": "it's showing v3.8 in colab pro but pytorch xla will use v2\nif you don't believe just search on google then,good day", "votes": null }, { "id": "924895", "postDate": "07/11/2020 17:18:50", "content": "Nice! That's new to me! Thanks for that
", "rawMarkdown": "Nice! That's new to me! Thanks for that", "votes": null }, { "id": "971417", "postDate": "08/15/2020 14:01:49", "content": "Thank you, I was wondering why my epoch trained with TPU (8 Cores) needs around 20 minutes while others with same model and same data train with Tensorflow TPU (also 8 Cores) only need 2 minutes for an epoch..
\nI hope the PyTorch XLA Team will do it!
", "rawMarkdown": "Thank you, I was wondering why my epoch trained with TPU (8 Cores) needs around 20 minutes while others with same model and same data train with Tensorflow TPU (also 8 Cores) only need 2 minutes for an epoch..\n\nI hope the PyTorch XLA Team will do it!", "votes": null }, { "id": "972537", "postDate": "08/16/2020 16:09:16", "content": "I tried pytorch lightning on colab pro. using 8 cores TPU is as slow as 1 GPU…
", "rawMarkdown": "I tried pytorch lightning on colab pro. using 8 cores TPU is as slow as 1 GPU...", "votes": null }, { "id": "972561", "postDate": "08/16/2020 16:31:23", "content": "Are you sure that you are using everything right?
\nPyTorch XLA is slow but should be atleast 3 times faster than GPU. For me it is faster, having 3 hours limitation on Kaggle makes it harder, so that using 9 hours GPU is in somecases better. Like the other comment mentioned, the PyTorch XLA team is working right now on a faster version which should come out this year.
", "rawMarkdown": "Are you sure that you are using everything right?\n\nPyTorch XLA is slow but should be atleast 3 times faster than GPU. For me it is faster, having 3 hours limitation on Kaggle makes it harder, so that using 9 hours GPU is in somecases better. Like the other comment mentioned, the PyTorch XLA team is working right now on a faster version which should come out this year.", "votes": null }, { "id": "972582", "postDate": "08/16/2020 16:52:39", "content": "there must be something i did wrong then. I would give it another try after the competition ended.
\ndo you mind proving me some benchmark so i know what to aim for? like the CNN arch, image size, and how long takes for 1 epoch with 8 TPU cores using pytorch etc. Thanks.
", "rawMarkdown": "there must be something i did wrong then. I would give it another try after the competition ended. \n\ndo you mind proving me some benchmark so i know what to aim for? like the CNN arch, image size, and how long takes for 1 epoch with 8 TPU cores using pytorch etc. Thanks.", "votes": null }, { "id": "972657", "postDate": "08/16/2020 17:56:34", "content": "I tried some benchmark by myself and had following results:
\nGPU:
\nImage-Size: 384x384
\nModel: EfficientNet-B4
\n1 Epoch took: ~50 Minutes with validation
PyTorch TPU (8 Cores):
\nImage-Size: 384x384
\nModel: EfficientNet-B4
\n1 Epoch took: ~20 Minutes with validation
I have seen equal settings for Tensorflow TPU and they only needed 2 minutes to train one epoch, which is definetly great…but unfortunately its not possible with PyTorch yet.
\nDue to 3 hours TPU limitation on Kaggle, GPU has 9 hours limitation on Kaggle and is sometimes more usefull because you can train more epochs eventhough it takes longer. I'm on a point where I run a notebook for each fold, so if I do 5-Fold-CV I have to create 5 notebook runs so I can train for 15 hours in total, 3 hours for each model. It works okay like that but seeing Tensorflow TPU notebooks with 40+ epochs looks even better.
", "rawMarkdown": "I tried some benchmark by myself and had following results:\n\nGPU:\nImage-Size: 384x384\nModel: EfficientNet-B4\n1 Epoch took: ~50 Minutes with validation\n\nPyTorch TPU (8 Cores):\nImage-Size: 384x384\nModel: EfficientNet-B4\n1 Epoch took: ~20 Minutes with validation\n\nI have seen equal settings for Tensorflow TPU and they only needed 2 minutes to train one epoch, which is definetly great...but unfortunately its not possible with PyTorch yet.\n\nDue to 3 hours TPU limitation on Kaggle, GPU has 9 hours limitation on Kaggle and is sometimes more usefull because you can train more epochs eventhough it takes longer. I'm on a point where I run a notebook for each fold, so if I do 5-Fold-CV I have to create 5 notebook runs so I can train for 15 hours in total, 3 hours for each model. It works okay like that but seeing Tensorflow TPU notebooks with 40+ epochs looks even better.", "votes": null }, { "id": "972670", "postDate": "08/16/2020 18:17:15", "content": "@aliabdin1 Thanks for the benchmark, Ali. really appreciate it.
\nthat's how i do it in colab as well, 5 notebooks at the same time, one for each fold. it's bit tedious to set it up, and still can't beat the speed of TF users.
\n2 mins in TF TPU vs 50 min pytorch, sounds depressing for pytorch users, makes me wondering where's the future for pytorch.
", "rawMarkdown": "aliabdin1 Thanks for the benchmark, Ali. really appreciate it. \n\nthat's how i do it in colab as well, 5 notebooks at the same time, one for each fold. it's bit tedious to set it up, and still can't beat the speed of TF users. \n\n2 mins in TF TPU vs 50 min pytorch, sounds depressing for pytorch users, makes me wondering where's the future for pytorch.", "votes": null }, { "id": "972810", "postDate": "08/16/2020 20:52:22", "content": "Yes its really a problem right now, but I can only refer to this comment:
\n\"[…] pytorch tpu is very very slow which is using v2 tpu where tf tpu is super fast which is using v3.8 and have more memory so can do fast calculation(pytorch xla team is working hard to make torch xla fast like tf tpu,wait for their next release which is expected to come before upcoming September)\"
\nSo it seems like its coming up this year still, lets hope atleast.
", "rawMarkdown": "Yes its really a problem right now, but I can only refer to this comment:\n\n\"[...] pytorch tpu is very very slow which is using v2 tpu where tf tpu is super fast which is using v3.8 and have more memory so can do fast calculation(pytorch xla team is working hard to make torch xla fast like tf tpu,wait for their next release which is expected to come before upcoming September)\"\n\nSo it seems like its coming up this year still, lets hope atleast.", "votes": null }, { "id": "973228", "postDate": "08/17/2020 07:24:54", "content": "great. i watched xla git repo! let's hope the best!
", "rawMarkdown": "great. i watched xla git repo! let's hope the best!", "votes": null } ], "comments": [ { "id": 972537, "author_name": "yimacs", "author_url": "", "post_date": "08/16/2020 16:09:16", "content": "I tried pytorch lightning on colab pro. using 8 cores TPU is as slow as 1 GPU…
", "votes": null, "replies": [ { "id": 972561, "author_name": "aliabdin1", "author_url": "", "post_date": "08/16/2020 16:31:23", "content": "Are you sure that you are using everything right?
\nPyTorch XLA is slow but should be atleast 3 times faster than GPU. For me it is faster, having 3 hours limitation on Kaggle makes it harder, so that using 9 hours GPU is in somecases better. Like the other comment mentioned, the PyTorch XLA team is working right now on a faster version which should come out this year.
", "votes": null, "replies": [] }, { "id": 972582, "author_name": "yimacs", "author_url": "", "post_date": "08/16/2020 16:52:39", "content": "there must be something i did wrong then. I would give it another try after the competition ended.
\ndo you mind proving me some benchmark so i know what to aim for? like the CNN arch, image size, and how long takes for 1 epoch with 8 TPU cores using pytorch etc. Thanks.
", "votes": null, "replies": [ { "id": 972657, "author_name": "aliabdin1", "author_url": "", "post_date": "08/16/2020 17:56:34", "content": "I tried some benchmark by myself and had following results:
\nGPU:
\nImage-Size: 384x384
\nModel: EfficientNet-B4
\n1 Epoch took: ~50 Minutes with validation
PyTorch TPU (8 Cores):
\nImage-Size: 384x384
\nModel: EfficientNet-B4
\n1 Epoch took: ~20 Minutes with validation
I have seen equal settings for Tensorflow TPU and they only needed 2 minutes to train one epoch, which is definetly great…but unfortunately its not possible with PyTorch yet.
\nDue to 3 hours TPU limitation on Kaggle, GPU has 9 hours limitation on Kaggle and is sometimes more usefull because you can train more epochs eventhough it takes longer. I'm on a point where I run a notebook for each fold, so if I do 5-Fold-CV I have to create 5 notebook runs so I can train for 15 hours in total, 3 hours for each model. It works okay like that but seeing Tensorflow TPU notebooks with 40+ epochs looks even better.
", "votes": null, "replies": [] }, { "id": 972670, "author_name": "yimacs", "author_url": "", "post_date": "08/16/2020 18:17:15", "content": "@aliabdin1 Thanks for the benchmark, Ali. really appreciate it.
\nthat's how i do it in colab as well, 5 notebooks at the same time, one for each fold. it's bit tedious to set it up, and still can't beat the speed of TF users.
\n2 mins in TF TPU vs 50 min pytorch, sounds depressing for pytorch users, makes me wondering where's the future for pytorch.
", "votes": null, "replies": [ { "id": 972810, "author_name": "aliabdin1", "author_url": "", "post_date": "08/16/2020 20:52:22", "content": "Yes its really a problem right now, but I can only refer to this comment:
\n\"[…] pytorch tpu is very very slow which is using v2 tpu where tf tpu is super fast which is using v3.8 and have more memory so can do fast calculation(pytorch xla team is working hard to make torch xla fast like tf tpu,wait for their next release which is expected to come before upcoming September)\"
\nSo it seems like its coming up this year still, lets hope atleast.
", "votes": null, "replies": [] }, { "id": 973228, "author_name": "yimacs", "author_url": "", "post_date": "08/17/2020 07:24:54", "content": "great. i watched xla git repo! let's hope the best!
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 924446, "author_name": "derinformatiker", "author_url": "", "post_date": "07/11/2020 12:29:46", "content": "When you open a jpeg file your compute has to decode it. But I think that TFRecord-Files are faster to open because you don´t need to decode it. That´s why they are much bigger than the jpg files
", "votes": null, "replies": [ { "id": 924469, "author_name": "doanquanvietnamca", "author_url": "", "post_date": "07/11/2020 12:49:36", "content": "Nice. Can we create something for Pytorch like TFRecord for everyone. That's cool
", "votes": null, "replies": [] }, { "id": 924867, "author_name": "cdeotte", "author_url": "", "post_date": "07/11/2020 17:03:40", "content": "The jpeg inside the TFRecord is encoded in the same way as a JPEG file on disk. Both my JPEG Kaggle datasets and TFRecord Kaggle datasets are the same size. So there must be another reason why one is slower.
", "votes": null, "replies": [] }, { "id": 924883, "author_name": "doanquanvietnamca", "author_url": "", "post_date": "07/11/2020 17:12:34", "content": "I use pytorch lighting TPU. Do you think it slowly than TF
", "votes": null, "replies": [] } ] }, { "id": 924637, "author_name": "mobassir", "author_url": "", "post_date": "07/11/2020 14:52:59", "content": "are you comparing performance between 2 tpu models?
", "votes": null, "replies": [ { "id": 924784, "author_name": "doanquanvietnamca", "author_url": "", "post_date": "07/11/2020 16:19:57", "content": "yeah. 2 model with the same config
", "votes": null, "replies": [] }, { "id": 924861, "author_name": "mobassir", "author_url": "", "post_date": "07/11/2020 17:00:55", "content": "@doanquanvietnamca because pytorch tpu is very very slow which is using v2 tpu where tf tpu is super fast which is using v3.8 and have more memory so can do fast calculation(pytorch xla team is working hard to make torch xla fast like tf tpu,,wait for their next release which is expected to come before upcoming September)
", "votes": null, "replies": [] }, { "id": 924877, "author_name": "doanquanvietnamca", "author_url": "", "post_date": "07/11/2020 17:11:10", "content": "I use tpu v3.8 on colab Pro. Don't think that XLA slower than TF
", "votes": null, "replies": [] }, { "id": 924884, "author_name": "mobassir", "author_url": "", "post_date": "07/11/2020 17:12:45", "content": "it's showing v3.8 in colab pro but pytorch xla will use v2\nif you don't believe just search on google then,good day
", "votes": null, "replies": [] }, { "id": 924895, "author_name": "doanquanvietnamca", "author_url": "", "post_date": "07/11/2020 17:18:50", "content": "Nice! That's new to me! Thanks for that
", "votes": null, "replies": [] }, { "id": 971417, "author_name": "aliabdin1", "author_url": "", "post_date": "08/15/2020 14:01:49", "content": "Thank you, I was wondering why my epoch trained with TPU (8 Cores) needs around 20 minutes while others with same model and same data train with Tensorflow TPU (also 8 Cores) only need 2 minutes for an epoch..
\nI hope the PyTorch XLA Team will do it!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "924424": "WIth PyTorch with jpeg image take 30min on 1 epoch.\nWIth Tensorflow with TFRecord only 6min each epoch.\nAnyone know why?", "924446": "When you open a jpeg file your compute has to decode it. But I think that TFRecord-Files are faster to open because you don´t need to decode it. That´s why they are much bigger than the jpg files", "924469": "Nice. Can we create something for Pytorch like TFRecord for everyone. That's cool", "924637": "are you comparing performance between 2 tpu models?", "924784": "yeah. 2 model with the same config", "924861": "doanquanvietnamca because pytorch tpu is very very slow which is using v2 tpu where tf tpu is super fast which is using v3.8 and have more memory so can do fast calculation(pytorch xla team is working hard to make torch xla fast like tf tpu,,wait for their next release which is expected to come before upcoming September)", "924867": "The jpeg inside the TFRecord is encoded in the same way as a JPEG file on disk. Both my JPEG Kaggle datasets and TFRecord Kaggle datasets are the same size. So there must be another reason why one is slower.", "924877": "I use tpu v3.8 on colab Pro. Don't think that XLA slower than TF", "924883": "I use pytorch lighting TPU. Do you think it slowly than TF", "924884": "it's showing v3.8 in colab pro but pytorch xla will use v2\nif you don't believe just search on google then,good day", "924895": "Nice! That's new to me! Thanks for that", "971417": "Thank you, I was wondering why my epoch trained with TPU (8 Cores) needs around 20 minutes while others with same model and same data train with Tensorflow TPU (also 8 Cores) only need 2 minutes for an epoch..\n\nI hope the PyTorch XLA Team will do it!", "972537": "I tried pytorch lightning on colab pro. using 8 cores TPU is as slow as 1 GPU...", "972561": "Are you sure that you are using everything right?\n\nPyTorch XLA is slow but should be atleast 3 times faster than GPU. For me it is faster, having 3 hours limitation on Kaggle makes it harder, so that using 9 hours GPU is in somecases better. Like the other comment mentioned, the PyTorch XLA team is working right now on a faster version which should come out this year.", "972582": "there must be something i did wrong then. I would give it another try after the competition ended. \n\ndo you mind proving me some benchmark so i know what to aim for? like the CNN arch, image size, and how long takes for 1 epoch with 8 TPU cores using pytorch etc. Thanks.", "972657": "I tried some benchmark by myself and had following results:\n\nGPU:\nImage-Size: 384x384\nModel: EfficientNet-B4\n1 Epoch took: ~50 Minutes with validation\n\nPyTorch TPU (8 Cores):\nImage-Size: 384x384\nModel: EfficientNet-B4\n1 Epoch took: ~20 Minutes with validation\n\nI have seen equal settings for Tensorflow TPU and they only needed 2 minutes to train one epoch, which is definetly great...but unfortunately its not possible with PyTorch yet.\n\nDue to 3 hours TPU limitation on Kaggle, GPU has 9 hours limitation on Kaggle and is sometimes more usefull because you can train more epochs eventhough it takes longer. I'm on a point where I run a notebook for each fold, so if I do 5-Fold-CV I have to create 5 notebook runs so I can train for 15 hours in total, 3 hours for each model. It works okay like that but seeing Tensorflow TPU notebooks with 40+ epochs looks even better.", "972670": "aliabdin1 Thanks for the benchmark, Ali. really appreciate it. \n\nthat's how i do it in colab as well, 5 notebooks at the same time, one for each fold. it's bit tedious to set it up, and still can't beat the speed of TF users. \n\n2 mins in TF TPU vs 50 min pytorch, sounds depressing for pytorch users, makes me wondering where's the future for pytorch.", "972810": "Yes its really a problem right now, but I can only refer to this comment:\n\n\"[...] pytorch tpu is very very slow which is using v2 tpu where tf tpu is super fast which is using v3.8 and have more memory so can do fast calculation(pytorch xla team is working hard to make torch xla fast like tf tpu,wait for their next release which is expected to come before upcoming September)\"\n\nSo it seems like its coming up this year still, lets hope atleast.", "973228": "great. i watched xla git repo! let's hope the best!" }, "source": "meta" }