TFRecordDataset for your data pipeline even for GPU here.\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1554871%2Fec719bdd48de66b00473e394a5156812%2Fkaggle_monitoring.png?generation=1609499291656543&alt=media\")
I'm new to Deep Learning, and it is my first time using kaggle kernel (actually first time ever training online), and even if I used GPU acceleration, training speed is still slow.
\nWith (150,150,3) * 64 batch_size input, tensorflow pretrained ResNet50V2(excluding top, freezing all layers) + 2 untrained fully connected layers (256 and 5) + some other layers (batchnorm, activation and dropout), it takes 300-400s to train each epoch on kaggle kernel. However, on my laptop with 1660ti + i5-9300h, it takes only 230-235s per epoch to train with exact same model and data (and same code except the paths to files).
\nTesla p100 is no doubt way better than a laptop 1660ti, but how can the fastest epoch on kaggle kernel be slower than the slowest epoch on my pc? Can someone point out possible problems?
\nThanks!
ps. it is not a temporary issue since I tried several times last three days, but it never gets fast, I even tried explicitly call with tf.device('/GPU:0'): nothing changed. And the gpu meter at the top is 0% at all time (maybe it is not even using gpu at all?)
\n and using tpu only gives me \"No registered 'PyFunc' OpKernel for 'CPU' devices compatible with node\"
update: I noticed that kernel cpu is always over 80 percent, maybe IO is making it slow? I am using flow_from_directory, which seems like how many people load data, if this is the problem, what is a better work around for this?
", "messages": [ { "id": "1134411", "postDate": "01/01/2021 09:26:40", "content": "I'm new to Deep Learning, and it is my first time using kaggle kernel (actually first time ever training online), and even if I used GPU acceleration, training speed is still slow.
\nWith (150,150,3) * 64 batch_size input, tensorflow pretrained ResNet50V2(excluding top, freezing all layers) + 2 untrained fully connected layers (256 and 5) + some other layers (batchnorm, activation and dropout), it takes 300-400s to train each epoch on kaggle kernel. However, on my laptop with 1660ti + i5-9300h, it takes only 230-235s per epoch to train with exact same model and data (and same code except the paths to files).
\nTesla p100 is no doubt way better than a laptop 1660ti, but how can the fastest epoch on kaggle kernel be slower than the slowest epoch on my pc? Can someone point out possible problems?
\nThanks!
ps. it is not a temporary issue since I tried several times last three days, but it never gets fast, I even tried explicitly call with tf.device('/GPU:0'): nothing changed. And the gpu meter at the top is 0% at all time (maybe it is not even using gpu at all?)
\n and using tpu only gives me \"No registered 'PyFunc' OpKernel for 'CPU' devices compatible with node\"
update: I noticed that kernel cpu is always over 80 percent, maybe IO is making it slow? I am using flow_from_directory, which seems like how many people load data, if this is the problem, what is a better work around for this?
", "rawMarkdown": "I'm new to Deep Learning, and it is my first time using kaggle kernel (actually first time ever training online), and even if I used GPU acceleration, training speed is still slow.\n\nWith (150,150,3) * 64 batch_size input, tensorflow pretrained ResNet50V2(excluding top, freezing all layers) + 2 untrained fully connected layers (256 and 5) + some other layers (batchnorm, activation and dropout), it takes 300-400s to train each epoch on kaggle kernel. However, on my **laptop** with 1660ti + i5-9300h, it takes only 230-235s per epoch to train with exact same model and data (and same code except the paths to files). \n\nTesla p100 is no doubt way better than a laptop 1660ti, but how can the fastest epoch on kaggle kernel be slower than the slowest epoch on my pc? Can someone point out possible problems?\nThanks!\n\nps. it is not a temporary issue since I tried several times last three days, but it never gets fast, I even tried explicitly call with tf.device('/GPU:0'): nothing changed. And the gpu meter at the top is 0% at all time (maybe it is not even using gpu at all?)\n and using tpu only gives me \"No registered 'PyFunc' OpKernel for 'CPU' devices compatible with node\"\n\nupdate: I noticed that kernel cpu is always over 80 percent, maybe IO is making it slow? I am using flow_from_directory, which seems like how many people load data, if this is the problem, what is a better work around for this?", "votes": null }, { "id": "1134510", "postDate": "01/01/2021 11:10:00", "content": "@mervynyang The Kaggle GPU kernels have a performance bottleneck due to the poor CPU performance. However, I feel like your situation is caused because you are not utilizing the GPU efficiently. For reference, I use PyTorch with 448x448 image size and a batch size of 16 with EfficientnetB4 and it takes around 800s per epoch on GPU. As shown in the image below, CPU usage is always above 80% and above 90% for GPU on my Kaggle kernel.
\nThe same setup with TPU and TFRecords using Tensorflow takes about 38s per epoch. Read about using TFRecords and TFRecordDataset for your data pipeline even for GPU here.
\n
Thank you so much! It seems like something's wrong with GPU, and no matter how I tune, it never works, I had to rewrite everything to tfrec + tpu, and now it runs fast!
", "rawMarkdown": "Thank you so much! It seems like something's wrong with GPU, and no matter how I tune, it never works, I had to rewrite everything to tfrec + tpu, and now it runs fast!", "votes": null }, { "id": "1136295", "postDate": "01/02/2021 23:36:50", "content": "Yeah, the IO can be huge bottleneck. Kaggle don't have SSD and the CPU are slightly slower, so when loading from the Hard Disk with lot of augmentation, it can be very slow if you are not prefetching properly.
", "rawMarkdown": "Yeah, the IO can be huge bottleneck. Kaggle don't have SSD and the CPU are slightly slower, so when loading from the Hard Disk with lot of augmentation, it can be very slow if you are not prefetching properly.", "votes": null } ], "comments": [ { "id": 1134510, "author_name": "yovinyahathugoda", "author_url": "", "post_date": "01/01/2021 11:10:00", "content": "@mervynyang The Kaggle GPU kernels have a performance bottleneck due to the poor CPU performance. However, I feel like your situation is caused because you are not utilizing the GPU efficiently. For reference, I use PyTorch with 448x448 image size and a batch size of 16 with EfficientnetB4 and it takes around 800s per epoch on GPU. As shown in the image below, CPU usage is always above 80% and above 90% for GPU on my Kaggle kernel.
\nThe same setup with TPU and TFRecords using Tensorflow takes about 38s per epoch. Read about using TFRecords and TFRecordDataset for your data pipeline even for GPU here.
\n
Thank you so much! It seems like something's wrong with GPU, and no matter how I tune, it never works, I had to rewrite everything to tfrec + tpu, and now it runs fast!
", "votes": null, "replies": [] } ] }, { "id": 1136295, "author_name": "louis925", "author_url": "", "post_date": "01/02/2021 23:36:50", "content": "Yeah, the IO can be huge bottleneck. Kaggle don't have SSD and the CPU are slightly slower, so when loading from the Hard Disk with lot of augmentation, it can be very slow if you are not prefetching properly.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1134411": "I'm new to Deep Learning, and it is my first time using kaggle kernel (actually first time ever training online), and even if I used GPU acceleration, training speed is still slow.\n\nWith (150,150,3) * 64 batch_size input, tensorflow pretrained ResNet50V2(excluding top, freezing all layers) + 2 untrained fully connected layers (256 and 5) + some other layers (batchnorm, activation and dropout), it takes 300-400s to train each epoch on kaggle kernel. However, on my **laptop** with 1660ti + i5-9300h, it takes only 230-235s per epoch to train with exact same model and data (and same code except the paths to files). \n\nTesla p100 is no doubt way better than a laptop 1660ti, but how can the fastest epoch on kaggle kernel be slower than the slowest epoch on my pc? Can someone point out possible problems?\nThanks!\n\nps. it is not a temporary issue since I tried several times last three days, but it never gets fast, I even tried explicitly call with tf.device('/GPU:0'): nothing changed. And the gpu meter at the top is 0% at all time (maybe it is not even using gpu at all?)\n and using tpu only gives me \"No registered 'PyFunc' OpKernel for 'CPU' devices compatible with node\"\n\nupdate: I noticed that kernel cpu is always over 80 percent, maybe IO is making it slow? I am using flow_from_directory, which seems like how many people load data, if this is the problem, what is a better work around for this?", "1134510": "mervynyang The Kaggle GPU kernels have a performance bottleneck due to the poor CPU performance. However, I feel like your situation is caused because you are not utilizing the GPU efficiently. For reference, I use PyTorch with 448x448 image size and a batch size of 16 with EfficientnetB4 and it takes around 800s per epoch on GPU. As shown in the image below, CPU usage is always above 80% and above 90% for GPU on my Kaggle kernel. \n\nThe same setup with TPU and TFRecords using Tensorflow takes about 38s per epoch. Read about using TFRecords and `TFRecordDataset` for your data pipeline even for GPU [here](https://www.tensorflow.org/tutorials/load_data/tfrecord).\n", "1135502": "Thank you so much! It seems like something's wrong with GPU, and no matter how I tune, it never works, I had to rewrite everything to tfrec + tpu, and now it runs fast!", "1136295": "Yeah, the IO can be huge bottleneck. Kaggle don't have SSD and the CPU are slightly slower, so when loading from the Hard Disk with lot of augmentation, it can be very slow if you are not prefetching properly." }, "source": "meta" }