![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2909429%2Fa0ba25a9aca43231af5499156e009095%2FUnbenannt.PNG?generation=1595079101158278&alt=media\")
Hi everyone,
\n\nI enabled GPU acceleration and computation on GPU device:
\n\n
However, training is super slow and the GPU is not utlized:
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2909429%2Fcbde38ec1b43f64a979a22c912252bc4%2FUnbenannt.PNG?generation=1595079305644850&alt=media\")
Is there some setting i missed?
", "messages": [ { "id": 934701, "postDate": "2020-07-18T16:53:51.280Z", "content": "From the screenshot, you are using most of your GPU memory, so your software is utilizing the GPU. But your CPU is at 100+%. That usually means you are trying to do too much in your data loading stage and the CPU cannot feed the GPU fast enough. Have you pre-processed your data? Best to do as much processing of the images outside of your data loading. Resize, normalize, crop, etc. and store the images on disk. Better yet, use TFRecords, which are even faster to read in.
\n\n-Rich
", "rawMarkdown": "From the screenshot, you are using most of your GPU memory, so your software is utilizing the GPU. But your CPU is at 100+%. That usually means you are trying to do too much in your data loading stage and the CPU cannot feed the GPU fast enough. Have you pre-processed your data? Best to do as much processing of the images outside of your data loading. Resize, normalize, crop, etc. and store the images on disk. Better yet, use TFRecords, which are even faster to read in.\n\n-Rich", "replies": [ { "id": 934919, "postDate": "2020-07-18T22:43:21.700Z", "content": "Hi Rich, thank you for your response. To be honest I didnt even noticed the GPU memory usage (I was confused by the empty usage bar).\nThis is how I load the data:\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2909429%2Fbac3a3acb850a9e009b70b7153992840%2FUnbenannt.PNG?generation=1595112160127616&alt=media\")
with…\ndef readimg(imagename, ):\n imgpath = TRAINIMGDIR + imagename + \".jpg\"\n rawimg = tf.io.readfile(imgpath)\n img = tf.io.decodejpeg(rawimg)\n return img, _
\n\ndef preprocess_img(img, _):\n img = tf.cast(img, tf.float32)\n img = img / 255.\n img = tf.image.resize(img, IMAGE_SHAPE[:2])\n return img, _
\n\nI am not very experienced with the tensorflow pipeline. Where is the bottleneck that causes the high cpu usage?
", "rawMarkdown": "Hi Rich, thank you for your response. To be honest I didnt even noticed the GPU memory usage (I was confused by the empty usage bar).\nThis is how I load the data:\n\n\n\nwith…\ndef readimg(imagename, ):\n imgpath = TRAINIMGDIR + imagename + \".jpg\"\n rawimg = tf.io.readfile(imgpath)\n img = tf.io.decodejpeg(rawimg)\n return img, _\n\ndef preprocess_img(img, _):\n img = tf.cast(img, tf.float32)\n img = img / 255.\n img = tf.image.resize(img, IMAGE_SHAPE[:2])\n return img, _\n\nI am not very experienced with the tensorflow pipeline. Where is the bottleneck that causes the high cpu usage?" }, { "id": 934937, "postDate": "2020-07-18T23:43:42.013Z", "content": "You're not doing a lot of preprocessing, so maybe your problem is elsewhere. But, \"resize\" can take some time. I would preprocess all the images with the size you need them and see if that helps.
", "rawMarkdown": "You're not doing a lot of preprocessing, so maybe your problem is elsewhere. But, \"resize\" can take some time. I would preprocess all the images with the size you need them and see if that helps." } ] }, { "id": 934490, "postDate": "2020-07-18T13:37:05.810Z", "content": "Hi everyone,
\n\nI enabled GPU acceleration and computation on GPU device:
\n\n
However, training is super slow and the GPU is not utlized:
\n\n
Is there some setting i missed?
", "rawMarkdown": "Hi everyone,\n\nI enabled GPU acceleration and computation on GPU device:\n\n\n\nHowever, training is super slow and the GPU is not utlized:\n\n\n\nIs there some setting i missed?\n" }, { "id": 934918, "postDate": "2020-07-18T22:41:55.067Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 934701, "author_name": "quadcore/Richard Epstein", "author_url": "", "post_date": "2020-07-18T16:53:51.280000", "content": "From the screenshot, you are using most of your GPU memory, so your software is utilizing the GPU. But your CPU is at 100+%. That usually means you are trying to do too much in your data loading stage and the CPU cannot feed the GPU fast enough. Have you pre-processed your data? Best to do as much processing of the images outside of your data loading. Resize, normalize, crop, etc. and store the images on disk. Better yet, use TFRecords, which are even faster to read in.
\n\n-Rich
", "votes": 0, "replies": [ { "id": 934919, "author_name": "Florian", "author_url": "", "post_date": "2020-07-18T22:43:21.700000", "content": "Hi Rich, thank you for your response. To be honest I didnt even noticed the GPU memory usage (I was confused by the empty usage bar).\nThis is how I load the data:\n
with…\ndef readimg(imagename, ):\n imgpath = TRAINIMGDIR + imagename + \".jpg\"\n rawimg = tf.io.readfile(imgpath)\n img = tf.io.decodejpeg(rawimg)\n return img, _
\n\ndef preprocess_img(img, _):\n img = tf.cast(img, tf.float32)\n img = img / 255.\n img = tf.image.resize(img, IMAGE_SHAPE[:2])\n return img, _
\n\nI am not very experienced with the tensorflow pipeline. Where is the bottleneck that causes the high cpu usage?
", "votes": 0, "replies": [] }, { "id": 934937, "author_name": "quadcore/Richard Epstein", "author_url": "", "post_date": "2020-07-18T23:43:42.013000", "content": "You're not doing a lot of preprocessing, so maybe your problem is elsewhere. But, \"resize\" can take some time. I would preprocess all the images with the size you need them and see if that helps.
", "votes": 0, "replies": [] } ] }, { "id": 934918, "author_name": "", "author_url": "", "post_date": "2020-07-18T22:41:55.067000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "934701": "From the screenshot, you are using most of your GPU memory, so your software is utilizing the GPU. But your CPU is at 100+%. That usually means you are trying to do too much in your data loading stage and the CPU cannot feed the GPU fast enough. Have you pre-processed your data? Best to do as much processing of the images outside of your data loading. Resize, normalize, crop, etc. and store the images on disk. Better yet, use TFRecords, which are even faster to read in.\n\n-Rich", "934490": "Hi everyone,\n\nI enabled GPU acceleration and computation on GPU device:\n\n\n\nHowever, training is super slow and the GPU is not utlized:\n\n\n\nIs there some setting i missed?\n", "934918": "" } }