I am using pre-trained resnet18 according to Human's kernels(https://www.kaggle.com/humananalog/keras-generator-for-reading-directly-from-bson) .
\n\nOne iteration (batch_size = 256), it took about 3.3 seconds to train, but the training only took 0.3 seconds and the reading data took 3 seconds .
\n\nMy gpu is 1080Ti. Anyone has advice to feed data into net faster?
", "messages": [ { "id": "226108", "postDate": "10/01/2017 03:43:23", "content": "I am using pre-trained resnet18 according to Human's kernels(https://www.kaggle.com/humananalog/keras-generator-for-reading-directly-from-bson) .
\n\nOne iteration (batch_size = 256), it took about 3.3 seconds to train, but the training only took 0.3 seconds and the reading data took 3 seconds .
\n\nMy gpu is 1080Ti. Anyone has advice to feed data into net faster?
", "rawMarkdown": "I am using pre-trained resnet18 according to Human's kernels(https://www.kaggle.com/humananalog/keras-generator-for-reading-directly-from-bson) . \n\nOne iteration (batch_size = 256), it took about 3.3 seconds to train, but the training only took 0.3 seconds and the reading data took 3 seconds .\n\nMy gpu is 1080Ti. Anyone has advice to feed data into net faster?", "votes": null }, { "id": "226143", "postDate": "10/01/2017 06:46:55", "content": "It seemed that multiple workers could solve it.
", "rawMarkdown": "It seemed that multiple workers could solve it.", "votes": null }, { "id": "245546", "postDate": "11/18/2017 19:53:19", "content": "I have the same problem, I am now using NASNet, my GPU is K5200, feeding data needs ten days.\nThe speed depends on these three factors: (1) the structure of the net, (2) the performance of GPU, (3) the input data.\n(1) and (3) are very hard to change. So I advise you to buy more GPUs.
", "rawMarkdown": "I have the same problem, I am now using NASNet, my GPU is K5200, feeding data needs ten days.\nThe speed depends on these three factors: (1) the structure of the net, (2) the performance of GPU, (3) the input data.\n(1) and (3) are very hard to change. So I advise you to buy more GPUs.", "votes": null }, { "id": "245699", "postDate": "11/19/2017 10:29:00", "content": "The generator from my kernel is not very fast, since it randomly skips through the BSON file, which is huge. The best way to speed it up is to put the BSON file on an SSD instead of a disk.
\n\nIf that still is not fast enough, consider using a different generator that reads the BSON file sequentially instead of in a random order. That will be way faster than doing random seeks.
", "rawMarkdown": "The generator from my kernel is not very fast, since it randomly skips through the BSON file, which is huge. The best way to speed it up is to put the BSON file on an SSD instead of a disk.\n\nIf that still is not fast enough, consider using a different generator that reads the BSON file sequentially instead of in a random order. That will be way faster than doing random seeks.", "votes": null } ], "comments": [ { "id": 226143, "author_name": "zhangsongwei", "author_url": "", "post_date": "10/01/2017 06:46:55", "content": "It seemed that multiple workers could solve it.
", "votes": null, "replies": [] }, { "id": 245546, "author_name": "cjj3779", "author_url": "", "post_date": "11/18/2017 19:53:19", "content": "I have the same problem, I am now using NASNet, my GPU is K5200, feeding data needs ten days.\nThe speed depends on these three factors: (1) the structure of the net, (2) the performance of GPU, (3) the input data.\n(1) and (3) are very hard to change. So I advise you to buy more GPUs.
", "votes": null, "replies": [] }, { "id": 245699, "author_name": "humananalog", "author_url": "", "post_date": "11/19/2017 10:29:00", "content": "The generator from my kernel is not very fast, since it randomly skips through the BSON file, which is huge. The best way to speed it up is to put the BSON file on an SSD instead of a disk.
\n\nIf that still is not fast enough, consider using a different generator that reads the BSON file sequentially instead of in a random order. That will be way faster than doing random seeks.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "226108": "I am using pre-trained resnet18 according to Human's kernels(https://www.kaggle.com/humananalog/keras-generator-for-reading-directly-from-bson) . \n\nOne iteration (batch_size = 256), it took about 3.3 seconds to train, but the training only took 0.3 seconds and the reading data took 3 seconds .\n\nMy gpu is 1080Ti. Anyone has advice to feed data into net faster?", "226143": "It seemed that multiple workers could solve it.", "245546": "I have the same problem, I am now using NASNet, my GPU is K5200, feeding data needs ten days.\nThe speed depends on these three factors: (1) the structure of the net, (2) the performance of GPU, (3) the input data.\n(1) and (3) are very hard to change. So I advise you to buy more GPUs.", "245699": "The generator from my kernel is not very fast, since it randomly skips through the BSON file, which is huge. The best way to speed it up is to put the BSON file on an SSD instead of a disk.\n\nIf that still is not fast enough, consider using a different generator that reads the BSON file sequentially instead of in a random order. That will be way faster than doing random seeks." }, "source": "meta" }