I implemented a MXNet backend keras code yesterday. However, training on two K80 only result in nan loss. I have tested various batch size [8,16,256,1024] and learning rate [ 0.01 ( default ), 0.001, 0.0001 ] all ended in the same results.
\n\nMy current guess was the problem maybe cause my bson generator implementation which was limited to a single workers. My bson generator code.
\n\nI would hope someone could tried my code using a more proper generator ( support multi workers ), and tell me if there's any speedup.
\n\nNote: \n 1. I tried Human Analog's bson generator, however, random access in HDD is too slow for practical training.\n 2. Currently MXNet still have compatibility issues with keras pretrained models, hence I use the resnet implementation from mxnet keras repository. \n 3. Training on cifar10 datasets shows proper speedup with decreasing loss.
\n\nGithub code: https://github.com/DBlackKat/mxnet_example
", "messages": [ { "id": "230880", "postDate": "10/13/2017 02:16:10", "content": "I implemented a MXNet backend keras code yesterday. However, training on two K80 only result in nan loss. I have tested various batch size [8,16,256,1024] and learning rate [ 0.01 ( default ), 0.001, 0.0001 ] all ended in the same results.
\n\nMy current guess was the problem maybe cause my bson generator implementation which was limited to a single workers. My bson generator code.
\n\nI would hope someone could tried my code using a more proper generator ( support multi workers ), and tell me if there's any speedup.
\n\nNote: \n 1. I tried Human Analog's bson generator, however, random access in HDD is too slow for practical training.\n 2. Currently MXNet still have compatibility issues with keras pretrained models, hence I use the resnet implementation from mxnet keras repository. \n 3. Training on cifar10 datasets shows proper speedup with decreasing loss.
\n\nGithub code: https://github.com/DBlackKat/mxnet_example
", "rawMarkdown": "I implemented a MXNet backend keras code yesterday. However, training on two K80 only result in nan loss. I have tested various batch size [8,16,256,1024] and learning rate [ 0.01 ( default ), 0.001, 0.0001 ] all ended in the same results. \n\nMy current guess was the problem maybe cause my bson generator implementation which was limited to a single workers. [My bson generator code.][1] \n\nI would hope someone could tried my code using a more proper generator ( support multi workers ), and tell me if there's any speedup. \n\nNote: \n 1. I tried Human Analog's [bson generator][2], however, random access in HDD is too slow for practical training.\n 2. Currently MXNet still have compatibility issues with keras pretrained models, hence I use the resnet implementation from [mxnet keras repository][3]. \n 3. Training on cifar10 datasets shows proper speedup with decreasing loss. \n\nGithub code: [https://github.com/DBlackKat/mxnet_example][4]\n\n\n [1]: https://www.kaggle.com/theblackcat/loading-bson-data-for-keras-fit-generator\n [2]: https://www.kaggle.com/humananalog/keras-generator-for-reading-directly-from-bson\n [3]: https://github.com/dmlc/keras/blob/master/examples/cifar10_resnet50.py\n [4]: https://github.com/DBlackKat/mxnet_example", "votes": null }, { "id": "231015", "postDate": "10/13/2017 13:24:13", "content": "Hi, \nI'm starting to implement a bson2rec\nhttps://gist.github.com/jpizarrom/ba35118f181dc00693286c469d6b856c
", "rawMarkdown": "Hi, \nI'm starting to implement a bson2rec\nhttps://gist.github.com/jpizarrom/ba35118f181dc00693286c469d6b856c", "votes": null }, { "id": "231265", "postDate": "10/14/2017 06:49:26", "content": "Hello Zhi,
\n\nI had the same data reading issues as you. Check out my kernels here for solution\nhttps://www.kaggle.com/lamdang/fast-shuffle-bson-generator-for-keras
", "rawMarkdown": "Hello Zhi,\n\nI had the same data reading issues as you. Check out my kernels here for solution\nhttps://www.kaggle.com/lamdang/fast-shuffle-bson-generator-for-keras", "votes": null }, { "id": "231285", "postDate": "10/14/2017 08:59:34", "content": "Thanks @Lam Dang, I will certainly test your generator later. Currently I was trying to dump the bson file into a mongdb and iterate from there,( which was shown in this post). I will test your generator and posted the result here as soon as possible.
", "rawMarkdown": "Thanks @Lam Dang, I will certainly test your generator later. Currently I was trying to dump the bson file into a mongdb and iterate from there,( [which was shown in this post][1]). I will test your generator and posted the result here as soon as possible. \n\n\n [1]: https://www.kaggle.com/c/cdiscount-image-classification-challenge/discussion/41193", "votes": null }, { "id": "231320", "postDate": "10/14/2017 12:34:21", "content": "Hi, I,ve see the same aproach in https://github.com/xkumiyu/cdiscount-kernel/blob/master/dataset.py
", "rawMarkdown": "Hi, I,ve see the same aproach in https://github.com/xkumiyu/cdiscount-kernel/blob/master/dataset.py", "votes": null }, { "id": "231324", "postDate": "10/14/2017 12:49:54", "content": "Hi Juan,\nIs there any information regarding about your iterator performance?
", "rawMarkdown": "Hi Juan,\nIs there any information regarding about your iterator performance?", "votes": null }, { "id": "231333", "postDate": "10/14/2017 13:24:17", "content": "This is early testing, I'm learning mxnet and deep learning.
\n\naccuracy is 0.000000 maybe the are something wrong with bson2rec
\n\nas bson2rec create a rec file, I'm testing the performance using examples of mxnet.\nI'm testing in azure STANDARD_NC24, has 4 x K80 GPU
\n\nexport MXNET_ENABLE_GPU_P2P=0\n\npython mxnet/example/image-classification/fine-tune.py --pretrained-model imagenet1k-resnet-50 \\\n--data-train /mnt/batch/tasks/shared/data/train-all.rec \\\n--batch-size 32 --num-classes 5270 --num-examples 7069696 \\\n--image-shape 3,180,180 --gpus 0,1,2,3\nINFO:root:Epoch[0] Batch [40] Speed: 65.71 samples/sec accuracy=0.000000\nINFO:root:Epoch[0] Batch [60] Speed: 61.08 samples/sec accuracy=0.000000\nINFO:root:Epoch[0] Batch [60] Speed: 60.68 samples/sec accuracy=0.000000\n126.14 samples/sec\nINFO:root:Epoch[0] Batch [20] Speed: 65.69 samples/sec accuracy=0.000000\n171.28 samples/sec\n201.91 samples/sec\nHi, \nI'm starting to implement a bson2rec\nhttps://gist.github.com/jpizarrom/ba35118f181dc00693286c469d6b856c
", "votes": null, "replies": [ { "id": 231324, "author_name": "theblackcat", "author_url": "", "post_date": "10/14/2017 12:49:54", "content": "Hi Juan,\nIs there any information regarding about your iterator performance?
", "votes": null, "replies": [] }, { "id": 231333, "author_name": "jpizarrom", "author_url": "", "post_date": "10/14/2017 13:24:17", "content": "This is early testing, I'm learning mxnet and deep learning.
\n\naccuracy is 0.000000 maybe the are something wrong with bson2rec
\n\nas bson2rec create a rec file, I'm testing the performance using examples of mxnet.\nI'm testing in azure STANDARD_NC24, has 4 x K80 GPU
\n\nexport MXNET_ENABLE_GPU_P2P=0\n\npython mxnet/example/image-classification/fine-tune.py --pretrained-model imagenet1k-resnet-50 \\\n--data-train /mnt/batch/tasks/shared/data/train-all.rec \\\n--batch-size 32 --num-classes 5270 --num-examples 7069696 \\\n--image-shape 3,180,180 --gpus 0,1,2,3\nINFO:root:Epoch[0] Batch [40] Speed: 65.71 samples/sec accuracy=0.000000\nINFO:root:Epoch[0] Batch [60] Speed: 61.08 samples/sec accuracy=0.000000\nINFO:root:Epoch[0] Batch [60] Speed: 60.68 samples/sec accuracy=0.000000\n126.14 samples/sec\nINFO:root:Epoch[0] Batch [20] Speed: 65.69 samples/sec accuracy=0.000000\n171.28 samples/sec\n201.91 samples/sec\nHello Zhi,
\n\nI had the same data reading issues as you. Check out my kernels here for solution\nhttps://www.kaggle.com/lamdang/fast-shuffle-bson-generator-for-keras
", "votes": null, "replies": [ { "id": 231285, "author_name": "theblackcat", "author_url": "", "post_date": "10/14/2017 08:59:34", "content": "Thanks @Lam Dang, I will certainly test your generator later. Currently I was trying to dump the bson file into a mongdb and iterate from there,( which was shown in this post). I will test your generator and posted the result here as soon as possible.
", "votes": null, "replies": [] }, { "id": 231320, "author_name": "jpizarrom", "author_url": "", "post_date": "10/14/2017 12:34:21", "content": "Hi, I,ve see the same aproach in https://github.com/xkumiyu/cdiscount-kernel/blob/master/dataset.py
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "230880": "I implemented a MXNet backend keras code yesterday. However, training on two K80 only result in nan loss. I have tested various batch size [8,16,256,1024] and learning rate [ 0.01 ( default ), 0.001, 0.0001 ] all ended in the same results. \n\nMy current guess was the problem maybe cause my bson generator implementation which was limited to a single workers. [My bson generator code.][1] \n\nI would hope someone could tried my code using a more proper generator ( support multi workers ), and tell me if there's any speedup. \n\nNote: \n 1. I tried Human Analog's [bson generator][2], however, random access in HDD is too slow for practical training.\n 2. Currently MXNet still have compatibility issues with keras pretrained models, hence I use the resnet implementation from [mxnet keras repository][3]. \n 3. Training on cifar10 datasets shows proper speedup with decreasing loss. \n\nGithub code: [https://github.com/DBlackKat/mxnet_example][4]\n\n\n [1]: https://www.kaggle.com/theblackcat/loading-bson-data-for-keras-fit-generator\n [2]: https://www.kaggle.com/humananalog/keras-generator-for-reading-directly-from-bson\n [3]: https://github.com/dmlc/keras/blob/master/examples/cifar10_resnet50.py\n [4]: https://github.com/DBlackKat/mxnet_example", "231015": "Hi, \nI'm starting to implement a bson2rec\nhttps://gist.github.com/jpizarrom/ba35118f181dc00693286c469d6b856c", "231265": "Hello Zhi,\n\nI had the same data reading issues as you. Check out my kernels here for solution\nhttps://www.kaggle.com/lamdang/fast-shuffle-bson-generator-for-keras", "231285": "Thanks @Lam Dang, I will certainly test your generator later. Currently I was trying to dump the bson file into a mongdb and iterate from there,( [which was shown in this post][1]). I will test your generator and posted the result here as soon as possible. \n\n\n [1]: https://www.kaggle.com/c/cdiscount-image-classification-challenge/discussion/41193", "231320": "Hi, I,ve see the same aproach in https://github.com/xkumiyu/cdiscount-kernel/blob/master/dataset.py", "231324": "Hi Juan,\nIs there any information regarding about your iterator performance?", "231333": "This is early testing, I'm learning mxnet and deep learning.\n\naccuracy is 0.000000 maybe the are something wrong with bson2rec\n\nas bson2rec create a rec file, I'm testing the performance using examples of mxnet.\nI'm testing in azure STANDARD_NC24, has 4 x K80 GPU\n\n export MXNET_ENABLE_GPU_P2P=0\n\n python mxnet/example/image-classification/fine-tune.py --pretrained-model imagenet1k-resnet-50 \\\n --data-train /mnt/batch/tasks/shared/data/train-all.rec \\\n --batch-size 32 --num-classes 5270 --num-examples 7069696 \\\n --image-shape 3,180,180 --gpus 0,1,2,3\n\n# 3,180,180 batch 128 gpus 0 data-nthreads 4\n INFO:root:Epoch[0] Batch [40] Speed: 65.71 samples/sec accuracy=0.000000\n\n# 3,180,180 batch 32 gpus 0 data-nthreads 4\n INFO:root:Epoch[0] Batch [60] Speed: 61.08 samples/sec accuracy=0.000000\n\n# 3,180,180 batch 32 gpus 0 data-nthreads 8\n INFO:root:Epoch[0] Batch [60] Speed: 60.68 samples/sec accuracy=0.000000\n\n# 3,180,180 batch 128 gpus 0,1 data-nthreads 4\n 126.14 samples/sec\n\n# 3,180,180 batch 128 gpus 0 data-nthreads 8\n INFO:root:Epoch[0] Batch [20] Speed: 65.69 samples/sec accuracy=0.000000\n\n# 3,180,180 batch 128 gpus 0,1,2 data-nthreads 4\n 171.28 samples/sec\n\n# 3,180,180 batch 128 gpus 0,1,2,3 data-nthreads 4\n 201.91 samples/sec" }, "source": "meta" }