Hi everyone!\nI am very glad to participate in this competition and thanks a lot those who contribute with different ideas and kernels, I found it very helpful and now my 256x256 uNet (from here: https://www.kaggle.com/c/carvana-image-masking-challenge/discussion/37523 + some additional things) gives me 0.9959 on LB. I am going to improve this, but training time is a bit painful for me.
\n\nIn discussions I have found different references about uNet training time and that's the source of my worries. I see that for people it usually takes about 4-5 mins to train 1 epoch of uNet512.
\n\nMy env is:
\n\nTraining time for me looks something like that:\n8 min for input size 128x128, 25 min for 256, 1h25m for 512, etc...
\n\nI also tried to switch to gpuarray backend, but it goes even slower.
\n\nDoes it look similar for someone else? If not, what is your training time? Any suggestions to improve mine?
", "messages": [ { "id": "220555", "postDate": "09/12/2017 15:05:03", "content": "Hi everyone!\nI am very glad to participate in this competition and thanks a lot those who contribute with different ideas and kernels, I found it very helpful and now my 256x256 uNet (from here: https://www.kaggle.com/c/carvana-image-masking-challenge/discussion/37523 + some additional things) gives me 0.9959 on LB. I am going to improve this, but training time is a bit painful for me.
\n\nIn discussions I have found different references about uNet training time and that's the source of my worries. I see that for people it usually takes about 4-5 mins to train 1 epoch of uNet512.
\n\nMy env is:
\n\nTraining time for me looks something like that:\n8 min for input size 128x128, 25 min for 256, 1h25m for 512, etc...
\n\nI also tried to switch to gpuarray backend, but it goes even slower.
\n\nDoes it look similar for someone else? If not, what is your training time? Any suggestions to improve mine?
", "rawMarkdown": "Hi everyone!\nI am very glad to participate in this competition and thanks a lot those who contribute with different ideas and kernels, I found it very helpful and now my 256x256 uNet (from here: https://www.kaggle.com/c/carvana-image-masking-challenge/discussion/37523 + some additional things) gives me 0.9959 on LB. I am going to improve this, but training time is a bit painful for me.\n\nIn discussions I have found different references about uNet training time and that's the source of my worries. I see that for people it usually takes about 4-5 mins to train 1 epoch of uNet512.\n\nMy env is: \n\n - Ubuntu 16.04\n - Python 3.4.4\n - GTX 1080\n - Keras 2.0.3 (Theano backend)\n - CUDA 8.0.61\n - cuDNN 5110\n\nTraining time for me looks something like that:\n**8 min** for input size 128x128, **25 min** for 256, **1h25m** for 512, etc...\n\nI also tried to switch to gpuarray backend, but it goes even slower.\n\nDoes it look similar for someone else? If not, what is your training time? Any suggestions to improve mine?", "votes": null }, { "id": "220556", "postDate": "09/12/2017 15:15:28", "content": "Kaggle, why don't you give me a chance to fix the title of the thread? I really need this 'r' letter
", "rawMarkdown": "Kaggle, why don't you give me a chance to fix the title of the thread? I really need this 'r' letter", "votes": null }, { "id": "220559", "postDate": "09/12/2017 15:21:51", "content": "Hi true_pk, I have pretty much the same exact setup. I can confirm that you can achieve .9967 on the lb. I am hitting the high .9969's on a single split CV and think I could break .997. You will likely need to increase your resolution to 1024 and use hq images. My training time is about 18.5 min per epoch and 16 hrs for a session. Also, you might have dead neurons the longer you train so maybe use PReLU, LeakyReLU, etc.
", "rawMarkdown": "Hi true_pk, I have pretty much the same exact setup. I can confirm that you can achieve .9967 on the lb. I am hitting the high .9969's on a single split CV and think I could break .997. You will likely need to increase your resolution to 1024 and use hq images. My training time is about 18.5 min per epoch and 16 hrs for a session. Also, you might have dead neurons the longer you train so maybe use PReLU, LeakyReLU, etc.", "votes": null }, { "id": "220579", "postDate": "09/12/2017 15:53:38", "content": "Now I started using 512 input size and it takes me 1hour 25 minutes per epoch. Could activation function influence so much? I haven't tried PReLU or LeakyReLU yet
", "rawMarkdown": "Now I started using 512 input size and it takes me **1hour 25 minutes** per epoch. Could activation function influence so much? I haven't tried PReLU or LeakyReLU yet", "votes": null }, { "id": "220599", "postDate": "09/12/2017 16:49:58", "content": "\n\n\nsome additional things
\n
Try without them?
\n\n\n\n\nKeras 2.0.3 (Theano backend)
\n
Try with Tensorflow?
\n\nI have 1080 ti, should be only 25% faster than 1080, but my training times are about 3-4 times faster.
", "rawMarkdown": "> some additional things \n\nTry without them?\n\n> Keras 2.0.3 (Theano backend) \n\nTry with Tensorflow?\n\nI have 1080 ti, should be only 25% faster than 1080, but my training times are about 3-4 times faster.", "votes": null }, { "id": "220616", "postDate": "09/12/2017 17:23:30", "content": "Thank you so much Roman! I had no idea that Tensorlow backend would work 10 times faster!!! oh man... I always use Theano and even read that it usually works a bit faster than Tensorflow
", "rawMarkdown": "Thank you so much Roman! I had no idea that Tensorlow backend would work 10 times faster!!! oh man... I always use Theano and even read that it usually works a bit faster than Tensorflow", "votes": null }, { "id": "220717", "postDate": "09/12/2017 22:09:53", "content": "\n\n\nI have 1080 ti, should be only 25% faster than 1080, but my training times are about 3-4 times faster.
\n
They could be bottlenecked by lack of VRAM when they're training on very large images.
", "rawMarkdown": "> I have 1080 ti, should be only 25% faster than 1080, but my training times are about 3-4 times faster.\n\nThey could be bottlenecked by lack of VRAM when they're training on very large images.", "votes": null }, { "id": "220745", "postDate": "09/13/2017 00:19:13", "content": "true_pk, there should not be that much difference between Theano and Tensorflow. Did your Theano backend actually use CUDA and cudnn?
", "rawMarkdown": "true_pk, there should not be that much difference between Theano and Tensorflow. Did your Theano backend actually use CUDA and cudnn?", "votes": null }, { "id": "220746", "postDate": "09/13/2017 00:25:02", "content": "\n\n\nthere should not be that much difference between Theano and Tensorflow
\n
Jurand, I absolutely agree.
\n\n\n\n\nDid your Theano backend actually use CUDA and cud?
\n
I guess so. At least Keras says my cuDNN v.5110 is used.
", "rawMarkdown": "> there should not be that much difference between Theano and Tensorflow\n\nJurand, I absolutely agree.\n> Did your Theano backend actually use CUDA and cud?\n\nI guess so. At least Keras says my cuDNN v.5110 is used.", "votes": null }, { "id": "220758", "postDate": "09/13/2017 01:07:48", "content": "It didn't improve my score as much as increasing resolution, using hq images, or changing the back-propagation optimizer. It has given me a ~0.0002 boost, which is significant given that we are working on our 3rd 9
", "rawMarkdown": "It didn't improve my score as much as increasing resolution, using hq images, or changing the back-propagation optimizer. It has given me a ~0.0002 boost, which is significant given that we are working on our 3rd 9", "votes": null } ], "comments": [ { "id": 220556, "author_name": "truepk", "author_url": "", "post_date": "09/12/2017 15:15:28", "content": "Kaggle, why don't you give me a chance to fix the title of the thread? I really need this 'r' letter
", "votes": null, "replies": [] }, { "id": 220559, "author_name": "cpruce", "author_url": "", "post_date": "09/12/2017 15:21:51", "content": "Hi true_pk, I have pretty much the same exact setup. I can confirm that you can achieve .9967 on the lb. I am hitting the high .9969's on a single split CV and think I could break .997. You will likely need to increase your resolution to 1024 and use hq images. My training time is about 18.5 min per epoch and 16 hrs for a session. Also, you might have dead neurons the longer you train so maybe use PReLU, LeakyReLU, etc.
", "votes": null, "replies": [ { "id": 220579, "author_name": "truepk", "author_url": "", "post_date": "09/12/2017 15:53:38", "content": "Now I started using 512 input size and it takes me 1hour 25 minutes per epoch. Could activation function influence so much? I haven't tried PReLU or LeakyReLU yet
", "votes": null, "replies": [] }, { "id": 220758, "author_name": "cpruce", "author_url": "", "post_date": "09/13/2017 01:07:48", "content": "It didn't improve my score as much as increasing resolution, using hq images, or changing the back-propagation optimizer. It has given me a ~0.0002 boost, which is significant given that we are working on our 3rd 9
", "votes": null, "replies": [] } ] }, { "id": 220599, "author_name": "inoryy", "author_url": "", "post_date": "09/12/2017 16:49:58", "content": "\n\n\nsome additional things
\n
Try without them?
\n\n\n\n\nKeras 2.0.3 (Theano backend)
\n
Try with Tensorflow?
\n\nI have 1080 ti, should be only 25% faster than 1080, but my training times are about 3-4 times faster.
", "votes": null, "replies": [ { "id": 220616, "author_name": "truepk", "author_url": "", "post_date": "09/12/2017 17:23:30", "content": "Thank you so much Roman! I had no idea that Tensorlow backend would work 10 times faster!!! oh man... I always use Theano and even read that it usually works a bit faster than Tensorflow
", "votes": null, "replies": [] }, { "id": 220717, "author_name": "stevenknguyen", "author_url": "", "post_date": "09/12/2017 22:09:53", "content": "\n\n\nI have 1080 ti, should be only 25% faster than 1080, but my training times are about 3-4 times faster.
\n
They could be bottlenecked by lack of VRAM when they're training on very large images.
", "votes": null, "replies": [] }, { "id": 220745, "author_name": "jurand", "author_url": "", "post_date": "09/13/2017 00:19:13", "content": "true_pk, there should not be that much difference between Theano and Tensorflow. Did your Theano backend actually use CUDA and cudnn?
", "votes": null, "replies": [] }, { "id": 220746, "author_name": "truepk", "author_url": "", "post_date": "09/13/2017 00:25:02", "content": "\n\n\nthere should not be that much difference between Theano and Tensorflow
\n
Jurand, I absolutely agree.
\n\n\n\n\nDid your Theano backend actually use CUDA and cud?
\n
I guess so. At least Keras says my cuDNN v.5110 is used.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "220555": "Hi everyone!\nI am very glad to participate in this competition and thanks a lot those who contribute with different ideas and kernels, I found it very helpful and now my 256x256 uNet (from here: https://www.kaggle.com/c/carvana-image-masking-challenge/discussion/37523 + some additional things) gives me 0.9959 on LB. I am going to improve this, but training time is a bit painful for me.\n\nIn discussions I have found different references about uNet training time and that's the source of my worries. I see that for people it usually takes about 4-5 mins to train 1 epoch of uNet512.\n\nMy env is: \n\n - Ubuntu 16.04\n - Python 3.4.4\n - GTX 1080\n - Keras 2.0.3 (Theano backend)\n - CUDA 8.0.61\n - cuDNN 5110\n\nTraining time for me looks something like that:\n**8 min** for input size 128x128, **25 min** for 256, **1h25m** for 512, etc...\n\nI also tried to switch to gpuarray backend, but it goes even slower.\n\nDoes it look similar for someone else? If not, what is your training time? Any suggestions to improve mine?", "220556": "Kaggle, why don't you give me a chance to fix the title of the thread? I really need this 'r' letter", "220559": "Hi true_pk, I have pretty much the same exact setup. I can confirm that you can achieve .9967 on the lb. I am hitting the high .9969's on a single split CV and think I could break .997. You will likely need to increase your resolution to 1024 and use hq images. My training time is about 18.5 min per epoch and 16 hrs for a session. Also, you might have dead neurons the longer you train so maybe use PReLU, LeakyReLU, etc.", "220579": "Now I started using 512 input size and it takes me **1hour 25 minutes** per epoch. Could activation function influence so much? I haven't tried PReLU or LeakyReLU yet", "220599": "> some additional things \n\nTry without them?\n\n> Keras 2.0.3 (Theano backend) \n\nTry with Tensorflow?\n\nI have 1080 ti, should be only 25% faster than 1080, but my training times are about 3-4 times faster.", "220616": "Thank you so much Roman! I had no idea that Tensorlow backend would work 10 times faster!!! oh man... I always use Theano and even read that it usually works a bit faster than Tensorflow", "220717": "> I have 1080 ti, should be only 25% faster than 1080, but my training times are about 3-4 times faster.\n\nThey could be bottlenecked by lack of VRAM when they're training on very large images.", "220745": "true_pk, there should not be that much difference between Theano and Tensorflow. Did your Theano backend actually use CUDA and cudnn?", "220746": "> there should not be that much difference between Theano and Tensorflow\n\nJurand, I absolutely agree.\n> Did your Theano backend actually use CUDA and cud?\n\nI guess so. At least Keras says my cuDNN v.5110 is used.", "220758": "It didn't improve my score as much as increasing resolution, using hq images, or changing the back-propagation optimizer. It has given me a ~0.0002 boost, which is significant given that we are working on our 3rd 9" }, "source": "meta" }