Hi everyone, any advise for the resource challenged, I meant CPU users not GPU for this competition?
\n\nUpdate on 08/30th/2017:
\n\nI signed up with Floyd paid account and used their GPU. I expected the GPU to be faster than it is when I ran my training job. I also tested prediction using their CPU which kept giving a memory error when the same thing runs on my CPU labtop without any memory issue.
\n\nMy verdict - Floyd is easy to setup, lacks AWS/ GCP features but much cheaper.
", "messages": [ { "id": "215357", "postDate": "08/21/2017 09:38:23", "content": "Hi everyone, any advise for the resource challenged, I meant CPU users not GPU for this competition?
\n\nUpdate on 08/30th/2017:
\n\nI signed up with Floyd paid account and used their GPU. I expected the GPU to be faster than it is when I ran my training job. I also tested prediction using their CPU which kept giving a memory error when the same thing runs on my CPU labtop without any memory issue.
\n\nMy verdict - Floyd is easy to setup, lacks AWS/ GCP features but much cheaper.
", "rawMarkdown": "Hi everyone, any advise for the resource challenged, I meant CPU users not GPU for this competition?\n\nUpdate on 08/30th/2017:\n\nI signed up with [Floyd](https://www.floydhub.com/projects) paid account and used their GPU. I expected the GPU to be faster than it is when I ran my training job. I also tested prediction using their CPU which kept giving a memory error when the same thing runs on my CPU labtop without any memory issue.\n\nMy verdict - [Floyd](https://www.floydhub.com/projects) is easy to setup, lacks AWS/ GCP features but much cheaper.", "votes": null }, { "id": "215374", "postDate": "08/21/2017 11:32:15", "content": "Unfortunately, I guess the only advice is to rent/buy GPU.
\n\nOk, maybe there are some other approaches that can benefit CPU users, but 99% of guys here use CNNs, that are much faster on GPU than CPU.
\n\nSo if you on a budget you can buy some old card - it will still be faster than i7. For example I had 660ti 2gb and it was capable to train 256 unet with batch size=2 - and you can score around 0.994 LB (on i7 it couldn't even start training:-)). If you will decide to buy new GPU - check out this great post http://timdettmers.com/2017/04/09/which-gpu-for-deep-learning/. Or try Google Cloud GPU - they have 300$ coupon.
", "rawMarkdown": "Unfortunately, I guess the only advice is to rent/buy GPU. \n\nOk, maybe there are some other approaches that can benefit CPU users, but 99% of guys here use CNNs, that are much faster on GPU than CPU. \n\nSo if you on a budget you can buy some old card - it will still be faster than i7. For example I had 660ti 2gb and it was capable to train 256 unet with batch size=2 - and you can score around 0.994 LB (on i7 it couldn't even start training:-)). If you will decide to buy new GPU - check out this great post http://timdettmers.com/2017/04/09/which-gpu-for-deep-learning/. Or try Google Cloud GPU - they have 300$ coupon.", "votes": null }, { "id": "215384", "postDate": "08/21/2017 12:19:19", "content": "Preemptible instances with 8-16 CPUs on Google Cloud are very cheap (> 0.1$ per hour) and can be reliably used given you implement a checkpoint mechanism on training and testing. It's not as fast but its time/cost can't be beat (http://minimaxir.com/2017/07/cpu-or-gpu/ - to be taken with a grain of salt but still interesting).
\n\nIf you really want/need a GPU, a new GC account come with 300$ like already mentioned.
", "rawMarkdown": "Preemptible instances with 8-16 CPUs on Google Cloud are very cheap (> 0.1$ per hour) and can be reliably used given you implement a checkpoint mechanism on training and testing. It's not as fast but its time/cost can't be beat (http://minimaxir.com/2017/07/cpu-or-gpu/ - to be taken with a grain of salt but still interesting).\n\nIf you really want/need a GPU, a new GC account come with 300$ like already mentioned.", "votes": null }, { "id": "215549", "postDate": "08/22/2017 05:26:34", "content": "Thanks @Feels_g00d_man and @Gaarv for your very useful suggestions.
\n\nI am thinking of trying Floyd. Has anybody here used it? Any advise on the best way to set it up for this contest? I briefly checked their website but will register in a few days when I get a chance.
", "rawMarkdown": "Thanks @Feels_g00d_man and @Gaarv for your very useful suggestions.\n\nI am thinking of trying [Floyd](https://www.floydhub.com/). Has anybody here used it? Any advise on the best way to set it up for this contest? I briefly checked their website but will register in a few days when I get a chance.", "votes": null }, { "id": "215558", "postDate": "08/22/2017 05:57:09", "content": "Floyd installs all of the libraries for you, so there is very little setup needed. You can install their client through pip and load the Carvana dataset separately from your code. They have pretty good documentation and it's easier than dealing with GPU drivers. That said, it's still probably more cost effective to buy a GPU for Kaggle contests.
", "rawMarkdown": "Floyd installs all of the libraries for you, so there is very little setup needed. You can install their client through pip and load the Carvana dataset separately from your code. They have pretty good documentation and it's easier than dealing with GPU drivers. That said, it's still probably more cost effective to buy a GPU for Kaggle contests.", "votes": null }, { "id": "215588", "postDate": "08/22/2017 08:18:10", "content": "Thanks @Steven.
\n\nI have a plan for the new hardware in a few weeks. For now I have to make do with my CPU plus cloud service of some kind in a cost effective way and conclude this competition.
", "rawMarkdown": "Thanks @Steven.\n\nI have a plan for the new hardware in a few weeks. For now I have to make do with my CPU plus cloud service of some kind in a cost effective way and conclude this competition.", "votes": null }, { "id": "216009", "postDate": "08/24/2017 00:09:25", "content": "Hello, \nI am running this competition with the \"keras starter\" with an AWS p2.xlarge \"spot instance\", I set the bid price at 0.2 usd so it only cost me 0.2 usd per hour in North Virginia region.\nIt took me about 8 hours for a complete training, testing, submit cycle.
\n\nThe \"spot instance\" is way much cheaper than normal \"on demand\" instance, with one big drawback that the data will disappear after shutdown.\nSo be prepared to let the instance run for a few hours or days to worth the effort setting it up, or use a \"on demand\" instance instead.
", "rawMarkdown": "Hello, \nI am running this competition with the \"keras starter\" with an AWS p2.xlarge \"spot instance\", I set the bid price at 0.2 usd so it only cost me 0.2 usd per hour in North Virginia region.\nIt took me about 8 hours for a complete training, testing, submit cycle.\n\nThe \"spot instance\" is way much cheaper than normal \"on demand\" instance, with one big drawback that the data will disappear after shutdown.\nSo be prepared to let the instance run for a few hours or days to worth the effort setting it up, or use a \"on demand\" instance instead.", "votes": null }, { "id": "216017", "postDate": "08/24/2017 01:35:12", "content": "Thanks @Pachinko, I will try that.
\n\nI am running Peter's Keras starter locally just to check that everything runs fine before I move it to the cloud but I am getting the following error in train.py do you have any idea why? It happens in the \"validation_steps=np.ceil(float(len(ids_valid_split)) / float(batch_size)))\" of the model.fit_generator function.
\n\nException in thread Thread-3:\nTraceback (most recent call last):\n File \"C:\\Python34\\lib\\threading.py\", line 914, in _bootstrap_inner\n self.run()\n File \"C:\\Python34\\lib\\threading.py\", line 862, in run\n self._target(*self._args, **self._kwargs)\n File \"C:\\Python34\\lib\\site-packages\\keras\\engine\\training.py\", line 606, in data_generator_task\n generator_output = next(self._generator)\n File \"C:\\DataScience\\Kaggle\\CarvanaImageMasking\\train.py\", line 137, in valid_generator\n mask = cv2.resize(mask, (input_size, input_size))\ncv2.error: D:\\Build\\OpenCV\\opencv-3.2.0\\modules\\imgproc\\src\\imgwarp.cpp:3492: error: (-215) ssize.width > 0 && ssize.height > 0 in function cv::resize
\n\nTraceback (most recent call last):\n File \"C:\\DataScience\\Kaggle\\CarvanaImageMasking\\train.py\", line 170, in \n validation_steps=np.ceil(float(len(ids_valid_split)) / float(batch_size)))\n File \"C:\\Python34\\lib\\site-packages\\keras\\legacy\\interfaces.py\", line 88, in wrapper\n return func(*args, **kwargs)\n File \"C:\\Python34\\lib\\site-packages\\keras\\engine\\training.py\", line 1898, in fit_generator\n pickle_safe=pickle_safe)\n File \"C:\\Python34\\lib\\site-packages\\keras\\legacy\\interfaces.py\", line 88, in wrapper\n return func(*args, **kwargs)\n File \"C:\\Python34\\lib\\site-packages\\keras\\engine\\training.py\", line 1984, in evaluate_generator\n str(generator_output))\nValueError: output of generator should be a tuple (x, y, sample_weight) or (x, y). Found: None
", "rawMarkdown": "Thanks @Pachinko, I will try that.\n\nI am running Peter's Keras starter locally just to check that everything runs fine before I move it to the cloud but I am getting the following error in train.py do you have any idea why? It happens in the \"validation_steps=np.ceil(float(len(ids_valid_split)) / float(batch_size)))\" of the model.fit_generator function.\n\nException in thread Thread-3:\nTraceback (most recent call last):\n File \"C:\\Python34\\lib\\threading.py\", line 914, in _bootstrap_inner\n self.run()\n File \"C:\\Python34\\lib\\threading.py\", line 862, in run\n self._target(*self._args, **self._kwargs)\n File \"C:\\Python34\\lib\\site-packages\\keras\\engine\\training.py\", line 606, in data_generator_task\n generator_output = next(self._generator)\n File \"C:\\DataScience\\Kaggle\\CarvanaImageMasking\\train.py\", line 137, in valid_generator\n mask = cv2.resize(mask, (input_size, input_size))\ncv2.error: D:\\Build\\OpenCV\\opencv-3.2.0\\modules\\imgproc\\src\\imgwarp.cpp:3492: error: (-215) ssize.width > 0 && ssize.height > 0 in function cv::resize\n\n\nTraceback (most recent call last):\n File \"C:\\DataScience\\Kaggle\\CarvanaImageMasking\\train.py\", line 170, in", "votes": null }, { "id": "216444", "postDate": "08/25/2017 20:02:46", "content": "It looks like a \"cv2 size error\". \nMaybe make sure cv2 really read the correct filename, correct full pathname, not an empty image?
\n\nI would put the codes doing read image in a separate sub function, then have the generators call it.\nIt's easier to debug.
", "rawMarkdown": "It looks like a \"cv2 size error\". \nMaybe make sure cv2 really read the correct filename, correct full pathname, not an empty image?\n\nI would put the codes doing read image in a separate sub function, then have the generators call it.\nIt's easier to debug.", "votes": null }, { "id": "216449", "postDate": "08/25/2017 20:27:47", "content": "If you're a student, you can get an AWS account through GitHub education with $200 free credit :).
", "rawMarkdown": "If you're a student, you can get an AWS account through GitHub education with $200 free credit :).", "votes": null }, { "id": "216481", "postDate": "08/25/2017 23:23:23", "content": "Thanks Pachinko. I found one missing file and added it. For some reason, my conversion SW missed a file whilst converting the masks to .png
\n\nConfirming the missing was the problem even though when I printed the ids of the input files it did not show that file. Anyway, I am able to train successfully.
", "rawMarkdown": "Thanks Pachinko. I found one missing file and added it. For some reason, my conversion SW missed a file whilst converting the masks to .png\n\nConfirming the missing was the problem even though when I printed the ids of the input files it did not show that file. Anyway, I am able to train successfully.", "votes": null }, { "id": "216482", "postDate": "08/25/2017 23:26:59", "content": "Thanks @CraigGlastonbury.
\n\nI am not a student but it is good to know as some of my family members may be interested in that.\nAre you from Glastonbury? The closest I got to there was living in Crawley :-)
", "rawMarkdown": "Thanks @CraigGlastonbury.\n\nI am not a student but it is good to know as some of my family members may be interested in that.\nAre you from Glastonbury? The closest I got to there was living in Crawley :-)", "votes": null }, { "id": "217570", "postDate": "08/31/2017 04:33:11", "content": "I went with Floyd see my comments above.
", "rawMarkdown": "I went with [Floyd](https://www.floydhub.com/projects) see my comments above.", "votes": null } ], "comments": [ { "id": 215374, "author_name": "heyt0ny", "author_url": "", "post_date": "08/21/2017 11:32:15", "content": "Unfortunately, I guess the only advice is to rent/buy GPU.
\n\nOk, maybe there are some other approaches that can benefit CPU users, but 99% of guys here use CNNs, that are much faster on GPU than CPU.
\n\nSo if you on a budget you can buy some old card - it will still be faster than i7. For example I had 660ti 2gb and it was capable to train 256 unet with batch size=2 - and you can score around 0.994 LB (on i7 it couldn't even start training:-)). If you will decide to buy new GPU - check out this great post http://timdettmers.com/2017/04/09/which-gpu-for-deep-learning/. Or try Google Cloud GPU - they have 300$ coupon.
", "votes": null, "replies": [] }, { "id": 215384, "author_name": "gaarv1911", "author_url": "", "post_date": "08/21/2017 12:19:19", "content": "Preemptible instances with 8-16 CPUs on Google Cloud are very cheap (> 0.1$ per hour) and can be reliably used given you implement a checkpoint mechanism on training and testing. It's not as fast but its time/cost can't be beat (http://minimaxir.com/2017/07/cpu-or-gpu/ - to be taken with a grain of salt but still interesting).
\n\nIf you really want/need a GPU, a new GC account come with 300$ like already mentioned.
", "votes": null, "replies": [] }, { "id": 215549, "author_name": "sheriytm", "author_url": "", "post_date": "08/22/2017 05:26:34", "content": "Thanks @Feels_g00d_man and @Gaarv for your very useful suggestions.
\n\nI am thinking of trying Floyd. Has anybody here used it? Any advise on the best way to set it up for this contest? I briefly checked their website but will register in a few days when I get a chance.
", "votes": null, "replies": [ { "id": 215558, "author_name": "stevenknguyen", "author_url": "", "post_date": "08/22/2017 05:57:09", "content": "Floyd installs all of the libraries for you, so there is very little setup needed. You can install their client through pip and load the Carvana dataset separately from your code. They have pretty good documentation and it's easier than dealing with GPU drivers. That said, it's still probably more cost effective to buy a GPU for Kaggle contests.
", "votes": null, "replies": [] }, { "id": 215588, "author_name": "sheriytm", "author_url": "", "post_date": "08/22/2017 08:18:10", "content": "Thanks @Steven.
\n\nI have a plan for the new hardware in a few weeks. For now I have to make do with my CPU plus cloud service of some kind in a cost effective way and conclude this competition.
", "votes": null, "replies": [] } ] }, { "id": 216009, "author_name": "pachinko", "author_url": "", "post_date": "08/24/2017 00:09:25", "content": "Hello, \nI am running this competition with the \"keras starter\" with an AWS p2.xlarge \"spot instance\", I set the bid price at 0.2 usd so it only cost me 0.2 usd per hour in North Virginia region.\nIt took me about 8 hours for a complete training, testing, submit cycle.
\n\nThe \"spot instance\" is way much cheaper than normal \"on demand\" instance, with one big drawback that the data will disappear after shutdown.\nSo be prepared to let the instance run for a few hours or days to worth the effort setting it up, or use a \"on demand\" instance instead.
", "votes": null, "replies": [ { "id": 216017, "author_name": "sheriytm", "author_url": "", "post_date": "08/24/2017 01:35:12", "content": "Thanks @Pachinko, I will try that.
\n\nI am running Peter's Keras starter locally just to check that everything runs fine before I move it to the cloud but I am getting the following error in train.py do you have any idea why? It happens in the \"validation_steps=np.ceil(float(len(ids_valid_split)) / float(batch_size)))\" of the model.fit_generator function.
\n\nException in thread Thread-3:\nTraceback (most recent call last):\n File \"C:\\Python34\\lib\\threading.py\", line 914, in _bootstrap_inner\n self.run()\n File \"C:\\Python34\\lib\\threading.py\", line 862, in run\n self._target(*self._args, **self._kwargs)\n File \"C:\\Python34\\lib\\site-packages\\keras\\engine\\training.py\", line 606, in data_generator_task\n generator_output = next(self._generator)\n File \"C:\\DataScience\\Kaggle\\CarvanaImageMasking\\train.py\", line 137, in valid_generator\n mask = cv2.resize(mask, (input_size, input_size))\ncv2.error: D:\\Build\\OpenCV\\opencv-3.2.0\\modules\\imgproc\\src\\imgwarp.cpp:3492: error: (-215) ssize.width > 0 && ssize.height > 0 in function cv::resize
\n\nTraceback (most recent call last):\n File \"C:\\DataScience\\Kaggle\\CarvanaImageMasking\\train.py\", line 170, in \n validation_steps=np.ceil(float(len(ids_valid_split)) / float(batch_size)))\n File \"C:\\Python34\\lib\\site-packages\\keras\\legacy\\interfaces.py\", line 88, in wrapper\n return func(*args, **kwargs)\n File \"C:\\Python34\\lib\\site-packages\\keras\\engine\\training.py\", line 1898, in fit_generator\n pickle_safe=pickle_safe)\n File \"C:\\Python34\\lib\\site-packages\\keras\\legacy\\interfaces.py\", line 88, in wrapper\n return func(*args, **kwargs)\n File \"C:\\Python34\\lib\\site-packages\\keras\\engine\\training.py\", line 1984, in evaluate_generator\n str(generator_output))\nValueError: output of generator should be a tuple (x, y, sample_weight) or (x, y). Found: None
", "votes": null, "replies": [] }, { "id": 216444, "author_name": "pachinko", "author_url": "", "post_date": "08/25/2017 20:02:46", "content": "It looks like a \"cv2 size error\". \nMaybe make sure cv2 really read the correct filename, correct full pathname, not an empty image?
\n\nI would put the codes doing read image in a separate sub function, then have the generators call it.\nIt's easier to debug.
", "votes": null, "replies": [] }, { "id": 216481, "author_name": "sheriytm", "author_url": "", "post_date": "08/25/2017 23:23:23", "content": "Thanks Pachinko. I found one missing file and added it. For some reason, my conversion SW missed a file whilst converting the masks to .png
\n\nConfirming the missing was the problem even though when I printed the ids of the input files it did not show that file. Anyway, I am able to train successfully.
", "votes": null, "replies": [] } ] }, { "id": 216449, "author_name": "craigglastonbury", "author_url": "", "post_date": "08/25/2017 20:27:47", "content": "If you're a student, you can get an AWS account through GitHub education with $200 free credit :).
", "votes": null, "replies": [ { "id": 216482, "author_name": "sheriytm", "author_url": "", "post_date": "08/25/2017 23:26:59", "content": "Thanks @CraigGlastonbury.
\n\nI am not a student but it is good to know as some of my family members may be interested in that.\nAre you from Glastonbury? The closest I got to there was living in Crawley :-)
", "votes": null, "replies": [] } ] }, { "id": 217570, "author_name": "sheriytm", "author_url": "", "post_date": "08/31/2017 04:33:11", "content": "I went with Floyd see my comments above.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "215357": "Hi everyone, any advise for the resource challenged, I meant CPU users not GPU for this competition?\n\nUpdate on 08/30th/2017:\n\nI signed up with [Floyd](https://www.floydhub.com/projects) paid account and used their GPU. I expected the GPU to be faster than it is when I ran my training job. I also tested prediction using their CPU which kept giving a memory error when the same thing runs on my CPU labtop without any memory issue.\n\nMy verdict - [Floyd](https://www.floydhub.com/projects) is easy to setup, lacks AWS/ GCP features but much cheaper.", "215374": "Unfortunately, I guess the only advice is to rent/buy GPU. \n\nOk, maybe there are some other approaches that can benefit CPU users, but 99% of guys here use CNNs, that are much faster on GPU than CPU. \n\nSo if you on a budget you can buy some old card - it will still be faster than i7. For example I had 660ti 2gb and it was capable to train 256 unet with batch size=2 - and you can score around 0.994 LB (on i7 it couldn't even start training:-)). If you will decide to buy new GPU - check out this great post http://timdettmers.com/2017/04/09/which-gpu-for-deep-learning/. Or try Google Cloud GPU - they have 300$ coupon.", "215384": "Preemptible instances with 8-16 CPUs on Google Cloud are very cheap (> 0.1$ per hour) and can be reliably used given you implement a checkpoint mechanism on training and testing. It's not as fast but its time/cost can't be beat (http://minimaxir.com/2017/07/cpu-or-gpu/ - to be taken with a grain of salt but still interesting).\n\nIf you really want/need a GPU, a new GC account come with 300$ like already mentioned.", "215549": "Thanks @Feels_g00d_man and @Gaarv for your very useful suggestions.\n\nI am thinking of trying [Floyd](https://www.floydhub.com/). Has anybody here used it? Any advise on the best way to set it up for this contest? I briefly checked their website but will register in a few days when I get a chance.", "215558": "Floyd installs all of the libraries for you, so there is very little setup needed. You can install their client through pip and load the Carvana dataset separately from your code. They have pretty good documentation and it's easier than dealing with GPU drivers. That said, it's still probably more cost effective to buy a GPU for Kaggle contests.", "215588": "Thanks @Steven.\n\nI have a plan for the new hardware in a few weeks. For now I have to make do with my CPU plus cloud service of some kind in a cost effective way and conclude this competition.", "216009": "Hello, \nI am running this competition with the \"keras starter\" with an AWS p2.xlarge \"spot instance\", I set the bid price at 0.2 usd so it only cost me 0.2 usd per hour in North Virginia region.\nIt took me about 8 hours for a complete training, testing, submit cycle.\n\nThe \"spot instance\" is way much cheaper than normal \"on demand\" instance, with one big drawback that the data will disappear after shutdown.\nSo be prepared to let the instance run for a few hours or days to worth the effort setting it up, or use a \"on demand\" instance instead.", "216017": "Thanks @Pachinko, I will try that.\n\nI am running Peter's Keras starter locally just to check that everything runs fine before I move it to the cloud but I am getting the following error in train.py do you have any idea why? It happens in the \"validation_steps=np.ceil(float(len(ids_valid_split)) / float(batch_size)))\" of the model.fit_generator function.\n\nException in thread Thread-3:\nTraceback (most recent call last):\n File \"C:\\Python34\\lib\\threading.py\", line 914, in _bootstrap_inner\n self.run()\n File \"C:\\Python34\\lib\\threading.py\", line 862, in run\n self._target(*self._args, **self._kwargs)\n File \"C:\\Python34\\lib\\site-packages\\keras\\engine\\training.py\", line 606, in data_generator_task\n generator_output = next(self._generator)\n File \"C:\\DataScience\\Kaggle\\CarvanaImageMasking\\train.py\", line 137, in valid_generator\n mask = cv2.resize(mask, (input_size, input_size))\ncv2.error: D:\\Build\\OpenCV\\opencv-3.2.0\\modules\\imgproc\\src\\imgwarp.cpp:3492: error: (-215) ssize.width > 0 && ssize.height > 0 in function cv::resize\n\n\nTraceback (most recent call last):\n File \"C:\\DataScience\\Kaggle\\CarvanaImageMasking\\train.py\", line 170, in", "216444": "It looks like a \"cv2 size error\". \nMaybe make sure cv2 really read the correct filename, correct full pathname, not an empty image?\n\nI would put the codes doing read image in a separate sub function, then have the generators call it.\nIt's easier to debug.", "216449": "If you're a student, you can get an AWS account through GitHub education with $200 free credit :).", "216481": "Thanks Pachinko. I found one missing file and added it. For some reason, my conversion SW missed a file whilst converting the masks to .png\n\nConfirming the missing was the problem even though when I printed the ids of the input files it did not show that file. Anyway, I am able to train successfully.", "216482": "Thanks @CraigGlastonbury.\n\nI am not a student but it is good to know as some of my family members may be interested in that.\nAre you from Glastonbury? The closest I got to there was living in Crawley :-)", "217570": "I went with [Floyd](https://www.floydhub.com/projects) see my comments above." }, "source": "meta" }