I am new to CV, Could you give me some advice?
", "messages": [ { "id": "365314", "postDate": "08/02/2018 10:31:54", "content": "I am new to CV, Could you give me some advice?
", "rawMarkdown": "I am new to CV, Could you give me some advice?", "votes": null }, { "id": "365449", "postDate": "08/02/2018 16:29:42", "content": "If you're newbee for CV, you should try Google Colab examples first or look at the general DL networks
", "rawMarkdown": "If you're newbee for CV, you should try Google Colab examples first or look at the general DL networks", "votes": null }, { "id": "365490", "postDate": "08/02/2018 19:10:22", "content": "In general, the more the better :) I would recommend to have a least a GPU with 6 GB of memory to run a decent model on full sized images.
", "rawMarkdown": "In general, the more the better :) I would recommend to have a least a GPU with 6 GB of memory to run a decent model on full sized images.", "votes": null }, { "id": "365596", "postDate": "08/03/2018 01:26:52", "content": "thanks for your reply, I would try it.
", "rawMarkdown": "thanks for your reply, I would try it.", "votes": null }, { "id": "365601", "postDate": "08/03/2018 01:40:08", "content": "Thanks for your advice, I have already got a gtx 1060 with 6G memory, 16GB ram and a I5 6500 cpu, can I start the competion? Beacuse I saw the training set and test set are both up to 12GB, it is scaring me.
", "rawMarkdown": "Thanks for your advice, I have already got a gtx 1060 with 6G memory, 16GB ram and a I5 6500 cpu, can I start the competion? Beacuse I saw the training set and test set are both up to 12GB, it is scaring me.", "votes": null }, { "id": "365605", "postDate": "08/03/2018 02:08:11", "content": "The issue is usually not the size of the dataset, but rather how big the images are. In this case they are 768x768 pixels, which should allow you to run a medium sized model on the full image. However inference (test time prediction) will take a long time due to the 88k images.
", "rawMarkdown": "The issue is usually not the size of the dataset, but rather how big the images are. In this case they are 768x768 pixels, which should allow you to run a medium sized model on the full image. However inference (test time prediction) will take a long time due to the 88k images.", "votes": null }, { "id": "365607", "postDate": "08/03/2018 02:22:41", "content": "Thank again, Michael, it help me a lot. I will download the dataset and try to run it
", "rawMarkdown": "Thank again, Michael, it help me a lot. I will download the dataset and try to run it", "votes": null }, { "id": "369069", "postDate": "08/11/2018 20:54:08", "content": "I am running a GE Force 1060 with 6 gig ram and I noticed that while running tensorflow and keras the GPU % used only goes between 2% to 4% while the main CPU (an intel i7 8th gen) goes up to about 20%. The entire GPU memory gets allocated but the cpu usage seems low?
\nWhen I run I see that tensorflow recognizes and uses the GPU.\nI expected the GPU or CPU to get pegged out at least between 80-100% but it doesn't get anywhere near that.\nThe hard drive is a 1 terabyte solid state drive.\nWhat could be the bottle neck?
\nAny recommendations on how to get the rag out of the carburetor?
Tensorflow automatically allocates all the resources on the GPU(s) it can find, that is normal. You might want to look at your dataloader/pre-processing parts. That's usually one of the main reasons why the GPU is not fully utilized
", "rawMarkdown": "Tensorflow automatically allocates all the resources on the GPU(s) it can find, that is normal. You might want to look at your dataloader/pre-processing parts. That's usually one of the main reasons why the GPU is not fully utilized", "votes": null }, { "id": "369350", "postDate": "08/12/2018 18:48:18", "content": "When I look at the code it's in this loop getting test images and predictiing where the ships are located.\nThat's when I see the GPU, CPU percentages I described.
\n\nfor c_img_name in tqdm_notebook(test_paths):\n c_path = os.path.join(test_image_dir, c_img_name)\n c_img = imread(c_path)\n c_img = np.expand_dims(c_img, 0)/255.0\n print(\"predicting :\" + c_img_name)\n cur_seg = fullres_model.predict(c_img)[0]\n print(\" prediction done opening binary\")\n cur_seg = binary_opening(cur_seg>0.5, np.expand_dims(disk(2), -1))\n cur_rles = multi_rle_encode(cur_seg)\n if len(cur_rles)>0:\n for c_rle in cur_rles:\n out_pred_rows += [{'ImageId': c_img_name, 'EncodedPixels': c_rle}]\n else:\n out_pred_rows += [{'ImageId': c_img_name, 'EncodedPixels': None}]\n gc.collect()
\n\nprior to that loop the model is loaded using:
\n\nfrom keras import models, layers\nfullres_model = models.load_model(\"C:\\Users\\Gerry\\ships\\fullres_model.h5\", compile=False)\nseg_in_shape = fullres_model.get_input_shape_at(0)[1:3]\nseg_out_shape = fullres_model.get_output_shape_at(0)[1:3]\nprint(seg_in_shape, '->', seg_out_shape)
", "rawMarkdown": "When I look at the code it's in this loop getting test images and predictiing where the ships are located.\nThat's when I see the GPU, CPU percentages I described.\n\nfor c_img_name in tqdm_notebook(test_paths):\n c_path = os.path.join(test_image_dir, c_img_name)\n c_img = imread(c_path)\n c_img = np.expand_dims(c_img, 0)/255.0\n print(\"predicting :\" + c_img_name)\n cur_seg = fullres_model.predict(c_img)[0]\n print(\" prediction done opening binary\")\n cur_seg = binary_opening(cur_seg>0.5, np.expand_dims(disk(2), -1))\n cur_rles = multi_rle_encode(cur_seg)\n if len(cur_rles)>0:\n for c_rle in cur_rles:\n out_pred_rows += [{'ImageId': c_img_name, 'EncodedPixels': c_rle}]\n else:\n out_pred_rows += [{'ImageId': c_img_name, 'EncodedPixels': None}]\n gc.collect()\n\nprior to that loop the model is loaded using:\n\nfrom keras import models, layers\nfullres_model = models.load_model(\"C:\\\\Users\\\\Gerry\\\\ships\\\\fullres_model.h5\", compile=False)\nseg_in_shape = fullres_model.get_input_shape_at(0)[1:3]\nseg_out_shape = fullres_model.get_output_shape_at(0)[1:3]\nprint(seg_in_shape, '->', seg_out_shape)", "votes": null }, { "id": "370111", "postDate": "08/14/2018 09:38:32", "content": "i am beginner, anyone can tell what basic things i have to learn before start working on this challenge??, i have studied basic ML algos and pandas, numpy.
", "rawMarkdown": "i am beginner, anyone can tell what basic things i have to learn before start working on this challenge??, i have studied basic ML algos and pandas, numpy.", "votes": null }, { "id": "370308", "postDate": "08/14/2018 15:44:26", "content": "For a medium sized introduction to Deep Learning I recommend Michael Nielsons online book, if you are familiar with the basics, you should be able to work through it in a couple days or less. If you want to go deeper, read the first two parts of Ian Goodfellows Deep Learning book. This will probably take you at least a week, possibly more (depending on your background).
\n\nOnce you are familiar with basic deep learning techniques, I recommend this blog post which gives a nice summary of the current deep learning computer vision tasks and the architectures which can accomplish them.
", "rawMarkdown": "## Deep Learning\nFor a medium sized introduction to Deep Learning I recommend [Michael Nielsons online book][1], if you are familiar with the basics, you should be able to work through it in a couple days or less. If you want to go deeper, read the first two parts of Ian [Goodfellows Deep Learning book][2]. This will probably take you at least a week, possibly more (depending on your background). \n\n## Computer Vision\nOnce you are familiar with basic deep learning techniques, I recommend this [blog post][3] which gives a nice summary of the current deep learning computer vision tasks and the architectures which can accomplish them.\n\n\n [1]: http://neuralnetworksanddeeplearning.com/\n [2]: https://www.deeplearningbook.org/\n [3]: https://medium.com/comet-app/review-of-deep-learning-algorithms-for-object-detection-c1f3d437b852", "votes": null }, { "id": "371089", "postDate": "08/16/2018 00:53:24", "content": "I'm running the same GPU and memory and you may need to scale the images and/or reduce the batch size to train a model.
", "rawMarkdown": "I'm running the same GPU and memory and you may need to scale the images and/or reduce the batch size to train a model.", "votes": null }, { "id": "371959", "postDate": "08/17/2018 22:59:30", "content": "I too have been experiencing some painfully slow prediction when trying to create a submission. I haven't made a submission since getting keras-gpu working, but when i was training on my CPU it took upwards of 24 hours to predict all ~88K test images.
\n\nI think the first thing to do (and what i'm going to experiment with next time i make a submission) is to call predict with a batch size > 1 (which i believe is what your code (and the example kernel by Kevin Mader does)). In other words preprocess a bunch of images first, then call predict and hopefully keep the GPU a little more busy. Then post-process and do any RLE encoding/instance segmentation required.
\n\nMy guess is that creating 3 worker threads (and queues between them), 1 to handle test image preprocessing, one to call predict, and one to handle post-processing would help speed that up immensely .
", "rawMarkdown": "I too have been experiencing some painfully slow prediction when trying to create a submission. I haven't made a submission since getting keras-gpu working, but when i was training on my CPU it took upwards of 24 hours to predict all ~88K test images. \n\nI think the first thing to do (and what i'm going to experiment with next time i make a submission) is to call predict with a batch size > 1 (which i believe is what your code (and the example kernel by Kevin Mader does)). In other words preprocess a bunch of images first, then call predict and hopefully keep the GPU a little more busy. Then post-process and do any RLE encoding/instance segmentation required.\n\nMy guess is that creating 3 worker threads (and queues between them), 1 to handle test image preprocessing, one to call predict, and one to handle post-processing would help speed that up immensely .", "votes": null } ], "comments": [ { "id": 365449, "author_name": "ykpgrr", "author_url": "", "post_date": "08/02/2018 16:29:42", "content": "If you're newbee for CV, you should try Google Colab examples first or look at the general DL networks
", "votes": null, "replies": [ { "id": 365596, "author_name": "xuxiaoyu", "author_url": "", "post_date": "08/03/2018 01:26:52", "content": "thanks for your reply, I would try it.
", "votes": null, "replies": [] } ] }, { "id": 365490, "author_name": "michaelheinzer", "author_url": "", "post_date": "08/02/2018 19:10:22", "content": "In general, the more the better :) I would recommend to have a least a GPU with 6 GB of memory to run a decent model on full sized images.
", "votes": null, "replies": [ { "id": 365601, "author_name": "xuxiaoyu", "author_url": "", "post_date": "08/03/2018 01:40:08", "content": "Thanks for your advice, I have already got a gtx 1060 with 6G memory, 16GB ram and a I5 6500 cpu, can I start the competion? Beacuse I saw the training set and test set are both up to 12GB, it is scaring me.
", "votes": null, "replies": [] }, { "id": 365605, "author_name": "michaelheinzer", "author_url": "", "post_date": "08/03/2018 02:08:11", "content": "The issue is usually not the size of the dataset, but rather how big the images are. In this case they are 768x768 pixels, which should allow you to run a medium sized model on the full image. However inference (test time prediction) will take a long time due to the 88k images.
", "votes": null, "replies": [] }, { "id": 365607, "author_name": "xuxiaoyu", "author_url": "", "post_date": "08/03/2018 02:22:41", "content": "Thank again, Michael, it help me a lot. I will download the dataset and try to run it
", "votes": null, "replies": [] }, { "id": 371089, "author_name": "gerryt100", "author_url": "", "post_date": "08/16/2018 00:53:24", "content": "I'm running the same GPU and memory and you may need to scale the images and/or reduce the batch size to train a model.
", "votes": null, "replies": [] } ] }, { "id": 369069, "author_name": "gerryt100", "author_url": "", "post_date": "08/11/2018 20:54:08", "content": "I am running a GE Force 1060 with 6 gig ram and I noticed that while running tensorflow and keras the GPU % used only goes between 2% to 4% while the main CPU (an intel i7 8th gen) goes up to about 20%. The entire GPU memory gets allocated but the cpu usage seems low?
\nWhen I run I see that tensorflow recognizes and uses the GPU.\nI expected the GPU or CPU to get pegged out at least between 80-100% but it doesn't get anywhere near that.\nThe hard drive is a 1 terabyte solid state drive.\nWhat could be the bottle neck?
\nAny recommendations on how to get the rag out of the carburetor?
Tensorflow automatically allocates all the resources on the GPU(s) it can find, that is normal. You might want to look at your dataloader/pre-processing parts. That's usually one of the main reasons why the GPU is not fully utilized
", "votes": null, "replies": [] }, { "id": 369350, "author_name": "gerryt100", "author_url": "", "post_date": "08/12/2018 18:48:18", "content": "When I look at the code it's in this loop getting test images and predictiing where the ships are located.\nThat's when I see the GPU, CPU percentages I described.
\n\nfor c_img_name in tqdm_notebook(test_paths):\n c_path = os.path.join(test_image_dir, c_img_name)\n c_img = imread(c_path)\n c_img = np.expand_dims(c_img, 0)/255.0\n print(\"predicting :\" + c_img_name)\n cur_seg = fullres_model.predict(c_img)[0]\n print(\" prediction done opening binary\")\n cur_seg = binary_opening(cur_seg>0.5, np.expand_dims(disk(2), -1))\n cur_rles = multi_rle_encode(cur_seg)\n if len(cur_rles)>0:\n for c_rle in cur_rles:\n out_pred_rows += [{'ImageId': c_img_name, 'EncodedPixels': c_rle}]\n else:\n out_pred_rows += [{'ImageId': c_img_name, 'EncodedPixels': None}]\n gc.collect()
\n\nprior to that loop the model is loaded using:
\n\nfrom keras import models, layers\nfullres_model = models.load_model(\"C:\\Users\\Gerry\\ships\\fullres_model.h5\", compile=False)\nseg_in_shape = fullres_model.get_input_shape_at(0)[1:3]\nseg_out_shape = fullres_model.get_output_shape_at(0)[1:3]\nprint(seg_in_shape, '->', seg_out_shape)
", "votes": null, "replies": [] }, { "id": 371959, "author_name": "oewyn000", "author_url": "", "post_date": "08/17/2018 22:59:30", "content": "I too have been experiencing some painfully slow prediction when trying to create a submission. I haven't made a submission since getting keras-gpu working, but when i was training on my CPU it took upwards of 24 hours to predict all ~88K test images.
\n\nI think the first thing to do (and what i'm going to experiment with next time i make a submission) is to call predict with a batch size > 1 (which i believe is what your code (and the example kernel by Kevin Mader does)). In other words preprocess a bunch of images first, then call predict and hopefully keep the GPU a little more busy. Then post-process and do any RLE encoding/instance segmentation required.
\n\nMy guess is that creating 3 worker threads (and queues between them), 1 to handle test image preprocessing, one to call predict, and one to handle post-processing would help speed that up immensely .
", "votes": null, "replies": [] } ] }, { "id": 370111, "author_name": "aabhardev", "author_url": "", "post_date": "08/14/2018 09:38:32", "content": "i am beginner, anyone can tell what basic things i have to learn before start working on this challenge??, i have studied basic ML algos and pandas, numpy.
", "votes": null, "replies": [ { "id": 370308, "author_name": "michaelheinzer", "author_url": "", "post_date": "08/14/2018 15:44:26", "content": "For a medium sized introduction to Deep Learning I recommend Michael Nielsons online book, if you are familiar with the basics, you should be able to work through it in a couple days or less. If you want to go deeper, read the first two parts of Ian Goodfellows Deep Learning book. This will probably take you at least a week, possibly more (depending on your background).
\n\nOnce you are familiar with basic deep learning techniques, I recommend this blog post which gives a nice summary of the current deep learning computer vision tasks and the architectures which can accomplish them.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "365314": "I am new to CV, Could you give me some advice?", "365449": "If you're newbee for CV, you should try Google Colab examples first or look at the general DL networks", "365490": "In general, the more the better :) I would recommend to have a least a GPU with 6 GB of memory to run a decent model on full sized images.", "365596": "thanks for your reply, I would try it.", "365601": "Thanks for your advice, I have already got a gtx 1060 with 6G memory, 16GB ram and a I5 6500 cpu, can I start the competion? Beacuse I saw the training set and test set are both up to 12GB, it is scaring me.", "365605": "The issue is usually not the size of the dataset, but rather how big the images are. In this case they are 768x768 pixels, which should allow you to run a medium sized model on the full image. However inference (test time prediction) will take a long time due to the 88k images.", "365607": "Thank again, Michael, it help me a lot. I will download the dataset and try to run it", "369069": "I am running a GE Force 1060 with 6 gig ram and I noticed that while running tensorflow and keras the GPU % used only goes between 2% to 4% while the main CPU (an intel i7 8th gen) goes up to about 20%. The entire GPU memory gets allocated but the cpu usage seems low? \nWhen I run I see that tensorflow recognizes and uses the GPU.\nI expected the GPU or CPU to get pegged out at least between 80-100% but it doesn't get anywhere near that.\nThe hard drive is a 1 terabyte solid state drive.\nWhat could be the bottle neck? \nAny recommendations on how to get the rag out of the carburetor?", "369090": "Tensorflow automatically allocates all the resources on the GPU(s) it can find, that is normal. You might want to look at your dataloader/pre-processing parts. That's usually one of the main reasons why the GPU is not fully utilized", "369350": "When I look at the code it's in this loop getting test images and predictiing where the ships are located.\nThat's when I see the GPU, CPU percentages I described.\n\nfor c_img_name in tqdm_notebook(test_paths):\n c_path = os.path.join(test_image_dir, c_img_name)\n c_img = imread(c_path)\n c_img = np.expand_dims(c_img, 0)/255.0\n print(\"predicting :\" + c_img_name)\n cur_seg = fullres_model.predict(c_img)[0]\n print(\" prediction done opening binary\")\n cur_seg = binary_opening(cur_seg>0.5, np.expand_dims(disk(2), -1))\n cur_rles = multi_rle_encode(cur_seg)\n if len(cur_rles)>0:\n for c_rle in cur_rles:\n out_pred_rows += [{'ImageId': c_img_name, 'EncodedPixels': c_rle}]\n else:\n out_pred_rows += [{'ImageId': c_img_name, 'EncodedPixels': None}]\n gc.collect()\n\nprior to that loop the model is loaded using:\n\nfrom keras import models, layers\nfullres_model = models.load_model(\"C:\\\\Users\\\\Gerry\\\\ships\\\\fullres_model.h5\", compile=False)\nseg_in_shape = fullres_model.get_input_shape_at(0)[1:3]\nseg_out_shape = fullres_model.get_output_shape_at(0)[1:3]\nprint(seg_in_shape, '->', seg_out_shape)", "370111": "i am beginner, anyone can tell what basic things i have to learn before start working on this challenge??, i have studied basic ML algos and pandas, numpy.", "370308": "## Deep Learning\nFor a medium sized introduction to Deep Learning I recommend [Michael Nielsons online book][1], if you are familiar with the basics, you should be able to work through it in a couple days or less. If you want to go deeper, read the first two parts of Ian [Goodfellows Deep Learning book][2]. This will probably take you at least a week, possibly more (depending on your background). \n\n## Computer Vision\nOnce you are familiar with basic deep learning techniques, I recommend this [blog post][3] which gives a nice summary of the current deep learning computer vision tasks and the architectures which can accomplish them.\n\n\n [1]: http://neuralnetworksanddeeplearning.com/\n [2]: https://www.deeplearningbook.org/\n [3]: https://medium.com/comet-app/review-of-deep-learning-algorithms-for-object-detection-c1f3d437b852", "371089": "I'm running the same GPU and memory and you may need to scale the images and/or reduce the batch size to train a model.", "371959": "I too have been experiencing some painfully slow prediction when trying to create a submission. I haven't made a submission since getting keras-gpu working, but when i was training on my CPU it took upwards of 24 hours to predict all ~88K test images. \n\nI think the first thing to do (and what i'm going to experiment with next time i make a submission) is to call predict with a batch size > 1 (which i believe is what your code (and the example kernel by Kevin Mader does)). In other words preprocess a bunch of images first, then call predict and hopefully keep the GPU a little more busy. Then post-process and do any RLE encoding/instance segmentation required.\n\nMy guess is that creating 3 worker threads (and queues between them), 1 to handle test image preprocessing, one to call predict, and one to handle post-processing would help speed that up immensely ." }, "source": "meta" }