I haven't done image analysis before, and this seemed like it might be a cool competition to use to learn. Whales are cool!
\n\nBut my current rig has 12 GB of RAM - which seems like it might be a little low given the data size? Does it make sense to me for try this competition out, or am I just going to frustrate myself due to hardware limitations?
", "messages": [ { "id": "90617", "postDate": "08/28/2015 06:54:17", "content": "I haven't done image analysis before, and this seemed like it might be a cool competition to use to learn. Whales are cool!
\n\nBut my current rig has 12 GB of RAM - which seems like it might be a little low given the data size? Does it make sense to me for try this competition out, or am I just going to frustrate myself due to hardware limitations?
", "rawMarkdown": "I haven't done image analysis before, and this seemed like it might be a cool competition to use to learn. Whales are cool! \r\n\r\n But my current rig has 12 GB of RAM - which seems like it might be a little low given the data size? Does it make sense to me for try this competition out, or am I just going to frustrate myself due to hardware limitations?", "votes": null }, { "id": "90632", "postDate": "08/28/2015 11:15:56", "content": "It depends on how you process the images. Most likely, you will process them one by one and stores them in a smaller format, to make a clean database you can work on. So you will not need to have much more RAM than the size of a picture.
", "rawMarkdown": "It depends on how you process the images. Most likely, you will process them one by one and stores them in a smaller format, to make a clean database you can work on. So you will not need to have much more RAM than the size of a picture.", "votes": null }, { "id": "90694", "postDate": "08/28/2015 20:00:02", "content": "You'll probably have to write some more code and load the images into memory only right before they are needed and free them afterwards. This introduces some disk overhead, but I've done this many times and it was ok. It helps if you process and save them in batches in some kind of your language native format (like pickle for Python).
", "rawMarkdown": "You'll probably have to write some more code and load the images into memory only right before they are needed and free them afterwards. This introduces some disk overhead, but I've done this many times and it was ok. It helps if you process and save them in batches in some kind of your language native format (like pickle for Python).", "votes": null }, { "id": "90699", "postDate": "08/28/2015 20:20:14", "content": "I agree with Julien and apaszke.\nIf you are concerned about loading your data, take a look at step 2 in this tutorial, imageSet in MATLAB lets you manage data on disk for easy sequential loading one-by-one in a loop. Similar tools I am sure exist in python and other languages where you can load only manageable chunks.
\n\nWhen it comes to learning, approaches like stochastic gradient descent don't require all of your data upfront, but rather tune the parameters iteratively as you provide more data. One of the most popular learning approach that does this are Convolutional Neural Networks which work with minibatches of images.
", "rawMarkdown": "I agree with Julien and apaszke.\r\nIf you are concerned about loading your data, take a look at step 2 in [this tutorial][1], imageSet in MATLAB lets you manage data on disk for easy sequential loading one-by-one in a loop. Similar tools I am sure exist in python and other languages where you can load only manageable chunks.\r\n\r\nWhen it comes to learning, approaches like stochastic gradient descent don't require all of your data upfront, but rather tune the parameters iteratively as you provide more data. One of the most popular learning approach that does this are Convolutional Neural Networks which work with minibatches of images.\r\n\r\n [1]: https://www.kaggle.com/c/noaa-right-whale-recognition/details/creating-a-face-detector-for-whales", "votes": null }, { "id": "90740", "postDate": "08/28/2015 23:19:26", "content": "Cool, thanks for the reassurance guys!
", "rawMarkdown": "Cool, thanks for the reassurance guys!", "votes": null } ], "comments": [ { "id": 90632, "author_name": "rejulien", "author_url": "", "post_date": "08/28/2015 11:15:56", "content": "It depends on how you process the images. Most likely, you will process them one by one and stores them in a smaller format, to make a clean database you can work on. So you will not need to have much more RAM than the size of a picture.
", "votes": null, "replies": [] }, { "id": 90694, "author_name": "apaszke", "author_url": "", "post_date": "08/28/2015 20:00:02", "content": "You'll probably have to write some more code and load the images into memory only right before they are needed and free them afterwards. This introduces some disk overhead, but I've done this many times and it was ok. It helps if you process and save them in batches in some kind of your language native format (like pickle for Python).
", "votes": null, "replies": [] }, { "id": 90699, "author_name": "shashankprasanna", "author_url": "", "post_date": "08/28/2015 20:20:14", "content": "I agree with Julien and apaszke.\nIf you are concerned about loading your data, take a look at step 2 in this tutorial, imageSet in MATLAB lets you manage data on disk for easy sequential loading one-by-one in a loop. Similar tools I am sure exist in python and other languages where you can load only manageable chunks.
\n\nWhen it comes to learning, approaches like stochastic gradient descent don't require all of your data upfront, but rather tune the parameters iteratively as you provide more data. One of the most popular learning approach that does this are Convolutional Neural Networks which work with minibatches of images.
", "votes": null, "replies": [] }, { "id": 90740, "author_name": "ftlftw", "author_url": "", "post_date": "08/28/2015 23:19:26", "content": "Cool, thanks for the reassurance guys!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "90617": "I haven't done image analysis before, and this seemed like it might be a cool competition to use to learn. Whales are cool! \r\n\r\n But my current rig has 12 GB of RAM - which seems like it might be a little low given the data size? Does it make sense to me for try this competition out, or am I just going to frustrate myself due to hardware limitations?", "90632": "It depends on how you process the images. Most likely, you will process them one by one and stores them in a smaller format, to make a clean database you can work on. So you will not need to have much more RAM than the size of a picture.", "90694": "You'll probably have to write some more code and load the images into memory only right before they are needed and free them afterwards. This introduces some disk overhead, but I've done this many times and it was ok. It helps if you process and save them in batches in some kind of your language native format (like pickle for Python).", "90699": "I agree with Julien and apaszke.\r\nIf you are concerned about loading your data, take a look at step 2 in [this tutorial][1], imageSet in MATLAB lets you manage data on disk for easy sequential loading one-by-one in a loop. Similar tools I am sure exist in python and other languages where you can load only manageable chunks.\r\n\r\nWhen it comes to learning, approaches like stochastic gradient descent don't require all of your data upfront, but rather tune the parameters iteratively as you provide more data. One of the most popular learning approach that does this are Convolutional Neural Networks which work with minibatches of images.\r\n\r\n [1]: https://www.kaggle.com/c/noaa-right-whale-recognition/details/creating-a-face-detector-for-whales", "90740": "Cool, thanks for the reassurance guys!" }, "source": "meta" }