![\"32bit_limit\"](\"https://i.imgur.com/BdCBRHH.jpg\")
Based on the discussions in the forum (for e.g.: here, here, and the replies by @ldm314 and @iafoss to my previous version of this post) I've modified my plots. I hope now it makes more sense. This is merely based on my experience of trying to get it working on the cloud platforms that I have access to, like Google Colab. I do try to use the memory map mode as much as possible.
\n\nHere, I show the CPU RAM requirements based on image size for this competition. I hope this will help you choose the size of the resampled input image on the hardware you have for training your model.
\n\nFor example:
\n\nIf you are using the Google Colab with a CPU limit of 12GB, you will have to stick to an 32 bit array size of 128x128.
\n\nI would be happy to get feedback from the community on how to increase this pixel size. I use Keras.
\n\nwith float32\n
with float16\n![\"16bit_limit\"](\"https://i.imgur.com/0sZy2US.jpg\")
Based on the discussions in the forum (for e.g.: here, here, and the replies by @ldm314 and @iafoss to my previous version of this post) I've modified my plots. I hope now it makes more sense. This is merely based on my experience of trying to get it working on the cloud platforms that I have access to, like Google Colab. I do try to use the memory map mode as much as possible.
\n\nHere, I show the CPU RAM requirements based on image size for this competition. I hope this will help you choose the size of the resampled input image on the hardware you have for training your model.
\n\nFor example:
\n\nIf you are using the Google Colab with a CPU limit of 12GB, you will have to stick to an 32 bit array size of 128x128.
\n\nI would be happy to get feedback from the community on how to increase this pixel size. I use Keras.
\n\nwith float32\n
with float16\n
How about reducing the batch size? With light models like ResNet34 one can go bs = 16 on 512x512 images on K80. Higher capacity models still can be trained, but with 8 or even 4 images per batch. The only problem is that K80 is quite slow...
", "rawMarkdown": "How about reducing the batch size? With light models like ResNet34 one can go bs = 16 on 512x512 images on K80. Higher capacity models still can be trained, but with 8 or even 4 images per batch. The only problem is that K80 is quite slow...", "votes": null }, { "id": "412812", "postDate": "10/30/2018 20:24:11", "content": "You're right. I didn't consider here the batch size or the model.
", "rawMarkdown": "You're right. I didn't consider here the batch size or the model.", "votes": null }, { "id": "412819", "postDate": "10/30/2018 20:53:18", "content": "I agree this is totally dependent on the model. I use 512x512x3 and works fine with the GTX Titan X. If I have all conv blocks frozen I can run batches of 64 a GTX 970 with only 3.5GB ram. The full model I can run in batches of 12 on the 970 or 64 on the Titan X.
", "rawMarkdown": "I agree this is totally dependent on the model. I use 512x512x3 and works fine with the GTX Titan X. If I have all conv blocks frozen I can run batches of 64 a GTX 970 with only 3.5GB ram. The full model I can run in batches of 12 on the 970 or 64 on the Titan X.", "votes": null }, { "id": "413256", "postDate": "10/31/2018 15:33:34", "content": "I like the update, it looks more clear to me this way. The biggest thing to do in Keras is to use generators that feed from a directory. The training set I use ranges from 30K to 125K 512x512x3. My validation set of ~7000 images I keep in memory and at 512x512 it takes up 40GB or so.
", "rawMarkdown": "I like the update, it looks more clear to me this way. The biggest thing to do in Keras is to use generators that feed from a directory. The training set I use ranges from 30K to 125K 512x512x3. My validation set of ~7000 images I keep in memory and at 512x512 it takes up 40GB or so.", "votes": null }, { "id": "413304", "postDate": "10/31/2018 16:49:31", "content": "Can you have a dataloader in keras that reads images from files rather than RAM (I do not use keras, so I'm not sure about it, but in fast.ai it is the default option)? To optimize loading, you can do some preprocessing the images, like scaling and fusion in single 4-channel images...
", "rawMarkdown": "Can you have a dataloader in keras that reads images from files rather than RAM (I do not use keras, so I'm not sure about it, but in fast.ai it is the default option)? To optimize loading, you can do some preprocessing the images, like scaling and fusion in single 4-channel images...", "votes": null }, { "id": "413346", "postDate": "10/31/2018 18:57:44", "content": "Yes, I preprocess to RGB images using image magick. You can use flow from directory to load images from disk with not much ram usage: https://keras.io/preprocessing/image/#flow_from_directory
", "rawMarkdown": "Yes, I preprocess to RGB images using image magick. You can use flow from directory to load images from disk with not much ram usage: https://keras.io/preprocessing/image/#flow_from_directory", "votes": null }, { "id": "415110", "postDate": "11/04/2018 12:27:11", "content": "Hi Brian,\nflow_from_directory does not seem to work for multi-label setting... Did you customize it yourself?
", "rawMarkdown": "Hi Brian,\nflow_from_directory does not seem to work for multi-label setting... Did you customize it yourself?", "votes": null }, { "id": "415916", "postDate": "11/05/2018 22:53:17", "content": "I started with code from this kernel: https://www.kaggle.com/nikhilroxtomar/transfer-learning-for-human-protein-submission
", "rawMarkdown": "I started with code from this kernel: https://www.kaggle.com/nikhilroxtomar/transfer-learning-for-human-protein-submission", "votes": null }, { "id": "416084", "postDate": "11/06/2018 07:04:45", "content": "thanks!
", "rawMarkdown": "thanks!", "votes": null } ], "comments": [ { "id": 412750, "author_name": "iafoss", "author_url": "", "post_date": "10/30/2018 18:03:07", "content": "How about reducing the batch size? With light models like ResNet34 one can go bs = 16 on 512x512 images on K80. Higher capacity models still can be trained, but with 8 or even 4 images per batch. The only problem is that K80 is quite slow...
", "votes": null, "replies": [ { "id": 412812, "author_name": "vignam", "author_url": "", "post_date": "10/30/2018 20:24:11", "content": "You're right. I didn't consider here the batch size or the model.
", "votes": null, "replies": [] } ] }, { "id": 412819, "author_name": "ldm314", "author_url": "", "post_date": "10/30/2018 20:53:18", "content": "I agree this is totally dependent on the model. I use 512x512x3 and works fine with the GTX Titan X. If I have all conv blocks frozen I can run batches of 64 a GTX 970 with only 3.5GB ram. The full model I can run in batches of 12 on the 970 or 64 on the Titan X.
", "votes": null, "replies": [] }, { "id": 413256, "author_name": "ldm314", "author_url": "", "post_date": "10/31/2018 15:33:34", "content": "I like the update, it looks more clear to me this way. The biggest thing to do in Keras is to use generators that feed from a directory. The training set I use ranges from 30K to 125K 512x512x3. My validation set of ~7000 images I keep in memory and at 512x512 it takes up 40GB or so.
", "votes": null, "replies": [] }, { "id": 413304, "author_name": "iafoss", "author_url": "", "post_date": "10/31/2018 16:49:31", "content": "Can you have a dataloader in keras that reads images from files rather than RAM (I do not use keras, so I'm not sure about it, but in fast.ai it is the default option)? To optimize loading, you can do some preprocessing the images, like scaling and fusion in single 4-channel images...
", "votes": null, "replies": [ { "id": 413346, "author_name": "ldm314", "author_url": "", "post_date": "10/31/2018 18:57:44", "content": "Yes, I preprocess to RGB images using image magick. You can use flow from directory to load images from disk with not much ram usage: https://keras.io/preprocessing/image/#flow_from_directory
", "votes": null, "replies": [] }, { "id": 415110, "author_name": "stecasasso", "author_url": "", "post_date": "11/04/2018 12:27:11", "content": "Hi Brian,\nflow_from_directory does not seem to work for multi-label setting... Did you customize it yourself?
", "votes": null, "replies": [] }, { "id": 415916, "author_name": "ldm314", "author_url": "", "post_date": "11/05/2018 22:53:17", "content": "I started with code from this kernel: https://www.kaggle.com/nikhilroxtomar/transfer-learning-for-human-protein-submission
", "votes": null, "replies": [] }, { "id": 416084, "author_name": "stecasasso", "author_url": "", "post_date": "11/06/2018 07:04:45", "content": "thanks!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "412703": "Based on the discussions in the forum (for e.g.: [here][1], [here][2], and the replies by @ldm314 and @iafoss to my previous version of this post) I've modified my plots. I hope now it makes more sense. This is merely based on my experience of trying to get it working on the cloud platforms that I have access to, like Google Colab. I do try to use the memory map mode as much as possible.\n\nHere, I show the CPU RAM requirements based on image size for this competition. I hope this will help you choose the size of the resampled input image on the hardware you have for training your model. \n\nFor example:\n\nIf you are using the Google Colab with a CPU limit of 12GB, you will have to stick to an 32 bit array size of 128x128. \n\nI would be happy to get feedback from the community on how to increase this pixel size. I use Keras.\n\n----------\nwith **float32**\n![32bit_limit][3]\n\nwith **float16**\n![16bit_limit][4]\n\n----------\n\n\n [1]: https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/70092#412817\n [2]: https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/70092#412784\n [3]: https://i.imgur.com/BdCBRHH.jpg\n [4]: https://i.imgur.com/0sZy2US.jpg", "412750": "How about reducing the batch size? With light models like ResNet34 one can go bs = 16 on 512x512 images on K80. Higher capacity models still can be trained, but with 8 or even 4 images per batch. The only problem is that K80 is quite slow...", "412812": "You're right. I didn't consider here the batch size or the model.", "412819": "I agree this is totally dependent on the model. I use 512x512x3 and works fine with the GTX Titan X. If I have all conv blocks frozen I can run batches of 64 a GTX 970 with only 3.5GB ram. The full model I can run in batches of 12 on the 970 or 64 on the Titan X.", "413256": "I like the update, it looks more clear to me this way. The biggest thing to do in Keras is to use generators that feed from a directory. The training set I use ranges from 30K to 125K 512x512x3. My validation set of ~7000 images I keep in memory and at 512x512 it takes up 40GB or so.", "413304": "Can you have a dataloader in keras that reads images from files rather than RAM (I do not use keras, so I'm not sure about it, but in fast.ai it is the default option)? To optimize loading, you can do some preprocessing the images, like scaling and fusion in single 4-channel images...", "413346": "Yes, I preprocess to RGB images using image magick. You can use flow from directory to load images from disk with not much ram usage: https://keras.io/preprocessing/image/#flow_from_directory", "415110": "Hi Brian,\nflow_from_directory does not seem to work for multi-label setting... Did you customize it yourself?", "415916": "I started with code from this kernel: https://www.kaggle.com/nikhilroxtomar/transfer-learning-for-human-protein-submission", "416084": "thanks!" }, "source": "meta" }