Till now i worked with images which are small and was able to feed them to CNN easily. In this competition each image is really large (12.8 Mb approximately). How can i feed such a large image to a CNN. One option is to reduce the size but again I am not sure if this is the best idea.
\nWhat are the best ways to handle such a large dataset. Any references to articles and research papers would be helpful.
", "messages": [ { "id": 1314251, "postDate": "2021-05-19T04:11:47.473Z", "content": "Till now i worked with images which are small and was able to feed them to CNN easily. In this competition each image is really large (12.8 Mb approximately). How can i feed such a large image to a CNN. One option is to reduce the size but again I am not sure if this is the best idea.
\nWhat are the best ways to handle such a large dataset. Any references to articles and research papers would be helpful.
", "rawMarkdown": "Till now i worked with images which are small and was able to feed them to CNN easily. In this competition each image is really large (12.8 Mb approximately). How can i feed such a large image to a CNN. One option is to reduce the size but again I am not sure if this is the best idea. \n\nWhat are the best ways to handle such a large dataset. Any references to articles and research papers would be helpful.", "votes": 4 }, { "id": 1314393, "postDate": "2021-05-19T06:28:02.717Z", "content": "Use a datagenerator or define your own generator rather than loading images individually.
", "rawMarkdown": "Use a datagenerator or define your own generator rather than loading images individually.", "votes": 1 }, { "id": 1314793, "postDate": "2021-05-19T11:39:08.960Z", "content": "Well, you definitively should resize to a smaller size.
\nThe approach I'm going to take is to preprocess images offline, transforming to JPEG for example 384x384 (but it is an hyper-parameter), and pack images in TFRecord format, which makes loading much faster, good for TPU.
Thanks for your reply @luigisaetta
\nIs it possible that someone can participate in this competition just depending on online computing support like Kaggle and collab.
", "rawMarkdown": "Thanks for your reply @luigisaetta \n\nIs it possible that someone can participate in this competition just depending on online computing support like Kaggle and collab." } ] }, { "id": 1314256, "postDate": "2021-05-19T04:15:54.050Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ], "comments": [ { "id": 1314393, "author_name": "Farhan Hai Khan", "author_url": "", "post_date": "2021-05-19T06:28:02.717000", "content": "Use a datagenerator or define your own generator rather than loading images individually.
", "votes": 1, "replies": [] }, { "id": 1314793, "author_name": "Luigi Saetta", "author_url": "", "post_date": "2021-05-19T11:39:08.960000", "content": "Well, you definitively should resize to a smaller size.
\nThe approach I'm going to take is to preprocess images offline, transforming to JPEG for example 384x384 (but it is an hyper-parameter), and pack images in TFRecord format, which makes loading much faster, good for TPU.
Thanks for your reply @luigisaetta
\nIs it possible that someone can participate in this competition just depending on online computing support like Kaggle and collab.
", "votes": 0, "replies": [] } ] }, { "id": 1314256, "author_name": "", "author_url": "", "post_date": "2021-05-19T04:15:54.050000", "content": "", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "1314251": "Till now i worked with images which are small and was able to feed them to CNN easily. In this competition each image is really large (12.8 Mb approximately). How can i feed such a large image to a CNN. One option is to reduce the size but again I am not sure if this is the best idea. \n\nWhat are the best ways to handle such a large dataset. Any references to articles and research papers would be helpful.", "1314393": "Use a datagenerator or define your own generator rather than loading images individually.", "1314793": "Well, you definitively should resize to a smaller size. \nThe approach I'm going to take is to preprocess images offline, transforming to JPEG for example 384x384 (but it is an hyper-parameter), and pack images in TFRecord format, which makes loading much faster, good for TPU.", "1314256": "" } }