A large portion of the time per epoch goes to data loading. With limited resources, we have to do everything we can to reduce the time taken per epoch. This can be done by loading the data fast. \nI have tested the following four methods of data loading in my kernel here\n1. Load Original Images and Resize + Create masks on spot\n2. Load Resized Images + Create masks on spot\n3. Load Resized Images in numpy format + Load masks in numpy format\n4. Load both images and maps from RAM (unfortunately, with the resources I have and the resources kaggle kernels provide, it's not possible to load both images and masks in RAM unless you reduce the size a lot)
\n\nPlease let me know if you know of more things I can do for speeding up the process. \n(Yes I've tried increasing the number of workers but written that in the kernel because of memory constraints)
", "messages": [ { "id": "648418", "postDate": "10/14/2019 06:34:42", "content": "A large portion of the time per epoch goes to data loading. With limited resources, we have to do everything we can to reduce the time taken per epoch. This can be done by loading the data fast. \nI have tested the following four methods of data loading in my kernel here\n1. Load Original Images and Resize + Create masks on spot\n2. Load Resized Images + Create masks on spot\n3. Load Resized Images in numpy format + Load masks in numpy format\n4. Load both images and maps from RAM (unfortunately, with the resources I have and the resources kaggle kernels provide, it's not possible to load both images and masks in RAM unless you reduce the size a lot)
\n\nPlease let me know if you know of more things I can do for speeding up the process. \n(Yes I've tried increasing the number of workers but written that in the kernel because of memory constraints)
", "rawMarkdown": "A large portion of the time per epoch goes to data loading. With limited resources, we have to do everything we can to reduce the time taken per epoch. This can be done by loading the data fast. \nI have tested the following four methods of data loading in my kernel [here](https://www.kaggle.com/timetraveller98/creating-and-testing-fast-dataloaders)\n1. Load Original Images and Resize + Create masks on spot\n2. Load Resized Images + Create masks on spot\n3. Load Resized Images in numpy format + Load masks in numpy format\n4. Load both images and maps from RAM (unfortunately, with the resources I have and the resources kaggle kernels provide, it's not possible to load both images and masks in RAM unless you reduce the size a lot)\n\nPlease let me know if you know of more things I can do for speeding up the process. \n(Yes I've tried increasing the number of workers but written that in the kernel because of memory constraints)", "votes": null }, { "id": "648496", "postDate": "10/14/2019 09:21:39", "content": "I didn't try it yet but nvidia dali looks a good option\nhttps://www.kaggle.com/hirune924/nvidia-dali-the-fastest-data-loading
", "rawMarkdown": "I didn't try it yet but nvidia dali looks a good option\nhttps://www.kaggle.com/hirune924/nvidia-dali-the-fastest-data-loading", "votes": null }, { "id": "648566", "postDate": "10/14/2019 11:16:53", "content": "One thing I did was using multiprocess to resize images faster, you can take a look here and the scripts are also here, maybe it can be even faster if you use multiprocess to transform images and masks to numpy.
", "rawMarkdown": "One thing I did was using multiprocess to resize images faster, you can take a look [here](https://www.kaggle.com/dimitreoliveira/cloud-segmentation-with-utility-scripts-and-keras) and the scripts are also [here](https://www.kaggle.com/dimitreoliveira/cloud-images-segmentation-utillity-script), maybe it can be even faster if you use multiprocess to transform images and masks to numpy.", "votes": null }, { "id": "649410", "postDate": "10/15/2019 10:05:24", "content": "Thanks for sharing, this is something new for me.\nI will take a look.
", "rawMarkdown": "Thanks for sharing, this is something new for me.\nI will take a look.", "votes": null }, { "id": "649412", "postDate": "10/15/2019 10:05:51", "content": "Yes, that could speed up resizing. But since its a one time process I don't this it should matter much.
", "rawMarkdown": "Yes, that could speed up resizing. But since its a one time process I don't this it should matter much.", "votes": null } ], "comments": [ { "id": 648496, "author_name": "mnpinto", "author_url": "", "post_date": "10/14/2019 09:21:39", "content": "I didn't try it yet but nvidia dali looks a good option\nhttps://www.kaggle.com/hirune924/nvidia-dali-the-fastest-data-loading
", "votes": null, "replies": [ { "id": 649410, "author_name": "timetraveller98", "author_url": "", "post_date": "10/15/2019 10:05:24", "content": "Thanks for sharing, this is something new for me.\nI will take a look.
", "votes": null, "replies": [] } ] }, { "id": 648566, "author_name": "dimitreoliveira", "author_url": "", "post_date": "10/14/2019 11:16:53", "content": "One thing I did was using multiprocess to resize images faster, you can take a look here and the scripts are also here, maybe it can be even faster if you use multiprocess to transform images and masks to numpy.
", "votes": null, "replies": [ { "id": 649412, "author_name": "timetraveller98", "author_url": "", "post_date": "10/15/2019 10:05:51", "content": "Yes, that could speed up resizing. But since its a one time process I don't this it should matter much.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "648418": "A large portion of the time per epoch goes to data loading. With limited resources, we have to do everything we can to reduce the time taken per epoch. This can be done by loading the data fast. \nI have tested the following four methods of data loading in my kernel [here](https://www.kaggle.com/timetraveller98/creating-and-testing-fast-dataloaders)\n1. Load Original Images and Resize + Create masks on spot\n2. Load Resized Images + Create masks on spot\n3. Load Resized Images in numpy format + Load masks in numpy format\n4. Load both images and maps from RAM (unfortunately, with the resources I have and the resources kaggle kernels provide, it's not possible to load both images and masks in RAM unless you reduce the size a lot)\n\nPlease let me know if you know of more things I can do for speeding up the process. \n(Yes I've tried increasing the number of workers but written that in the kernel because of memory constraints)", "648496": "I didn't try it yet but nvidia dali looks a good option\nhttps://www.kaggle.com/hirune924/nvidia-dali-the-fastest-data-loading", "648566": "One thing I did was using multiprocess to resize images faster, you can take a look [here](https://www.kaggle.com/dimitreoliveira/cloud-segmentation-with-utility-scripts-and-keras) and the scripts are also [here](https://www.kaggle.com/dimitreoliveira/cloud-images-segmentation-utillity-script), maybe it can be even faster if you use multiprocess to transform images and masks to numpy.", "649410": "Thanks for sharing, this is something new for me.\nI will take a look.", "649412": "Yes, that could speed up resizing. But since its a one time process I don't this it should matter much." }, "source": "meta" }