{
  "id": 69399,
  "title": "Techniques for loading images faster?",
  "url": "/competitions/human-protein-atlas-image-classification/discussion/69399",
  "author_name": "",
  "post_date": "2018-10-23T14:43:47.671069300Z",
  "votes": 1,
  "comment_count": 10,
  "views": 0,
  "content": "<p>Hello there. </p>\n\n<p>I suspect I have a speed limitation regarding loading images (the batch generator seems to be taking longer than the model training). </p>\n\n<p>What strategies do you use to load images faster for training? And for Augmentations?</p>\n\n<p>So far I checked these on a Kaggle kernel:</p>\n\n<ul>\n<li>PIL is the fastest to load images (1.5 seconds a batch)\n<ul><li>compared to CV2 (1.75 seconds the same batch)    </li>\n<li>and imageio (1.6 seconds the same batch)    </li></ul></li>\n<li>numpy.flip and numpy.transpose don't impact the generator   </li>\n<li>Resize and Transforms with CV2 are very quick and nothing compared to the loading time    </li>\n<li>Light augmentations by power, multiplication, sum on the np arrays can add a significant time</li>\n</ul>\n\n<p>I'm loading images one by one and stacking channels in a numpy array. </p>",
  "messages": [
    {
      "id": "408854",
      "postDate": "10/23/2018 14:43:47",
      "content": "<p>Hello there. </p>\n\n<p>I suspect I have a speed limitation regarding loading images (the batch generator seems to be taking longer than the model training). </p>\n\n<p>What strategies do you use to load images faster for training? And for Augmentations?</p>\n\n<p>So far I checked these on a Kaggle kernel:</p>\n\n<ul>\n<li>PIL is the fastest to load images (1.5 seconds a batch)\n<ul><li>compared to CV2 (1.75 seconds the same batch)    </li>\n<li>and imageio (1.6 seconds the same batch)    </li></ul></li>\n<li>numpy.flip and numpy.transpose don't impact the generator   </li>\n<li>Resize and Transforms with CV2 are very quick and nothing compared to the loading time    </li>\n<li>Light augmentations by power, multiplication, sum on the np arrays can add a significant time</li>\n</ul>\n\n<p>I'm loading images one by one and stacking channels in a numpy array. </p>",
      "rawMarkdown": "Hello there. \n\nI suspect I have a speed limitation regarding loading images (the batch generator seems to be taking longer than the model training). \n\nWhat strategies do you use to load images faster for training? And for Augmentations?\n\nSo far I checked these on a Kaggle kernel:\n\n- PIL is the fastest to load images (1.5 seconds a batch)\n    - compared to CV2 (1.75 seconds the same batch)    \n    - and imageio (1.6 seconds the same batch)    \n- numpy.flip and numpy.transpose don't impact the generator   \n- Resize and Transforms with CV2 are very quick and nothing compared to the loading time    \n- Light augmentations by power, multiplication, sum on the np arrays can add a significant time\n\nI'm loading images one by one and stacking channels in a numpy array.",
      "votes": null
    },
    {
      "id": "408871",
      "postDate": "10/23/2018 15:01:30",
      "content": "<p>Combining the 4 images into 1 image with 4 channels helped</p>",
      "rawMarkdown": "Combining the 4 images into 1 image with 4 channels helped",
      "votes": null
    },
    {
      "id": "408875",
      "postDate": "10/23/2018 15:04:41",
      "content": "<p>If you have enough memory you can load them all into memory. 512x512 images take around 40Gb of memory, 256x256 around 10Gb. </p>",
      "rawMarkdown": "If you have enough memory you can load them all into memory. 512x512 images take around 40Gb of memory, 256x256 around 10Gb.",
      "votes": null
    },
    {
      "id": "408878",
      "postDate": "10/23/2018 15:07:22",
      "content": "<p>You mean save them as 4 channels? Good one :)</p>",
      "rawMarkdown": "You mean save them as 4 channels? Good one :)",
      "votes": null
    },
    {
      "id": "409083",
      "postDate": "10/23/2018 20:04:52",
      "content": "<p>I first used ImageMagick to combine all the images.  Then I use the Keras image generator and fit_generator with workers=24. With minimal augmentations it becomes limited  by the SSD at 250MB/sec. For more complicated models my system is limited by the GPU speed. If I add several augmentations like rotate, zoom, shift, then my system becomes CPU limited. For my lightweight models, I had to load the images into memory. My full train set includes 60k images and in memory I can get one epoch in ~30 seconds.</p>",
      "rawMarkdown": "I first used ImageMagick to combine all the images.  Then I use the Keras image generator and fit_generator with workers=24. With minimal augmentations it becomes limited  by the SSD at 250MB/sec. For more complicated models my system is limited by the GPU speed. If I add several augmentations like rotate, zoom, shift, then my system becomes CPU limited. For my lightweight models, I had to load the images into memory. My full train set includes 60k images and in memory I can get one epoch in ~30 seconds.",
      "votes": null
    },
    {
      "id": "411433",
      "postDate": "10/28/2018 06:09:58",
      "content": "<p>I combined them all using 4 separate ImageDataGenerators, and then saved them as .npy files with all 4 channels. Then for training I load the .npy files and apply any data augmentation</p>",
      "rawMarkdown": "I combined them all using 4 separate ImageDataGenerators, and then saved them as .npy files with all 4 channels. Then for training I load the .npy files and apply any data augmentation",
      "votes": null
    },
    {
      "id": "411435",
      "postDate": "10/28/2018 06:18:13",
      "content": "<p>How much memory do you have? I find myself limited by cpu if I use multiple augmentations </p>",
      "rawMarkdown": "How much memory do you have? I find myself limited by cpu if I use multiple augmentations",
      "votes": null
    },
    {
      "id": "411695",
      "postDate": "10/28/2018 20:08:17",
      "content": "<p><a href=\"https://www.kaggle.com/guglielmocamporese/lighter-dataset-in-rgby-format-for-fast-uploading\">Here</a> I write a kernel to save your dataset in RGBY images. With that you can load all the dataset into memory.</p>",
      "rawMarkdown": "[Here][1] I write a kernel to save your dataset in RGBY images. With that you can load all the dataset into memory.\n\n\n  [1]: https://www.kaggle.com/guglielmocamporese/lighter-dataset-in-rgby-format-for-fast-uploading",
      "votes": null
    },
    {
      "id": "411810",
      "postDate": "10/29/2018 04:07:24",
      "content": "<p>32GB</p>",
      "rawMarkdown": "32GB",
      "votes": null
    },
    {
      "id": "412609",
      "postDate": "10/30/2018 13:48:03",
      "content": "<p>Save as \".npy\" with dtype = \"uint8\". Before augmentation, convert back to float32 and normalize them with dataset mean/std.</p>",
      "rawMarkdown": "Save as \".npy\" with dtype = \"uint8\". Before augmentation, convert back to float32 and normalize them with dataset mean/std.",
      "votes": null
    },
    {
      "id": "412737",
      "postDate": "10/30/2018 17:41:17",
      "content": "<p>So far if I run with minimal augmentations I can train as fast as my SSD will go, sustained 240MB/sec. It can go faster but I have to be careful on memory with 64GB. Working with the 512x512 images.</p>",
      "rawMarkdown": "So far if I run with minimal augmentations I can train as fast as my SSD will go, sustained 240MB/sec. It can go faster but I have to be careful on memory with 64GB. Working with the 512x512 images.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 408871,
      "author_name": "christofhenkel",
      "author_url": "",
      "post_date": "10/23/2018 15:01:30",
      "content": "<p>Combining the 4 images into 1 image with 4 channels helped</p>",
      "votes": null,
      "replies": [
        {
          "id": 408878,
          "author_name": "danmoller",
          "author_url": "",
          "post_date": "10/23/2018 15:07:22",
          "content": "<p>You mean save them as 4 channels? Good one :)</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 408875,
      "author_name": "sakvaua",
      "author_url": "",
      "post_date": "10/23/2018 15:04:41",
      "content": "<p>If you have enough memory you can load them all into memory. 512x512 images take around 40Gb of memory, 256x256 around 10Gb. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 409083,
      "author_name": "ldm314",
      "author_url": "",
      "post_date": "10/23/2018 20:04:52",
      "content": "<p>I first used ImageMagick to combine all the images.  Then I use the Keras image generator and fit_generator with workers=24. With minimal augmentations it becomes limited  by the SSD at 250MB/sec. For more complicated models my system is limited by the GPU speed. If I add several augmentations like rotate, zoom, shift, then my system becomes CPU limited. For my lightweight models, I had to load the images into memory. My full train set includes 60k images and in memory I can get one epoch in ~30 seconds.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 411433,
      "author_name": "christopherberner",
      "author_url": "",
      "post_date": "10/28/2018 06:09:58",
      "content": "<p>I combined them all using 4 separate ImageDataGenerators, and then saved them as .npy files with all 4 channels. Then for training I load the .npy files and apply any data augmentation</p>",
      "votes": null,
      "replies": [
        {
          "id": 411435,
          "author_name": "ldm314",
          "author_url": "",
          "post_date": "10/28/2018 06:18:13",
          "content": "<p>How much memory do you have? I find myself limited by cpu if I use multiple augmentations </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 411810,
          "author_name": "christopherberner",
          "author_url": "",
          "post_date": "10/29/2018 04:07:24",
          "content": "<p>32GB</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 412737,
          "author_name": "ldm314",
          "author_url": "",
          "post_date": "10/30/2018 17:41:17",
          "content": "<p>So far if I run with minimal augmentations I can train as fast as my SSD will go, sustained 240MB/sec. It can go faster but I have to be careful on memory with 64GB. Working with the 512x512 images.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 411695,
      "author_name": "guglielmocamporese",
      "author_url": "",
      "post_date": "10/28/2018 20:08:17",
      "content": "<p><a href=\"https://www.kaggle.com/guglielmocamporese/lighter-dataset-in-rgby-format-for-fast-uploading\">Here</a> I write a kernel to save your dataset in RGBY images. With that you can load all the dataset into memory.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 412609,
      "author_name": "alexanderliao",
      "author_url": "",
      "post_date": "10/30/2018 13:48:03",
      "content": "<p>Save as \".npy\" with dtype = \"uint8\". Before augmentation, convert back to float32 and normalize them with dataset mean/std.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "408854": "Hello there. \n\nI suspect I have a speed limitation regarding loading images (the batch generator seems to be taking longer than the model training). \n\nWhat strategies do you use to load images faster for training? And for Augmentations?\n\nSo far I checked these on a Kaggle kernel:\n\n- PIL is the fastest to load images (1.5 seconds a batch)\n    - compared to CV2 (1.75 seconds the same batch)    \n    - and imageio (1.6 seconds the same batch)    \n- numpy.flip and numpy.transpose don't impact the generator   \n- Resize and Transforms with CV2 are very quick and nothing compared to the loading time    \n- Light augmentations by power, multiplication, sum on the np arrays can add a significant time\n\nI'm loading images one by one and stacking channels in a numpy array.",
    "408871": "Combining the 4 images into 1 image with 4 channels helped",
    "408875": "If you have enough memory you can load them all into memory. 512x512 images take around 40Gb of memory, 256x256 around 10Gb.",
    "408878": "You mean save them as 4 channels? Good one :)",
    "409083": "I first used ImageMagick to combine all the images.  Then I use the Keras image generator and fit_generator with workers=24. With minimal augmentations it becomes limited  by the SSD at 250MB/sec. For more complicated models my system is limited by the GPU speed. If I add several augmentations like rotate, zoom, shift, then my system becomes CPU limited. For my lightweight models, I had to load the images into memory. My full train set includes 60k images and in memory I can get one epoch in ~30 seconds.",
    "411433": "I combined them all using 4 separate ImageDataGenerators, and then saved them as .npy files with all 4 channels. Then for training I load the .npy files and apply any data augmentation",
    "411435": "How much memory do you have? I find myself limited by cpu if I use multiple augmentations",
    "411695": "[Here][1] I write a kernel to save your dataset in RGBY images. With that you can load all the dataset into memory.\n\n\n  [1]: https://www.kaggle.com/guglielmocamporese/lighter-dataset-in-rgby-format-for-fast-uploading",
    "411810": "32GB",
    "412609": "Save as \".npy\" with dtype = \"uint8\". Before augmentation, convert back to float32 and normalize them with dataset mean/std.",
    "412737": "So far if I run with minimal augmentations I can train as fast as my SSD will go, sustained 240MB/sec. It can go faster but I have to be careful on memory with 64GB. Working with the 512x512 images."
  },
  "source": "meta"
}