If you are doing deep learning like UNet etc. and your training feels very slow, the bottleneck is almost certainly I/O, not your model. TIFF loading in Python is extremely slow — especially for 3D volumes — and it can easily dominate your epoch time.
\nI benchmarked several 3D image data storage formats on the Vesuvius dataset (tiff, npy, npz, zarr), measuring average load time and file size. npy is a well known format as part of numpy. zarr was used as the data in the CZII-CryoET challenge, .. maybe in other too. npz is an unstructured and compressed version of npy.
npy is the best in terms of speed. Gives about ~25x speedup in image loading times in isolation and decreased my overall training time by about ~10x.
For labels, plain npy wastes a ton of disk space (tif labels are < 1MB whereas npy labels are ~32 MB), as the actual label information does not need us to store the whole array. I used RLE (Run-Length Encoding) to compress, it's not that fast as npy but still 2-3x better than loading from the tif label files.
This notebook does the npy and rle conversion. You can directly use the output of the notebook as a replacement of the competition data. However note, I downsampled the data from 320 to 256 pixels for some extra speed and space saving as I didn't feel there's much of a difference.
If you feel the need to have the original 320px resolution, or want to save the labels in npy, it's pretty straightforward to do so too.
Happy training!
", "messages": [ { "id": 3332707, "postDate": "2025-11-16T20:39:23.453Z", "content": "If you are doing deep learning like UNet etc. and your training feels very slow, the bottleneck is almost certainly I/O, not your model. TIFF loading in Python is extremely slow — especially for 3D volumes — and it can easily dominate your epoch time.
\nI benchmarked several 3D image data storage formats on the Vesuvius dataset (tiff, npy, npz, zarr), measuring average load time and file size. npy is a well known format as part of numpy. zarr was used as the data in the CZII-CryoET challenge, .. maybe in other too. npz is an unstructured and compressed version of npy.
npy is the best in terms of speed. Gives about ~25x speedup in image loading times in isolation and decreased my overall training time by about ~10x.
For labels, plain npy wastes a ton of disk space (tif labels are < 1MB whereas npy labels are ~32 MB), as the actual label information does not need us to store the whole array. I used RLE (Run-Length Encoding) to compress, it's not that fast as npy but still 2-3x better than loading from the tif label files.
This notebook does the npy and rle conversion. You can directly use the output of the notebook as a replacement of the competition data. However note, I downsampled the data from 320 to 256 pixels for some extra speed and space saving as I didn't feel there's much of a difference.
If you feel the need to have the original 320px resolution, or want to save the labels in npy, it's pretty straightforward to do so too.
Happy training!
", "rawMarkdown": "If you are doing deep learning like UNet etc. and your training feels very slow, the bottleneck is almost certainly I/O, not your model. TIFF loading in Python is extremely slow — especially for 3D volumes — and it can easily dominate your epoch time.\n\nI benchmarked several 3D image data storage formats on the Vesuvius dataset (`tiff`, `npy`, `npz`, `zarr`), measuring average load time and file size. `npy` is a well known format as part of numpy. `zarr` was used as the data in the [CZII-CryoET](https://www.kaggle.com/competitions/czii-cryo-et-object-identification) challenge, .. maybe in other too. `npz` is an unstructured and compressed version of `npy`.\n\n`npy` is the best in terms of speed. Gives about ~25x speedup in image loading times in isolation and decreased my overall training time by about ~10x.\n\nFor labels, plain `npy` wastes a ton of disk space (`tif` labels are < 1MB whereas `npy` labels are ~32 MB), as the actual label information does not need us to store the whole array. I used RLE (Run-Length Encoding) to compress, it's not that fast as `npy` but still 2-3x better than loading from the `tif` label files.\n\nThis [notebook](https://www.kaggle.com/code/mayukh18/speed-up-data-load-npy-and-rle-data) does the `npy` and `rle` conversion. You can directly use the output of the notebook as a replacement of the competition data. However note, I downsampled the data from 320 to 256 pixels for some extra speed and space saving as I didn't feel there's much of a difference.\n\nIf you feel the need to have the original 320px resolution, or want to save the labels in `npy`, it's pretty straightforward to do so too.\n\nHappy training!", "votes": 17 }, { "id": 3401577, "postDate": "2026-02-04T03:50:08.423Z", "content": "has anyone tried this conversion method during inference? if so did it improve your runtime? I am working on increasing TTA and postprocessing steps but my notebook timed out during scoring (10+ hrs), I am wondering if doing this tiff conversion can help during inference. TIA.
", "rawMarkdown": "has anyone tried this conversion method during inference? if so did it improve your runtime? I am working on increasing TTA and postprocessing steps but my notebook timed out during scoring (10+ hrs), I am wondering if doing this tiff conversion can help during inference. TIA.", "votes": 1 }, { "id": 3336061, "postDate": "2025-11-18T10:06:54.610Z", "content": "If you do cropping, load your npy arrays as memmap (numpy.load(file, mmap_mode='r')). This is even more efficient.
Yes this can be much faster, even uncompressing the original tifs (and using tifffile to load instead of say, opencv) can lead to pretty large speed gains. For the most part, i do most of my training using Zarr, but for the purposes of the competition i wanted more generalized data formats just for ease of use.
", "rawMarkdown": "Yes this can be much faster, even uncompressing the original tifs (and using tifffile to load instead of say, opencv) can lead to pretty large speed gains. For the most part, i do most of my training using Zarr, but for the purposes of the competition i wanted more generalized data formats just for ease of use. ", "votes": 1 }, { "id": 3341367, "postDate": "2025-11-20T06:09:21.237Z", "content": "Another bottleneck could be augmentation, in particular if you try to do 3D segmentation, so doing it on GPU instead of CPU is also almost 10x boost, see https://www.kaggle.com/code/jirkaborovec/surface-detect-segment-3d-gpu-augmentation
", "rawMarkdown": "Another bottleneck could be augmentation, in particular if you try to do 3D segmentation, so doing it on GPU instead of CPU is also almost 10x boost, see https://www.kaggle.com/code/jirkaborovec/surface-detect-segment-3d-gpu-augmentation", "votes": 1 }, { "id": 3341371, "postDate": "2025-11-20T06:11:49.307Z", "content": "seems someone did this conversion, and it seems that the converted dataset is about 2.5x larger\nhttps://www.kaggle.com/competitions/vesuvius-challenge-surface-detection/discussion/630100
", "rawMarkdown": "seems someone did this conversion, and it seems that the converted dataset is about 2.5x larger\nhttps://www.kaggle.com/competitions/vesuvius-challenge-surface-detection/discussion/630100", "replies": [ { "id": 3341424, "postDate": "2025-11-20T06:54:17.990Z", "content": "That's not unexpected. The provided tiff files are compressed. .npy arrays are not. If you use compressed numpy arrays you save space again but then loading is slow again (+memmap won't work anymore). If you want the best of both worlds you need blosc2 (or hdf5, but I have not experimented with that extensively), but blosc2 is not easy to get a hang of
", "rawMarkdown": "That's not unexpected. The provided tiff files are compressed. .npy arrays are not. If you use compressed numpy arrays you save space again but then loading is slow again (+memmap won't work anymore). If you want the best of both worlds you need blosc2 (or hdf5, but I have not experimented with that extensively), but blosc2 is not easy to get a hang of", "votes": 1, "replies": [ { "id": 3341504, "postDate": "2025-11-20T07:58:01.643Z", "content": "Sure, just observation, not a complain… :) Have you also benchmarked npz?
", "rawMarkdown": "Sure, just observation, not a complain... :) Have you also benchmarked npz?" }, { "id": 3341512, "postDate": "2025-11-20T08:06:55.853Z", "content": "I have not done anything for this challenge yet, but I have benchmarked a variety of file formats for precisely this type of application in the past. The issue with npz is that it will force you uncompress and load an entire 3D image before you can crop. This is quite wasteful unless you train with a patch size that is roughly the same as the image shape. So ideally you want a file format that allows partial reading. Loading npy as memmap and blosc2 allow you to do that. npy with mmap_mode='r' is probably the easiest to get into but it requires more storage + I/O because of the lack of compression. This is why we switched to blosc2 in nnU-Net
", "rawMarkdown": "I have not done anything for this challenge yet, but I have benchmarked a variety of file formats for precisely this type of application in the past. The issue with npz is that it will force you uncompress and load an entire 3D image before you can crop. This is quite wasteful unless you train with a patch size that is roughly the same as the image shape. So ideally you want a file format that allows partial reading. Loading npy as memmap and blosc2 allow you to do that. npy with mmap_mode='r' is probably the easiest to get into but it requires more storage + I/O because of the lack of compression. This is why we switched to blosc2 in nnU-Net" } ] } ] }, { "id": 3335997, "postDate": "2025-11-18T09:22:18.830Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 3401577, "author_name": "kawaii", "author_url": "", "post_date": "2026-02-04T03:50:08.423000", "content": "has anyone tried this conversion method during inference? if so did it improve your runtime? I am working on increasing TTA and postprocessing steps but my notebook timed out during scoring (10+ hrs), I am wondering if doing this tiff conversion can help during inference. TIA.
", "votes": 1, "replies": [] }, { "id": 3336061, "author_name": "FabianIsensee", "author_url": "", "post_date": "2025-11-18T10:06:54.610000", "content": "If you do cropping, load your npy arrays as memmap (numpy.load(file, mmap_mode='r')). This is even more efficient.
Yes this can be much faster, even uncompressing the original tifs (and using tifffile to load instead of say, opencv) can lead to pretty large speed gains. For the most part, i do most of my training using Zarr, but for the purposes of the competition i wanted more generalized data formats just for ease of use.
", "votes": 1, "replies": [] }, { "id": 3341367, "author_name": "Jirka", "author_url": "", "post_date": "2025-11-20T06:09:21.237000", "content": "Another bottleneck could be augmentation, in particular if you try to do 3D segmentation, so doing it on GPU instead of CPU is also almost 10x boost, see https://www.kaggle.com/code/jirkaborovec/surface-detect-segment-3d-gpu-augmentation
", "votes": 1, "replies": [] }, { "id": 3341371, "author_name": "Jirka", "author_url": "", "post_date": "2025-11-20T06:11:49.307000", "content": "seems someone did this conversion, and it seems that the converted dataset is about 2.5x larger\nhttps://www.kaggle.com/competitions/vesuvius-challenge-surface-detection/discussion/630100
", "votes": 0, "replies": [ { "id": 3341424, "author_name": "FabianIsensee", "author_url": "", "post_date": "2025-11-20T06:54:17.990000", "content": "That's not unexpected. The provided tiff files are compressed. .npy arrays are not. If you use compressed numpy arrays you save space again but then loading is slow again (+memmap won't work anymore). If you want the best of both worlds you need blosc2 (or hdf5, but I have not experimented with that extensively), but blosc2 is not easy to get a hang of
", "votes": 1, "replies": [ { "id": 3341504, "author_name": "Jirka", "author_url": "", "post_date": "2025-11-20T07:58:01.643000", "content": "Sure, just observation, not a complain… :) Have you also benchmarked npz?
", "votes": 0, "replies": [] }, { "id": 3341512, "author_name": "FabianIsensee", "author_url": "", "post_date": "2025-11-20T08:06:55.853000", "content": "I have not done anything for this challenge yet, but I have benchmarked a variety of file formats for precisely this type of application in the past. The issue with npz is that it will force you uncompress and load an entire 3D image before you can crop. This is quite wasteful unless you train with a patch size that is roughly the same as the image shape. So ideally you want a file format that allows partial reading. Loading npy as memmap and blosc2 allow you to do that. npy with mmap_mode='r' is probably the easiest to get into but it requires more storage + I/O because of the lack of compression. This is why we switched to blosc2 in nnU-Net
", "votes": 0, "replies": [] } ] } ] }, { "id": 3335997, "author_name": "", "author_url": "", "post_date": "2025-11-18T09:22:18.830000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "3332707": "If you are doing deep learning like UNet etc. and your training feels very slow, the bottleneck is almost certainly I/O, not your model. TIFF loading in Python is extremely slow — especially for 3D volumes — and it can easily dominate your epoch time.\n\nI benchmarked several 3D image data storage formats on the Vesuvius dataset (`tiff`, `npy`, `npz`, `zarr`), measuring average load time and file size. `npy` is a well known format as part of numpy. `zarr` was used as the data in the [CZII-CryoET](https://www.kaggle.com/competitions/czii-cryo-et-object-identification) challenge, .. maybe in other too. `npz` is an unstructured and compressed version of `npy`.\n\n`npy` is the best in terms of speed. Gives about ~25x speedup in image loading times in isolation and decreased my overall training time by about ~10x.\n\nFor labels, plain `npy` wastes a ton of disk space (`tif` labels are < 1MB whereas `npy` labels are ~32 MB), as the actual label information does not need us to store the whole array. I used RLE (Run-Length Encoding) to compress, it's not that fast as `npy` but still 2-3x better than loading from the `tif` label files.\n\nThis [notebook](https://www.kaggle.com/code/mayukh18/speed-up-data-load-npy-and-rle-data) does the `npy` and `rle` conversion. You can directly use the output of the notebook as a replacement of the competition data. However note, I downsampled the data from 320 to 256 pixels for some extra speed and space saving as I didn't feel there's much of a difference.\n\nIf you feel the need to have the original 320px resolution, or want to save the labels in `npy`, it's pretty straightforward to do so too.\n\nHappy training!", "3401577": "has anyone tried this conversion method during inference? if so did it improve your runtime? I am working on increasing TTA and postprocessing steps but my notebook timed out during scoring (10+ hrs), I am wondering if doing this tiff conversion can help during inference. TIA.", "3336061": "If you do cropping, load your npy arrays as memmap (`numpy.load(file, mmap_mode='r')`). This is even more efficient.", "3341890": "Yes this can be much faster, even uncompressing the original tifs (and using tifffile to load instead of say, opencv) can lead to pretty large speed gains. For the most part, i do most of my training using Zarr, but for the purposes of the competition i wanted more generalized data formats just for ease of use. ", "3341367": "Another bottleneck could be augmentation, in particular if you try to do 3D segmentation, so doing it on GPU instead of CPU is also almost 10x boost, see https://www.kaggle.com/code/jirkaborovec/surface-detect-segment-3d-gpu-augmentation", "3341371": "seems someone did this conversion, and it seems that the converted dataset is about 2.5x larger\nhttps://www.kaggle.com/competitions/vesuvius-challenge-surface-detection/discussion/630100", "3335997": "" } }