Hello,
\nI'm trying to participate in the competition, however I have a problem of each training epoch taking 3-4 hours of time on GPU.
\nI used python's profiler in lightning and found that most of my time is spent at the dataloader.
\nthrough more inspection, turns out that scipy.ndimage.zoom (function to resize 3D images) take 2+ seconds per image which was the reason my training was taking so long.
My question is, does each epoch take 3-4 hours for everybody ?
\nIf not what else can be used instead of scipy.ndimage.zoom ?
I try to make images (64, 256, 256). I'm thinking if image depth < 64 , pad it with all black images.
\nIf it is greater than 64, perform slabbing (i.e. divide the images into 64 groups and average each group together).
I plan on doing this tomorrow, but I'm asking in case somebody can point me to a better direction or confirm that scipy.ndimage.zoom is the way to go.
Thanks for reading
", "messages": [ { "id": "1534401", "postDate": "10/04/2021 21:12:59", "content": "Hello,
\nI'm trying to participate in the competition, however I have a problem of each training epoch taking 3-4 hours of time on GPU.
\nI used python's profiler in lightning and found that most of my time is spent at the dataloader.
\nthrough more inspection, turns out that scipy.ndimage.zoom (function to resize 3D images) take 2+ seconds per image which was the reason my training was taking so long.
My question is, does each epoch take 3-4 hours for everybody ?
\nIf not what else can be used instead of scipy.ndimage.zoom ?
I try to make images (64, 256, 256). I'm thinking if image depth < 64 , pad it with all black images.
\nIf it is greater than 64, perform slabbing (i.e. divide the images into 64 groups and average each group together).
I plan on doing this tomorrow, but I'm asking in case somebody can point me to a better direction or confirm that scipy.ndimage.zoom is the way to go.
Thanks for reading
", "rawMarkdown": "Hello,\n\nI'm trying to participate in the competition, however I have a problem of each training epoch taking 3-4 hours of time on GPU.\n\nI used python's profiler in lightning and found that most of my time is spent at the dataloader.\n\nthrough more inspection, turns out that `scipy.ndimage.zoom` (function to resize 3D images) take 2+ seconds per image which was the reason my training was taking so long.\n\nMy question is, does each epoch take 3-4 hours for everybody ?\n\nIf not what else can be used instead of `scipy.ndimage.zoom` ?\n\nI try to make images (64, 256, 256). I'm thinking if image depth < 64 , pad it with all black images.\nIf it is greater than 64, perform slabbing (i.e. divide the images into 64 groups and average each group together).\n\nI plan on doing this tomorrow, but I'm asking in case somebody can point me to a better direction or confirm that `scipy.ndimage.zoom` is the way to go.\n\nThanks for reading", "votes": null }, { "id": "1534515", "postDate": "10/05/2021 01:57:05", "content": "You can change scipy.ndimage.zoom's order. The default value is 3. This means 3 spline interpolation.
\nIf you apply order=0 (nearest-neighbor) or order=1 (linear interpolation), your training job might be fast.
You can save resized data as npy or pkl file and then load it for training.
", "rawMarkdown": "You can save resized data as npy or pkl file and then load it for training.", "votes": null }, { "id": "1534566", "postDate": "10/05/2021 03:52:38", "content": "For tf.keras users, to resize the volume of the 3D image, you can simply use layers.Resizing, which works on single or batch of 3D images (bs, h, w, depth). Note, you can use resizing layer in the custom data loader (Sequence) or tf.data API or more precisely during model construction which would take GPU acceleration for resizing during training which should be fast enough.
You can also use open cv's resize, which is more optimal than scipy zoom, although it only operates on 2d images / matrices.
", "rawMarkdown": "You can also use open cv's resize, which is more optimal than scipy zoom, although it only operates on 2d images / matrices.", "votes": null } ], "comments": [ { "id": 1534515, "author_name": "atsunorifujita", "author_url": "", "post_date": "10/05/2021 01:57:05", "content": "You can change scipy.ndimage.zoom's order. The default value is 3. This means 3 spline interpolation.
\nIf you apply order=0 (nearest-neighbor) or order=1 (linear interpolation), your training job might be fast.
You can save resized data as npy or pkl file and then load it for training.
", "votes": null, "replies": [] }, { "id": 1534566, "author_name": "ipythonx", "author_url": "", "post_date": "10/05/2021 03:52:38", "content": "For tf.keras users, to resize the volume of the 3D image, you can simply use layers.Resizing, which works on single or batch of 3D images (bs, h, w, depth). Note, you can use resizing layer in the custom data loader (Sequence) or tf.data API or more precisely during model construction which would take GPU acceleration for resizing during training which should be fast enough.
You can also use open cv's resize, which is more optimal than scipy zoom, although it only operates on 2d images / matrices.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1534401": "Hello,\n\nI'm trying to participate in the competition, however I have a problem of each training epoch taking 3-4 hours of time on GPU.\n\nI used python's profiler in lightning and found that most of my time is spent at the dataloader.\n\nthrough more inspection, turns out that `scipy.ndimage.zoom` (function to resize 3D images) take 2+ seconds per image which was the reason my training was taking so long.\n\nMy question is, does each epoch take 3-4 hours for everybody ?\n\nIf not what else can be used instead of `scipy.ndimage.zoom` ?\n\nI try to make images (64, 256, 256). I'm thinking if image depth < 64 , pad it with all black images.\nIf it is greater than 64, perform slabbing (i.e. divide the images into 64 groups and average each group together).\n\nI plan on doing this tomorrow, but I'm asking in case somebody can point me to a better direction or confirm that `scipy.ndimage.zoom` is the way to go.\n\nThanks for reading", "1534515": "You can change `scipy.ndimage.zoom`'s order. The default value is 3. This means 3 spline interpolation.\nIf you apply order=0 (nearest-neighbor) or order=1 (linear interpolation), your training job might be fast.", "1534561": "You can save resized data as npy or pkl file and then load it for training.", "1534566": "For `tf.keras` users, to resize the volume of the 3D image, you can simply use [**`layers.Resizing`**](https://keras.io/api/layers/preprocessing_layers/image_preprocessing/resizing/), which works on single or batch of 3D images (`bs, h, w, depth`). Note, you can use resizing layer in the custom data loader (`Sequence`) or `tf.data` API or more precisely during model construction which would take GPU acceleration for resizing during training which should be fast enough.", "1535222": "You can also use open cv's resize, which is more optimal than scipy zoom, although it only operates on 2d images / matrices." }, "source": "meta" }