iterations in the line\n\nblackhat_struct = ndimage.iterate_structure(blackhat_struct, iterations)\n\nfrom 8 to 1, I could reduce this time to ~100ms and still produce masks like these:\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F915913%2F43bae87c5eed350e33d2b750baa35920%2FScreen%20Shot%202020-07-28%20at%2000.03.36.png?generation=1595905482560661&alt=media\")
Thanks to @carlossouza for pointing the issue of scalability.
\n\n\n\n\nIt takes 8-9 seconds to generate a single mask (for a single slice). Assuming that I resize all 3D images to 40 slices, it would take over 17 hours to generate all masks! Although this is doable for training (as it may be done only once, locally), it will become impractical in inference on Kaggle's servers. We don't even know the size of the test set, but assuming it is the same size, that would not be feasible.
\n
Has anyone got an idea how to fix this?
\n\n\n", "messages": [ { "id": "948502", "postDate": "07/28/2020 02:47:47", "content": "The mask generation that we're talking about is based on my Lung Segmentation kernel.
\n
Thanks to @carlossouza for pointing the issue of scalability.
\n\n\n\n\nIt takes 8-9 seconds to generate a single mask (for a single slice). Assuming that I resize all 3D images to 40 slices, it would take over 17 hours to generate all masks! Although this is doable for training (as it may be done only once, locally), it will become impractical in inference on Kaggle's servers. We don't even know the size of the test set, but assuming it is the same size, that would not be feasible.
\n
Has anyone got an idea how to fix this?
\n\n\n", "rawMarkdown": "Thanks to @carlossouza for pointing the issue of scalability. \n\n> It takes 8-9 seconds to generate a single mask (for a single slice). Assuming that I resize all 3D images to 40 slices, it would take over 17 hours to generate all masks! Although this is doable for training (as it may be done only once, locally), it will become impractical in inference on Kaggle's servers. We don't even know the size of the test set, but assuming it is the same size, that would not be feasible.\n\nHas anyone got an idea how to fix this?\n\n> The mask generation that we're talking about is based on my Lung Segmentation kernel.", "votes": null }, { "id": "948512", "postDate": "07/28/2020 03:05:36", "content": "The mask generation that we're talking about is based on my Lung Segmentation kernel.
\n
Hey @aadhavvignesh ! I think I actually found a compromise solution. In your code, changing the iterations in the line\n\nblackhat_struct = ndimage.iterate_structure(blackhat_struct, iterations)\n\nfrom 8 to 1, I could reduce this time to ~100ms and still produce masks like these:\n
It's not as good as the original code, but I could preprocess all 176 CT scans in less than 1h :)
", "rawMarkdown": "Hey @aadhavvignesh ! I think I actually found a compromise solution. In your code, changing the `iterations` in the line\n```\nblackhat_struct = ndimage.iterate_structure(blackhat_struct, iterations)\n```\nfrom 8 to 1, I could reduce this time to ~100ms and still produce masks like these:\n\n\nIt's not as good as the original code, but I could preprocess all 176 CT scans in less than 1h :)", "votes": null }, { "id": "948525", "postDate": "07/28/2020 03:25:13", "content": "@carlossouza Thanks for the info! I'll change some code in notebook to notify about this!
\n\n\n", "rawMarkdown": "carlossouza Thanks for the info! I'll change some code in notebook to notify about this!\n\n> Update: The notebook has been updated with comparisons of different amount of iterations.", "votes": null } ], "comments": [ { "id": 948512, "author_name": "carlossouza", "author_url": "", "post_date": "07/28/2020 03:05:36", "content": "Update: The notebook has been updated with comparisons of different amount of iterations.
\n
Hey @aadhavvignesh ! I think I actually found a compromise solution. In your code, changing the iterations in the line\n\nblackhat_struct = ndimage.iterate_structure(blackhat_struct, iterations)\n\nfrom 8 to 1, I could reduce this time to ~100ms and still produce masks like these:\n
It's not as good as the original code, but I could preprocess all 176 CT scans in less than 1h :)
", "votes": null, "replies": [ { "id": 948525, "author_name": "aadhavvignesh", "author_url": "", "post_date": "07/28/2020 03:25:13", "content": "@carlossouza Thanks for the info! I'll change some code in notebook to notify about this!
\n\n\n", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "948502": "Thanks to @carlossouza for pointing the issue of scalability. \n\n> It takes 8-9 seconds to generate a single mask (for a single slice). Assuming that I resize all 3D images to 40 slices, it would take over 17 hours to generate all masks! Although this is doable for training (as it may be done only once, locally), it will become impractical in inference on Kaggle's servers. We don't even know the size of the test set, but assuming it is the same size, that would not be feasible.\n\nHas anyone got an idea how to fix this?\n\n> The mask generation that we're talking about is based on my Lung Segmentation kernel.", "948512": "Hey @aadhavvignesh ! I think I actually found a compromise solution. In your code, changing the `iterations` in the line\n```\nblackhat_struct = ndimage.iterate_structure(blackhat_struct, iterations)\n```\nfrom 8 to 1, I could reduce this time to ~100ms and still produce masks like these:\n\n\nIt's not as good as the original code, but I could preprocess all 176 CT scans in less than 1h :)", "948525": "carlossouza Thanks for the info! I'll change some code in notebook to notify about this!\n\n> Update: The notebook has been updated with comparisons of different amount of iterations." }, "source": "meta" }Update: The notebook has been updated with comparisons of different amount of iterations.
\n