![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F82f420f454bed7b7e03282ad4fa7d0b5%2FPeek%202023-11-27%2008-04.gif?generation=1701043897784706&alt=media\")
this competition is about 3d vasculature segmentation.
\nif you read the data description page, and dataset paper carefully, are not detecting \"just any vessel\".
\nour target is \"vascular tree\".
This means that all our vessels are connected (like a tree : from trunk to branch to twigs). hence there actually a few 3d connected component objects.
\nyou can check this with 3d visualisation software
\nto avoid OOM, your solution need to:
\n1) either remove false positive
\n2) limit the number of 3d connected component objects
in fact to score well, you need to think of how to grow/flood-fill your tree to detect the smallest vessel
\n( the % smallest vessel will determine how much these contrubute to toal score)
this competition is about 3d vasculature segmentation.
\nif you read the data description page, and dataset paper carefully, are not detecting \"just any vessel\".
\nour target is \"vascular tree\".
This means that all our vessels are connected (like a tree : from trunk to branch to twigs). hence there actually a few 3d connected component objects.
\nyou can check this with 3d visualisation software
\nto avoid OOM, your solution need to:
\n1) either remove false positive
\n2) limit the number of 3d connected component objects
in fact to score well, you need to think of how to grow/flood-fill your tree to detect the smallest vessel
\n( the % smallest vessel will determine how much these contrubute to toal score)
i cannot get pyvisata to work in kaggle notebbok, but a 3d visualisation code cann be:
\n pyvista pv\n\n == : file = [f i range(, +)] \n mask=[]\n i,f enumerate(file):\n print(,i, end=)\n v = cv2.imread(f,cv2.IMREAD_GRAYSCALE)\n mask.append(v) \n mask = np.stack(mask)\n print()\n mask = mask/\n\n :\n pl = pv.Plotter()\n = np.stack(np.(mask > )).T\n pd = pv.PolyData()\n mesh = pd.glyph(geom=pv.())\n pl.add_mesh(mesh, show_scalar_bar=)\n pl.()\n
blue: ground truth
\nred : predicted (threshold at 0.10)
so many false positives !!!!
\nHINT: you can use 3d max pooling for 3d connect component analysis
\n ... d voxel\nnon_zero (>)\n\n ... set random seed ...\n\nfor iter in range()\n F.max_pool_d ( kernel_size) \n *non_zero\n\n# get d componets\n.. torch.unique( ...) #use return_count get area\nreference:
\nwe cannot be sure that the images are in a ordered stack that would yield a 3D \"Vascular Tree\", for the test set. All we know for sure is that it is about 1500 images, and that it may be more than one kidney or even more than one resolution. I want to include methodologies that would use the 3D component but it is hard to include this when i dont know the structure of the test set.
", "rawMarkdown": "we cannot be sure that the images are in a ordered stack that would yield a 3D \"Vascular Tree\", for the test set. All we know for sure is that it is about 1500 images, and that it may be more than one kidney or even more than one resolution. I want to include methodologies that would use the 3D component but it is hard to include this when i dont know the structure of the test set.", "votes": null }, { "id": "2539412", "postDate": "11/27/2023 04:04:45", "content": "same model different resolution leads to OOM error example model trained on seresnext(resolution 10241024 provides a score in LB) , (1303912 provides OOM error) @hengck23 do you know why?
", "rawMarkdown": "same model different resolution leads to OOM error example model trained on seresnext(resolution 1024*1024 provides a score in LB) , (1303*912 provides OOM error) @hengck23 do you know why?", "votes": null }, { "id": "2539415", "postDate": "11/27/2023 04:15:20", "content": "you should test locally to ensure:
\nit is important train, test scale are the same
\nthere are two type of vein: artery (oxygen in) and vein (oxygen out).
\nour target in this competition is artery.
if you consider single scale, these two are different in apperance.
\nif you consider multiscale, they may become indistinguishable (lead to may false positive).
these are my conjecture/thoughts only. please verify by experiments
", "rawMarkdown": "you should test locally to ensure: \n1. does you model work for multi scale? (e.g. 50 um, 40 um, 30 um ...) \n\nit is important train, test scale are the same\n\n---\n\nthere are two type of vein: artery (oxygen in) and vein (oxygen out). \nour target in this competition is artery. \n\nif you consider single scale, these two are different in apperance. \nif you consider multiscale, they may become indistinguishable (lead to may false positive). \n\nthese are my conjecture/thoughts only. please verify by experiments", "votes": null }, { "id": "2539418", "postDate": "11/27/2023 04:19:15", "content": "check this
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F1ee3f20c41b7e35611195d1d16775256%2FSelection_999(4064).png?generation=1701058754075775&alt=media\")
thank you that is very helpful!
", "rawMarkdown": "thank you that is very helpful!", "votes": null }, { "id": "2541539", "postDate": "11/28/2023 15:11:42", "content": "Heng, how do you plan to check the \"multiscale\" inference? I don't see how we can do that from purely 50um images. No matter how you scale it up you won't get higher resolution. You can get to a lower resolution (to 50um) from a higher one (10 um). But not the other way around.
\nThanks!
\"how do you plan to check the \"multiscale\" inference?\"
\nwe have 50 um images. actually the hip-ct website dataset have 20 um images.
\nyou can actually download and resize images of 20 um images and see if i can represent 50 um images
we can then predict this:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fe3acdee196459f2c249f8a9a76241e97%2FSelection_999(4103).png?generation=1701186337638325&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F561a17f928d1c3489b844093a6599003%2FSelection_999(4104).png?generation=1701186511085167&alt=media\")
matching kaggle image and the hip-ct website dataset
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F6dc1c4a40a83b1b7b67827df9aa5db16%2FSelection_999(4106).png?generation=1701187013655311&alt=media\")
HINT: use higer resolution (25 um) to create more 50um train data (e.g. 3d rotation)
", "rawMarkdown": "matching kaggle image and the hip-ct website dataset\n\n.png?generation=1701187013655311&alt=media)\n\nHINT: use higer resolution (25 um) to create more 50um train data (e.g. 3d rotation)", "votes": null }, { "id": "2542072", "postDate": "11/29/2023 03:50:17", "content": "understanding the data
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F15277aba1ea62c62aeacc577e830108e%2FSelection_999(4111).png?generation=1701229815720130&alt=media\")
Does that mean the test data will be a stack of images in order, ie one kidney?
", "rawMarkdown": "Does that mean the test data will be a stack of images in order, ie one kidney?", "votes": null }, { "id": "2546468", "postDate": "12/02/2023 13:23:57", "content": "good job！Hope your amazing work,it help me a lot
", "rawMarkdown": "good job！Hope your amazing work,it help me a lot", "votes": null }, { "id": "2548421", "postDate": "12/04/2023 12:11:20", "content": "so does it mean the data(slices) in the test set would be ordered in vertical order?
", "rawMarkdown": "so does it mean the data(slices) in the test set would be ordered in vertical order?", "votes": null }, { "id": "2548667", "postDate": "12/04/2023 15:38:29", "content": "@hengck23 can you please share the link to those images?
", "rawMarkdown": "hengck23 can you please share the link to those images?", "votes": null }, { "id": "2550604", "postDate": "12/06/2023 06:17:17", "content": "Most likely, this is the link - https://human-organ-atlas.esrf.eu/datasets/572182201.
", "rawMarkdown": "Most likely, this is the link - https://human-organ-atlas.esrf.eu/datasets/572182201.", "votes": null }, { "id": "2589856", "postDate": "01/06/2024 17:38:52", "content": "Hello @hengck23,
\nMay I ask what 3d visualisation software you are using ? Thanks in advance
i cannot get pyvisata to work in kaggle notebbok, but a 3d visualisation code cann be:
\n pyvista pv\n\n == : file = [f i range(, +)] \n mask=[]\n i,f enumerate(file):\n print(,i, end=)\n v = cv2.imread(f,cv2.IMREAD_GRAYSCALE)\n mask.append(v) \n mask = np.stack(mask)\n print()\n mask = mask/\n\n :\n pl = pv.Plotter()\n = np.stack(np.(mask > )).T\n pd = pv.PolyData()\n mesh = pd.glyph(geom=pv.())\n pl.add_mesh(mesh, show_scalar_bar=)\n pl.()\n
blue: ground truth
\nred : predicted (threshold at 0.10)
so many false positives !!!!
\nHINT: you can use 3d max pooling for 3d connect component analysis
\n ... d voxel\nnon_zero (>)\n\n ... set random seed ...\n\nfor iter in range()\n F.max_pool_d ( kernel_size) \n *non_zero\n\n# get d componets\n.. torch.unique( ...) #use return_count get area\nreference:
\nHello @hengck23,
\nMay I ask what 3d visualisation software you are using ? Thanks in advance
we cannot be sure that the images are in a ordered stack that would yield a 3D \"Vascular Tree\", for the test set. All we know for sure is that it is about 1500 images, and that it may be more than one kidney or even more than one resolution. I want to include methodologies that would use the 3D component but it is hard to include this when i dont know the structure of the test set.
", "votes": null, "replies": [ { "id": 2539418, "author_name": "hengck23", "author_url": "", "post_date": "11/27/2023 04:19:15", "content": "check this
\n
thank you that is very helpful!
", "votes": null, "replies": [] }, { "id": 2548421, "author_name": "bhavesjain", "author_url": "", "post_date": "12/04/2023 12:11:20", "content": "so does it mean the data(slices) in the test set would be ordered in vertical order?
", "votes": null, "replies": [] } ] }, { "id": 2545503, "author_name": "simonegilbert", "author_url": "", "post_date": "12/01/2023 14:26:40", "content": "Does that mean the test data will be a stack of images in order, ie one kidney?
", "votes": null, "replies": [] } ] }, { "id": 2539412, "author_name": "arunodhayan", "author_url": "", "post_date": "11/27/2023 04:04:45", "content": "same model different resolution leads to OOM error example model trained on seresnext(resolution 10241024 provides a score in LB) , (1303912 provides OOM error) @hengck23 do you know why?
", "votes": null, "replies": [ { "id": 2539415, "author_name": "hengck23", "author_url": "", "post_date": "11/27/2023 04:15:20", "content": "you should test locally to ensure:
\nit is important train, test scale are the same
\nthere are two type of vein: artery (oxygen in) and vein (oxygen out).
\nour target in this competition is artery.
if you consider single scale, these two are different in apperance.
\nif you consider multiscale, they may become indistinguishable (lead to may false positive).
these are my conjecture/thoughts only. please verify by experiments
", "votes": null, "replies": [ { "id": 2541539, "author_name": "sakvaua", "author_url": "", "post_date": "11/28/2023 15:11:42", "content": "Heng, how do you plan to check the \"multiscale\" inference? I don't see how we can do that from purely 50um images. No matter how you scale it up you won't get higher resolution. You can get to a lower resolution (to 50um) from a higher one (10 um). But not the other way around.
\nThanks!
\"how do you plan to check the \"multiscale\" inference?\"
\nwe have 50 um images. actually the hip-ct website dataset have 20 um images.
\nyou can actually download and resize images of 20 um images and see if i can represent 50 um images
we can then predict this:
\n
matching kaggle image and the hip-ct website dataset
\n
HINT: use higer resolution (25 um) to create more 50um train data (e.g. 3d rotation)
", "votes": null, "replies": [ { "id": 2542072, "author_name": "hengck23", "author_url": "", "post_date": "11/29/2023 03:50:17", "content": "understanding the data
\n
@hengck23 can you please share the link to those images?
", "votes": null, "replies": [ { "id": 2550604, "author_name": "phanisrikanth", "author_url": "", "post_date": "12/06/2023 06:17:17", "content": "Most likely, this is the link - https://human-organ-atlas.esrf.eu/datasets/572182201.
", "votes": null, "replies": [] } ] } ] } ] } ] } ] }, { "id": 2546468, "author_name": "dingzhen111", "author_url": "", "post_date": "12/02/2023 13:23:57", "content": "good job！Hope your amazing work,it help me a lot
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2538610": "this competition is about 3d vasculature segmentation. \nif you read the data description page, and dataset paper carefully, are not detecting \"just any vessel\". \nour target is \"vascular tree\". \n \n\nThis means that all our vessels are connected (like a tree : from trunk to branch to twigs). hence there actually a few 3d connected component objects.\n\nyou can check this with 3d visualisation software\n\n---\n\nto avoid OOM, your solution need to:\n1) either remove false positive\n2) limit the number of 3d connected component objects \n\n---\n\nin fact to score well, you need to think of how to grow/flood-fill your tree to detect the smallest vessel\n( the % smallest vessel will determine how much these contrubute to toal score)", "2538616": "i cannot get pyvisata to work in kaggle notebbok, but a 3d visualisation code cann be:\n\n\n```\nimport pyvista as pv\n\n\tif name == 'kidney_3_dense': file = [f'{data_dir}/train/{name}/labels/{i:04d}.tif' for i in range(496, 496+500)] \n\tmask=[]\n\tfor i,f in enumerate(file):\n\t\tprint('\\r',i, end='')\n\t\tv = cv2.imread(f,cv2.IMREAD_GRAYSCALE)\n\t\tmask.append(v) \n\tmask = np.stack(mask)\n\tprint('')\n\tmask = mask/255\n \n\tif 1:\n\t\tpl = pv.Plotter()\n\t\tpoint = np.stack(np.where(mask > 0)).T\n\t\tpd = pv.PolyData(point)\n\t\tmesh = pd.glyph(geom=pv.Cube())\n\t\tpl.add_mesh(mesh, show_scalar_bar=True)\n\t\tpl.show()\n\n```", "2539292": "\n\nblue: ground truth\nred : predicted (threshold at 0.10)\n\nso many false positives !!!!\n\nHINT: you can use 3d max pooling for 3d connect component analysis\n\n```\nx = ... 3d voxel\nnon_zero = (x>0)\n\nx = ... set random seed ...\n\nfor iter in range(10000)\n x = F.max_pool_3d ( x, kernel_size=3) \n x = x*non_zero\n\n#to get 3d componets\n.. = torch.unique(x, ...) #use return_count = true to get area\n\n```\n\nreference:\n- https://github.com/prittt/YACCLAB/issues/28\n- https://kornia.readthedocs.io/en/latest/_modules/kornia/contrib/connected_components.html#connected_components", "2539406": "we cannot be sure that the images are in a ordered stack that would yield a 3D \"Vascular Tree\", for the test set. All we know for sure is that it is about 1500 images, and that it may be more than one kidney or even more than one resolution. I want to include methodologies that would use the 3D component but it is hard to include this when i dont know the structure of the test set.", "2539412": "same model different resolution leads to OOM error example model trained on seresnext(resolution 1024*1024 provides a score in LB) , (1303*912 provides OOM error) @hengck23 do you know why?", "2539415": "you should test locally to ensure: \n1. does you model work for multi scale? (e.g. 50 um, 40 um, 30 um ...) \n\nit is important train, test scale are the same\n\n---\n\nthere are two type of vein: artery (oxygen in) and vein (oxygen out). \nour target in this competition is artery. \n\nif you consider single scale, these two are different in apperance. \nif you consider multiscale, they may become indistinguishable (lead to may false positive). \n\nthese are my conjecture/thoughts only. please verify by experiments", "2539418": "check this\n.png?generation=1701058754075775&alt=media)", "2539471": "thank you that is very helpful!", "2541539": "Heng, how do you plan to check the \"multiscale\" inference? I don't see how we can do that from purely 50um images. No matter how you scale it up you won't get higher resolution. You can get to a lower resolution (to 50um) from a higher one (10 um). But not the other way around.\nThanks!", "2541573": "\"how do you plan to check the \"multiscale\" inference?\"\n\nwe have 50 um images. actually the hip-ct website dataset have 20 um images.\nyou can actually download and resize images of 20 um images and see if i can represent 50 um images\n\nwe can then predict this:\n.png?generation=1701186337638325&alt=media)\n\n.png?generation=1701186511085167&alt=media)", "2541591": "matching kaggle image and the hip-ct website dataset\n\n.png?generation=1701187013655311&alt=media)\n\nHINT: use higer resolution (25 um) to create more 50um train data (e.g. 3d rotation)", "2542072": "understanding the data\n\n.png?generation=1701229815720130&alt=media)", "2545503": "Does that mean the test data will be a stack of images in order, ie one kidney?", "2546468": "good job！Hope your amazing work,it help me a lot", "2548421": "so does it mean the data(slices) in the test set would be ordered in vertical order?", "2548667": "hengck23 can you please share the link to those images?", "2550604": "Most likely, this is the link - https://human-organ-atlas.esrf.eu/datasets/572182201.", "2589856": "Hello @hengck23,\nMay I ask what 3d visualisation software you are using ? Thanks in advance" }, "source": "meta" }