\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fbb45e4d2e7577cc28d56107a3be9ea5f%2FSelection_721.png?generation=1732277368555041&alt=media\")
How large is the test dataset? My submission script takes 15 minutes on the \"visible\" test dataset with three tomograms but it's already been six hours and my submission is still not finished. I wonder if that is due to some bug or if the hidden test set is so much larger (More than 70 tomograms).
", "messages": [ { "id": 3049776, "postDate": "2024-11-19T13:59:02.003Z", "content": "How large is the test dataset? My submission script takes 15 minutes on the \"visible\" test dataset with three tomograms but it's already been six hours and my submission is still not finished. I wonder if that is due to some bug or if the hidden test set is so much larger (More than 70 tomograms).
", "rawMarkdown": "How large is the test dataset? My submission script takes 15 minutes on the \"visible\" test dataset with three tomograms but it's already been six hours and my submission is still not finished. I wonder if that is due to some bug or if the hidden test set is so much larger (More than 70 tomograms).", "votes": 7 }, { "id": 3052405, "postDate": "2024-11-22T12:10:15.913Z", "content": "seems that chatgpt offer another solution without CCL. Use pytorch distance transform:
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fbcdaadebd6e89e7348a175a75e4a999b%2FSelection_723.png?generation=1732277379369178&alt=media\")
one trick to speedup connected component labeling (CCL) is to make your predicted object (threshold binary sphere) small, i.e. smaller radius. there is less +ve pixel.
\nalternatively,use scale =0 for small particles, and scale=1 for larger particles (just resize from 0 to 1 for CCL. for segmentation we can still use scale 0 for all)
\nnow segmentation networks typically take about 10 sec to predict per pixel label.
\nspeedup GPU CL processing is about 3 sec.
to process all 500 hidden test tomography,that will be about 2hr
", "rawMarkdown": "one trick to speedup connected component labeling (CCL) is to make your predicted object (threshold binary sphere) small, i.e. smaller radius. there is less +ve pixel.\n\nalternatively,use scale =0 for small particles, and scale=1 for larger particles (just resize from 0 to 1 for CCL. for segmentation we can still use scale 0 for all)\n\nnow segmentation networks typically take about 10 sec to predict per pixel label.\nspeedup GPU CL processing is about 3 sec.\n\nto process all 500 hidden test tomography,that will be about 2hr", "votes": 1, "replies": [ { "id": 3052845, "postDate": "2024-11-22T22:00:38.973Z", "content": "https://www.kaggle.com/code/hengck23/1-hr-fast-2d-3d-unet-resnet34d-scanner-tta
\n4x speedup and reduce 5hr to 1~1.5hr for submission by post-processing at 0.5 scale for connected component labeling.
\nIt loses about 0.003 accuracy.
one can use it for fast development and revert to the original version for final submission.
", "rawMarkdown": "https://www.kaggle.com/code/hengck23/1-hr-fast-2d-3d-unet-resnet34d-scanner-tta\n\n4x speedup and reduce 5hr to 1~1.5hr for submission by post-processing at 0.5 scale for connected component labeling.\nIt loses about 0.003 accuracy.\n\none can use it for fast development and revert to the original version for final submission.", "votes": 2 } ] }, { "id": 3052382, "postDate": "2024-11-22T11:40:25.773Z", "content": "if that was not fast enough, then ….
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F03cbf6b7524d93bfa4b2de2eb0605d3a%2FSelection_720.png?generation=1732275583993758&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F75501691217b805a99f99dc4312a039c%2FSelection_719.png?generation=1732275596479102&alt=media\")
It is about 500 hidden test tomographs. mine takes about 4hr to score. you do not need cocomplicated network
", "rawMarkdown": "It is about 500 hidden test tomographs. mine takes about 4hr to score. you do not need cocomplicated network", "votes": 1, "replies": [ { "id": 3050407, "postDate": "2024-11-20T07:10:59.280Z", "content": "Nice! I have a pretty small model. It's the postprocessing part to extract centroids that takes most of the time.
", "rawMarkdown": "Nice! I have a pretty small model. It's the postprocessing part to extract centroids that takes most of the time.", "replies": [ { "id": 3050435, "postDate": "2024-11-20T07:56:16.560Z", "content": "what post- processing are you using?
\nthe fasting is using connected component + centroid
\n(14 sec for one tomo for one cpu. you can break tomo into 4 parts and run at 4 threads or compile a multi-thread c code)
\nhttps://www.kaggle.com/code/hengck23/speed-up-connected-component-analysis-with-pytorch
https://github.com/DanuserLab/u-segment3D
\nA general algorithm for consensus 3D cell segmentation from 2D segmented stacks
\nhttps://www.biorxiv.org/content/10.1101/2024.05.03.592249v2.full.pdf
i think you can find other parallel implementation. i haven't got time to check in details yet
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Ff7bb3a78b8021255c284fbf9383a767d%2FSelection_999(6864).png?generation=1732090310624912&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F2e48c3bf8bff69b576e905f53744ac7f%2FSelection_999(6863).png?generation=1732090319628289&alt=media\")
https://wrfranklin.org/nikola/pages/connect/
\nhttps://wrfranklin.org/pmwiki/pmwiki.php/Research/ConnectedComponents
1024x1088x1088
\n2GHz IBM T43p laptop
\n51 CPU seconds
The longest part for now is a watershed algorithm to split touching components. Will probably remove it and see how it impacts the final score.
", "rawMarkdown": "The longest part for now is a watershed algorithm to split touching components. Will probably remove it and see how it impacts the final score." }, { "id": 3050469, "postDate": "2024-11-20T08:33:21.050Z", "content": "i think watershed is not required:
\nthis explain why the particles don't touch each other in 2d or 3d or in physical setup:
\nhttps://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/547091
I am curious about how much speedup you were able to get with the gpu-based connected components algorithm. Would it be possible to share some numbers?
", "rawMarkdown": "I am curious about how much speedup you were able to get with the gpu-based connected components algorithm. Would it be possible to share some numbers? ", "isDeleted": true } ] } ] }, { "id": 3049825, "postDate": "2024-11-19T14:48:30.190Z", "content": "https://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/544867 Apparently 500
", "rawMarkdown": "https://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/544867 Apparently 500", "votes": 1, "replies": [ { "id": 3049834, "postDate": "2024-11-19T14:55:01.943Z", "content": "Holly molly, that leaves us with less than 90 seconds per image. And it means that the test 3 images should be finished in less than 5 minutes.
", "rawMarkdown": "Holly molly, that leaves us with less than 90 seconds per image. And it means that the test 3 images should be finished in less than 5 minutes.", "votes": 1, "replies": [ { "id": 3049840, "postDate": "2024-11-19T14:58:48.593Z", "content": "You're right. 500 images.
\nhttps://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/544702#3038910
\nWill probably have to use multiprocessing and the like to the max to fit in the allowed timeframe.
All test is processed and scored at submission. Although only public score is shown.
", "rawMarkdown": "All test is processed and scored at submission. Although only public score is shown.", "votes": 1 }, { "id": 3050827, "postDate": "2024-11-20T15:22:26.830Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3050929, "postDate": "2024-11-20T16:51:48.447Z", "content": "Again. No, they've already calculated, simply not shown yet.
", "rawMarkdown": "Again. No, they've already calculated, simply not shown yet.", "votes": 1 }, { "id": 3050980, "postDate": "2024-11-20T17:36:29.470Z", "content": "\"Again. No, they've already calculated, simply not shown yet.\"
\nThis statement is correct.
hey man do you still have this problem and if you dont how can you solve it
", "rawMarkdown": "hey man do you still have this problem and if you dont how can you solve it" }, { "id": 3049814, "postDate": "2024-11-19T14:38:08.323Z", "content": "My visible test takes also around 15 minuts. About to run my first submission. I'll let you know when it's finished. If it finish.
\nEDIT: I'm currently running 5 2D model, 7 fold and 4 rot90 inference because I though it was fast enough.
", "rawMarkdown": "My visible test takes also around 15 minuts. About to run my first submission. I'll let you know when it's finished. If it finish.\n\nEDIT: I'm currently running 5 2D model, 7 fold and 4 rot90 inference because I though it was fast enough.", "replies": [ { "id": 3049820, "postDate": "2024-11-19T14:43:53.707Z", "content": "That's a lot of models :)
", "rawMarkdown": "That's a lot of models :)" }, { "id": 3049895, "postDate": "2024-11-19T15:39:28.773Z", "content": "multiple 2d encode and a few 3d/2d encoder
", "rawMarkdown": "multiple 2d encode and a few 3d/2d encoder" } ] } ], "comments": [ { "id": 3052405, "author_name": "hengck23", "author_url": "", "post_date": "2024-11-22T12:10:15.913000", "content": "seems that chatgpt offer another solution without CCL. Use pytorch distance transform:
\n
one trick to speedup connected component labeling (CCL) is to make your predicted object (threshold binary sphere) small, i.e. smaller radius. there is less +ve pixel.
\nalternatively,use scale =0 for small particles, and scale=1 for larger particles (just resize from 0 to 1 for CCL. for segmentation we can still use scale 0 for all)
\nnow segmentation networks typically take about 10 sec to predict per pixel label.
\nspeedup GPU CL processing is about 3 sec.
to process all 500 hidden test tomography,that will be about 2hr
", "votes": 1, "replies": [ { "id": 3052845, "author_name": "hengck23", "author_url": "", "post_date": "2024-11-22T22:00:38.973000", "content": "https://www.kaggle.com/code/hengck23/1-hr-fast-2d-3d-unet-resnet34d-scanner-tta
\n4x speedup and reduce 5hr to 1~1.5hr for submission by post-processing at 0.5 scale for connected component labeling.
\nIt loses about 0.003 accuracy.
one can use it for fast development and revert to the original version for final submission.
", "votes": 2, "replies": [] } ] }, { "id": 3052382, "author_name": "hengck23", "author_url": "", "post_date": "2024-11-22T11:40:25.773000", "content": "if that was not fast enough, then ….
\n
It is about 500 hidden test tomographs. mine takes about 4hr to score. you do not need cocomplicated network
", "votes": 1, "replies": [ { "id": 3050407, "author_name": "DennisSakva", "author_url": "", "post_date": "2024-11-20T07:10:59.280000", "content": "Nice! I have a pretty small model. It's the postprocessing part to extract centroids that takes most of the time.
", "votes": 0, "replies": [ { "id": 3050435, "author_name": "hengck23", "author_url": "", "post_date": "2024-11-20T07:56:16.560000", "content": "what post- processing are you using?
\nthe fasting is using connected component + centroid
\n(14 sec for one tomo for one cpu. you can break tomo into 4 parts and run at 4 threads or compile a multi-thread c code)
\nhttps://www.kaggle.com/code/hengck23/speed-up-connected-component-analysis-with-pytorch
https://github.com/DanuserLab/u-segment3D
\nA general algorithm for consensus 3D cell segmentation from 2D segmented stacks
\nhttps://www.biorxiv.org/content/10.1101/2024.05.03.592249v2.full.pdf
i think you can find other parallel implementation. i haven't got time to check in details yet
\n
https://wrfranklin.org/nikola/pages/connect/
\nhttps://wrfranklin.org/pmwiki/pmwiki.php/Research/ConnectedComponents
1024x1088x1088
\n2GHz IBM T43p laptop
\n51 CPU seconds
The longest part for now is a watershed algorithm to split touching components. Will probably remove it and see how it impacts the final score.
", "votes": 0, "replies": [] }, { "id": 3050469, "author_name": "hengck23", "author_url": "", "post_date": "2024-11-20T08:33:21.050000", "content": "i think watershed is not required:
\nthis explain why the particles don't touch each other in 2d or 3d or in physical setup:
\nhttps://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/547091
I am curious about how much speedup you were able to get with the gpu-based connected components algorithm. Would it be possible to share some numbers?
", "votes": 0, "replies": [] } ] } ] }, { "id": 3049825, "author_name": "Ángel Jacinto Sánchez Ruiz", "author_url": "", "post_date": "2024-11-19T14:48:30.190000", "content": "https://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/544867 Apparently 500
", "votes": 1, "replies": [ { "id": 3049834, "author_name": "DennisSakva", "author_url": "", "post_date": "2024-11-19T14:55:01.943000", "content": "Holly molly, that leaves us with less than 90 seconds per image. And it means that the test 3 images should be finished in less than 5 minutes.
", "votes": 1, "replies": [ { "id": 3049840, "author_name": "DennisSakva", "author_url": "", "post_date": "2024-11-19T14:58:48.593000", "content": "You're right. 500 images.
\nhttps://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/544702#3038910
\nWill probably have to use multiprocessing and the like to the max to fit in the allowed timeframe.
All test is processed and scored at submission. Although only public score is shown.
", "votes": 1, "replies": [] }, { "id": 3050827, "author_name": "", "author_url": "", "post_date": "2024-11-20T15:22:26.830000", "content": "", "votes": 0, "replies": [] }, { "id": 3050929, "author_name": "Ángel Jacinto Sánchez Ruiz", "author_url": "", "post_date": "2024-11-20T16:51:48.447000", "content": "Again. No, they've already calculated, simply not shown yet.
", "votes": 1, "replies": [] }, { "id": 3050980, "author_name": "Reza Paraan", "author_url": "", "post_date": "2024-11-20T17:36:29.470000", "content": "\"Again. No, they've already calculated, simply not shown yet.\"
\nThis statement is correct.
hey man do you still have this problem and if you dont how can you solve it
", "votes": 0, "replies": [] }, { "id": 3049814, "author_name": "Ángel Jacinto Sánchez Ruiz", "author_url": "", "post_date": "2024-11-19T14:38:08.323000", "content": "My visible test takes also around 15 minuts. About to run my first submission. I'll let you know when it's finished. If it finish.
\nEDIT: I'm currently running 5 2D model, 7 fold and 4 rot90 inference because I though it was fast enough.
", "votes": 0, "replies": [ { "id": 3049820, "author_name": "DennisSakva", "author_url": "", "post_date": "2024-11-19T14:43:53.707000", "content": "That's a lot of models :)
", "votes": 0, "replies": [] }, { "id": 3049895, "author_name": "hengck23", "author_url": "", "post_date": "2024-11-19T15:39:28.773000", "content": "multiple 2d encode and a few 3d/2d encoder
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "3049776": "How large is the test dataset? My submission script takes 15 minutes on the \"visible\" test dataset with three tomograms but it's already been six hours and my submission is still not finished. I wonder if that is due to some bug or if the hidden test set is so much larger (More than 70 tomograms).", "3052405": "seems that chatgpt offer another solution without CCL. Use pytorch distance transform:\n\n\n", "3052394": "one trick to speedup connected component labeling (CCL) is to make your predicted object (threshold binary sphere) small, i.e. smaller radius. there is less +ve pixel.\n\nalternatively,use scale =0 for small particles, and scale=1 for larger particles (just resize from 0 to 1 for CCL. for segmentation we can still use scale 0 for all)\n\nnow segmentation networks typically take about 10 sec to predict per pixel label.\nspeedup GPU CL processing is about 3 sec.\n\nto process all 500 hidden test tomography,that will be about 2hr", "3052382": "if that was not fast enough, then ....\n\n\n\n\n", "3049892": "It is about 500 hidden test tomographs. mine takes about 4hr to score. you do not need cocomplicated network", "3049825": "https://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/544867 Apparently 500", "3051874": "hey man do you still have this problem and if you dont how can you solve it", "3049814": "My visible test takes also around 15 minuts. About to run my first submission. I'll let you know when it's finished. If it finish.\n\nEDIT: I'm currently running 5 2D model, 7 fold and 4 rot90 inference because I though it was fast enough." } }