{ "id": 561844, "title": "10th place solution", "url": "/competitions/czii-cryo-et-object-identification/writeups/josef-slavicek-10th-place-solution", "author_name": "", "post_date": "2025-02-15T13:36:18.827Z", "votes": 15, "comment_count": 5, "views": 0, "content": "

I wish to apologize to organizers: they gave us very well prepared interesting competition in the hope to promote substantial progress in CryoET processing. What they get from me is just ensemble of vintage 3DUNets.

\n

And I wish to thank to authors of some helpful notebooks, especially this. Without them I would not be able to develop anything working.

\n

Short description

\n

My solution is ensemble of 9 3DUnets. All except of one were first pretrained on simulated data and then finetuned on competition train data. I ensemble them by just averaging equaly weighted logits. Then I compute probabilities, threshold them to obtain regions of detection and then I apply some postprocessing to split regions into individual particles. All my trainings were in fact Optuna hyperparam-searching sessions from which I chose best models according to held-out local validation set.

\n

Validation

\n

I made the train / validation split just by experiment ID. For finetuning on competition train data, I train on 6 volumes and validate on one remaining. For pretraining on simulated data I still use performance on one of train data experiments for validation.

\n

Data processing, augmentations

\n

The only data processing I use was normalization with monai.transforms.NormalizeIntensityd. Regarding augmentations, I use RandFlipd, RandRotated and RandRotate90d from monai.transforms. During training I use simple implementation of mixup but Optuna chose to give it very small alpha (~0.03).

\n

NN architecture

\n

All my nets are the same architecture: monai.networks.net.UNet with spatial_dims=3, channels=(48, 64, 80, 80, 128), stride_patterns=(2,2,2,1).

\n

Training

\n

All my trainings were done as Optuna hyperparam-search session where the inner trainig was done in pytorch lightning. I was using AdamW optimizer with cosine LR scheduler. Optuna settled to some unusual values of Adams beta parameters (e.g. beta1=0.7, beta2=0.9996).

\n

Training objective

\n

For finetuning on competition data I was asking UNet to predict particle classes map. As a loss function I was using weighted combination of Tversky loss and multiclass crossentropy. I clipped input logits for Tversky loss. For pretraining on simulated data I used 3 versions of objective.

\n

1) the same as fintuning.

\n

2) with particle orientation vector as another objective. I hoped that it will force the network to understand particle shapes, but in fact the network didn't predict the orientation better than random guessing.

\n

3) some my earlier implementation of 2) where there was bug making the orientation really random. Ironically, this was my most successful pretraining, present in 4 out of 9 my final networks

\n

Postprocessing

\n

After fusion, computing probabilities and thresholding, we get binary maps representing particles. Most of the time, isolated clusters of positive values really correspond to just single particle. But sometimes I have seen something like this:
\n\"\"
\nSo we need to recognize and split such multiple detection. I guess there are some well established, proven solution how to do it, but I took it as a nice opportunity to reinvent wheel an I end up doing following (explained on 1D image for clarity):
\n\"\"
\nWe have probabilty density function (pdf) which is prediction of our model (image A) - let's name it P. We know very well how pdf of single particle (let's call it Q) looks like (image B), only we don't know where it is and how many of them is there. But if we place Q over P such way to minimize KL divergence, we get exactly what we want (image C). Only we must be careful because KL is not symmetric, so inserting P and Q in wrong order into KL formula would lead to undesirable result (image D). But if we are doing it correctly, we have placed first particle and if we place other, it will occupy other node of P (image E). We can continue doing this. Once our placed Qs start to overlap (image F), we are done.

\n

So this is first part of my crazy KL machinery. To explain the remainder, let's switch to 2D view and forget meaning of colors from previous image - it will now be different:
\n\"\"
\nSo we have our model prediction, from which we can draw binary mask, from which we can detect border - red line in image A. And we know centroids of individual particle pdfs - green and blue cross. Just using these centroids would give us good prediction but I got some smallish improvement from following trickery. We can split the red border to parts belonging to individual particles (image B). Now let's process them separately. If we randomly pick 4 points (4 points in real 3D problem, while in our 2D image it would be 3 points) and find their circumcenter (unique point equally distant from them), we get good candidate for particle centroid (image C). So I use this to obtain clouds of particle center candidates and then I find the final centroid prediction by averaging some dense core of this cluster (image D).

\n

That's it. Sorry. I couldn't resist :-)

\n

This whole KL magic improve my LB score by ~0.015 and it takes 90min to run on P100.

", "messages": [ { "id": "3118591", "postDate": "02/08/2025 08:39:16", "content": "

I wish to apologize to organizers: they gave us very well prepared interesting competition in the hope to promote substantial progress in CryoET processing. What they get from me is just ensemble of vintage 3DUNets.

\n

And I wish to thank to authors of some helpful notebooks, especially this. Without them I would not be able to develop anything working.

\n

Short description

\n

My solution is ensemble of 9 3DUnets. All except of one were first pretrained on simulated data and then finetuned on competition train data. I ensemble them by just averaging equaly weighted logits. Then I compute probabilities, threshold them to obtain regions of detection and then I apply some postprocessing to split regions into individual particles. All my trainings were in fact Optuna hyperparam-searching sessions from which I chose best models according to held-out local validation set.

\n

Validation

\n

I made the train / validation split just by experiment ID. For finetuning on competition train data, I train on 6 volumes and validate on one remaining. For pretraining on simulated data I still use performance on one of train data experiments for validation.

\n

Data processing, augmentations

\n

The only data processing I use was normalization with monai.transforms.NormalizeIntensityd. Regarding augmentations, I use RandFlipd, RandRotated and RandRotate90d from monai.transforms. During training I use simple implementation of mixup but Optuna chose to give it very small alpha (~0.03).

\n

NN architecture

\n

All my nets are the same architecture: monai.networks.net.UNet with spatial_dims=3, channels=(48, 64, 80, 80, 128), stride_patterns=(2,2,2,1).

\n

Training

\n

All my trainings were done as Optuna hyperparam-search session where the inner trainig was done in pytorch lightning. I was using AdamW optimizer with cosine LR scheduler. Optuna settled to some unusual values of Adams beta parameters (e.g. beta1=0.7, beta2=0.9996).

\n

Training objective

\n

For finetuning on competition data I was asking UNet to predict particle classes map. As a loss function I was using weighted combination of Tversky loss and multiclass crossentropy. I clipped input logits for Tversky loss. For pretraining on simulated data I used 3 versions of objective.

\n

1) the same as fintuning.

\n

2) with particle orientation vector as another objective. I hoped that it will force the network to understand particle shapes, but in fact the network didn't predict the orientation better than random guessing.

\n

3) some my earlier implementation of 2) where there was bug making the orientation really random. Ironically, this was my most successful pretraining, present in 4 out of 9 my final networks

\n

Postprocessing

\n

After fusion, computing probabilities and thresholding, we get binary maps representing particles. Most of the time, isolated clusters of positive values really correspond to just single particle. But sometimes I have seen something like this:
\n\"\"
\nSo we need to recognize and split such multiple detection. I guess there are some well established, proven solution how to do it, but I took it as a nice opportunity to reinvent wheel an I end up doing following (explained on 1D image for clarity):
\n\"\"
\nWe have probabilty density function (pdf) which is prediction of our model (image A) - let's name it P. We know very well how pdf of single particle (let's call it Q) looks like (image B), only we don't know where it is and how many of them is there. But if we place Q over P such way to minimize KL divergence, we get exactly what we want (image C). Only we must be careful because KL is not symmetric, so inserting P and Q in wrong order into KL formula would lead to undesirable result (image D). But if we are doing it correctly, we have placed first particle and if we place other, it will occupy other node of P (image E). We can continue doing this. Once our placed Qs start to overlap (image F), we are done.

\n

So this is first part of my crazy KL machinery. To explain the remainder, let's switch to 2D view and forget meaning of colors from previous image - it will now be different:
\n\"\"
\nSo we have our model prediction, from which we can draw binary mask, from which we can detect border - red line in image A. And we know centroids of individual particle pdfs - green and blue cross. Just using these centroids would give us good prediction but I got some smallish improvement from following trickery. We can split the red border to parts belonging to individual particles (image B). Now let's process them separately. If we randomly pick 4 points (4 points in real 3D problem, while in our 2D image it would be 3 points) and find their circumcenter (unique point equally distant from them), we get good candidate for particle centroid (image C). So I use this to obtain clouds of particle center candidates and then I find the final centroid prediction by averaging some dense core of this cluster (image D).

\n

That's it. Sorry. I couldn't resist :-)

\n

This whole KL magic improve my LB score by ~0.015 and it takes 90min to run on P100.

", "rawMarkdown": "I wish to apologize to organizers: they gave us very well prepared interesting competition in the hope to promote substantial progress in CryoET processing. What they get from me is just ensemble of vintage 3DUNets.\n\nAnd I wish to thank to authors of some helpful notebooks, especially [this](https://www.kaggle.com/code/fnands/baseline-unet-train-submit). Without them I would not be able to develop anything working.\n\n### Short description\n\nMy solution is ensemble of 9 3DUnets. All except of one were first pretrained on [simulated data](https://cryoetdataportal.czscience.com/datasets/10441) and then finetuned on competition train data. I ensemble them by just averaging equaly weighted logits. Then I compute probabilities, threshold them to obtain regions of detection and then I apply some postprocessing to split regions into individual particles. All my trainings were in fact Optuna hyperparam-searching sessions from which I chose best models according to held-out local validation set.\n\n### Validation\nI made the train / validation split just by experiment ID. For finetuning on competition train data, I train on 6 volumes and validate on one remaining. For pretraining on simulated data I still use performance on one of train data experiments for validation.\n\n### Data processing, augmentations\nThe only data processing I use was normalization with [monai.transforms.NormalizeIntensityd](https://docs.monai.io/en/stable/transforms.html#normalizeintensityd). Regarding augmentations, I use RandFlipd, RandRotated and RandRotate90d from monai.transforms. During training I use simple implementation of mixup but Optuna chose to give it very small `alpha` (~0.03).\n\n### NN architecture\nAll my nets are the same architecture: [monai.networks.net.UNet](https://docs.monai.io/en/0.9.1/_modules/monai/networks/nets/unet.html) with `spatial_dims=3, channels=(48, 64, 80, 80, 128), stride_patterns=(2,2,2,1)`.\n\n### Training\nAll my trainings were done as Optuna hyperparam-search session where the inner trainig was done in pytorch lightning. I was using AdamW optimizer with cosine LR scheduler. Optuna settled to some unusual values of Adams beta parameters (e.g. `beta1=0.7, beta2=0.9996`). \n\n### Training objective\nFor finetuning on competition data I was asking UNet to predict particle classes map. As a loss function I was using weighted combination of Tversky loss and multiclass crossentropy. I clipped input logits for Tversky loss. For pretraining on simulated data I used 3 versions of objective. \n\n1) the same as fintuning. \n\n2) with particle orientation vector as another objective. I hoped that it will force the network to understand particle shapes, but in fact the network didn't predict the orientation better than random guessing.\n\n3) some my earlier implementation of 2) where there was bug making the orientation really random. Ironically, this was my most successful pretraining, present in 4 out of 9 my final networks\n\n### Postprocessing\n\nAfter fusion, computing probabilities and thresholding, we get binary maps representing particles. Most of the time, isolated clusters of positive values really correspond to just single particle. But sometimes I have seen something like this:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F59264%2F761fac4d035de4125600736f6bb87828%2Fdouble_detection.png?generation=1739000719683072&alt=media)\nSo we need to recognize and split such multiple detection. I guess there are some well established, proven solution how to do it, but I took it as a nice opportunity to reinvent wheel an I end up doing following (explained on 1D image for clarity):\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F59264%2F6cd5fff5add3fc510c964e50b2560ff0%2FNotes_250208_085947.jpg?generation=1739001737372679&alt=media)\nWe have probabilty density function (pdf) which is prediction of our model (image A) - let's name it P. We know very well how pdf of single particle (let's call it Q) looks like (image B), only we don't know where it is and how many of them is there. But if we place Q over P such way to minimize KL divergence, we get exactly what we want (image C). Only we must be careful because KL is not symmetric, so inserting P and Q in wrong order into KL formula would lead to undesirable result (image D). But if we are doing it correctly, we have placed first particle and if we place other, it will occupy other node of P (image E). We can continue doing this. Once our placed Qs start to overlap (image F), we are done.\n\nSo this is first part of my crazy KL machinery. To explain the remainder, let's switch to 2D view and forget meaning of colors from previous image - it will now be different:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F59264%2F02c8242e88a3406d217ff2a80c9a9c1d%2FNotes_250208_092243.jpg?generation=1739003049901595&alt=media)\nSo we have our model prediction, from which we can draw binary mask, from which we can detect border - red line in image A. And we know centroids of individual particle pdfs - green and blue cross. Just using these centroids would give us good prediction but I got some smallish improvement from following trickery. We can split the red border to parts belonging to individual particles (image B). Now let's process them separately. If we randomly pick 4 points (4 points in real 3D problem, while in our 2D image it would be 3 points) and find their circumcenter (unique point equally distant from them), we get good candidate for particle centroid (image C). So I use this to obtain clouds of particle center candidates and then I find the final centroid prediction by averaging some dense core of this cluster (image D).\n\nThat's it. Sorry. I couldn't resist :-)\n\nThis whole KL magic improve my LB score by ~0.015 and it takes 90min to run on P100.", "votes": null }, { "id": "3118646", "postDate": "02/08/2025 10:05:21", "content": "

This is amazing, i'd love to see your Notebook regarding this if you'd publish it!

", "rawMarkdown": "This is amazing, i'd love to see your Notebook regarding this if you'd publish it!", "votes": null }, { "id": "3128478", "postDate": "02/19/2025 15:44:57", "content": "

Hi Josef. Thanks again for participating and for your valuable report. I have a question regarding your post-processing. Do you calculate particle diameters after this step? If so, do you use that information to decide whether that particle belongs to the right class or not? For example, a lot of apoferritin particles cluster together. Without your postprocessing this cluster might seem like one big particle such as a VLP. I'm just curious if your method takes care of that kind of particle class confusion. Thank you so much.

", "rawMarkdown": "Hi Josef. Thanks again for participating and for your valuable report. I have a question regarding your post-processing. Do you calculate particle diameters after this step? If so, do you use that information to decide whether that particle belongs to the right class or not? For example, a lot of apoferritin particles cluster together. Without your postprocessing this cluster might seem like one big particle such as a VLP. I'm just curious if your method takes care of that kind of particle class confusion. Thank you so much.", "votes": null }, { "id": "3130672", "postDate": "02/21/2025 22:21:48", "content": "

Hello Reza, I apologize for the late reply, I was not monitoring this discussion.
\nNo, I process prediction of each class separately. Particle diameter enters this calculation as a fixed parameter and my postprocessing is trying to do just single thing: to place single particle predictions over the NN output for that class.

", "rawMarkdown": "Hello Reza, I apologize for the late reply, I was not monitoring this discussion.\nNo, I process prediction of each class separately. Particle diameter enters this calculation as a fixed parameter and my postprocessing is trying to do just single thing: to place single particle predictions over the NN output for that class.", "votes": null }, { "id": "3130676", "postDate": "02/21/2025 22:36:41", "content": "

Hi Josef, thanks for your response.

", "rawMarkdown": "Hi Josef, thanks for your response.", "votes": null }, { "id": "3186952", "postDate": "04/25/2025 10:37:53", "content": "

This is amazing, would love to see your notebook, especially the KL section.
\nthanks for the explanation.

", "rawMarkdown": "This is amazing, would love to see your notebook, especially the KL section.\nthanks for the explanation.", "votes": null } ], "comments": [ { "id": 3118646, "author_name": "alaasweed", "author_url": "", "post_date": "02/08/2025 10:05:21", "content": "

This is amazing, i'd love to see your Notebook regarding this if you'd publish it!

", "votes": null, "replies": [] }, { "id": 3128478, "author_name": "rezaparaan", "author_url": "", "post_date": "02/19/2025 15:44:57", "content": "

Hi Josef. Thanks again for participating and for your valuable report. I have a question regarding your post-processing. Do you calculate particle diameters after this step? If so, do you use that information to decide whether that particle belongs to the right class or not? For example, a lot of apoferritin particles cluster together. Without your postprocessing this cluster might seem like one big particle such as a VLP. I'm just curious if your method takes care of that kind of particle class confusion. Thank you so much.

", "votes": null, "replies": [ { "id": 3130672, "author_name": "josefslavicek", "author_url": "", "post_date": "02/21/2025 22:21:48", "content": "

Hello Reza, I apologize for the late reply, I was not monitoring this discussion.
\nNo, I process prediction of each class separately. Particle diameter enters this calculation as a fixed parameter and my postprocessing is trying to do just single thing: to place single particle predictions over the NN output for that class.

", "votes": null, "replies": [ { "id": 3130676, "author_name": "rezaparaan", "author_url": "", "post_date": "02/21/2025 22:36:41", "content": "

Hi Josef, thanks for your response.

", "votes": null, "replies": [] } ] } ] }, { "id": 3186952, "author_name": "arpit1bansal", "author_url": "", "post_date": "04/25/2025 10:37:53", "content": "

This is amazing, would love to see your notebook, especially the KL section.
\nthanks for the explanation.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3118591": "I wish to apologize to organizers: they gave us very well prepared interesting competition in the hope to promote substantial progress in CryoET processing. What they get from me is just ensemble of vintage 3DUNets.\n\nAnd I wish to thank to authors of some helpful notebooks, especially [this](https://www.kaggle.com/code/fnands/baseline-unet-train-submit). Without them I would not be able to develop anything working.\n\n### Short description\n\nMy solution is ensemble of 9 3DUnets. All except of one were first pretrained on [simulated data](https://cryoetdataportal.czscience.com/datasets/10441) and then finetuned on competition train data. I ensemble them by just averaging equaly weighted logits. Then I compute probabilities, threshold them to obtain regions of detection and then I apply some postprocessing to split regions into individual particles. All my trainings were in fact Optuna hyperparam-searching sessions from which I chose best models according to held-out local validation set.\n\n### Validation\nI made the train / validation split just by experiment ID. For finetuning on competition train data, I train on 6 volumes and validate on one remaining. For pretraining on simulated data I still use performance on one of train data experiments for validation.\n\n### Data processing, augmentations\nThe only data processing I use was normalization with [monai.transforms.NormalizeIntensityd](https://docs.monai.io/en/stable/transforms.html#normalizeintensityd). Regarding augmentations, I use RandFlipd, RandRotated and RandRotate90d from monai.transforms. During training I use simple implementation of mixup but Optuna chose to give it very small `alpha` (~0.03).\n\n### NN architecture\nAll my nets are the same architecture: [monai.networks.net.UNet](https://docs.monai.io/en/0.9.1/_modules/monai/networks/nets/unet.html) with `spatial_dims=3, channels=(48, 64, 80, 80, 128), stride_patterns=(2,2,2,1)`.\n\n### Training\nAll my trainings were done as Optuna hyperparam-search session where the inner trainig was done in pytorch lightning. I was using AdamW optimizer with cosine LR scheduler. Optuna settled to some unusual values of Adams beta parameters (e.g. `beta1=0.7, beta2=0.9996`). \n\n### Training objective\nFor finetuning on competition data I was asking UNet to predict particle classes map. As a loss function I was using weighted combination of Tversky loss and multiclass crossentropy. I clipped input logits for Tversky loss. For pretraining on simulated data I used 3 versions of objective. \n\n1) the same as fintuning. \n\n2) with particle orientation vector as another objective. I hoped that it will force the network to understand particle shapes, but in fact the network didn't predict the orientation better than random guessing.\n\n3) some my earlier implementation of 2) where there was bug making the orientation really random. Ironically, this was my most successful pretraining, present in 4 out of 9 my final networks\n\n### Postprocessing\n\nAfter fusion, computing probabilities and thresholding, we get binary maps representing particles. Most of the time, isolated clusters of positive values really correspond to just single particle. But sometimes I have seen something like this:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F59264%2F761fac4d035de4125600736f6bb87828%2Fdouble_detection.png?generation=1739000719683072&alt=media)\nSo we need to recognize and split such multiple detection. I guess there are some well established, proven solution how to do it, but I took it as a nice opportunity to reinvent wheel an I end up doing following (explained on 1D image for clarity):\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F59264%2F6cd5fff5add3fc510c964e50b2560ff0%2FNotes_250208_085947.jpg?generation=1739001737372679&alt=media)\nWe have probabilty density function (pdf) which is prediction of our model (image A) - let's name it P. We know very well how pdf of single particle (let's call it Q) looks like (image B), only we don't know where it is and how many of them is there. But if we place Q over P such way to minimize KL divergence, we get exactly what we want (image C). Only we must be careful because KL is not symmetric, so inserting P and Q in wrong order into KL formula would lead to undesirable result (image D). But if we are doing it correctly, we have placed first particle and if we place other, it will occupy other node of P (image E). We can continue doing this. Once our placed Qs start to overlap (image F), we are done.\n\nSo this is first part of my crazy KL machinery. To explain the remainder, let's switch to 2D view and forget meaning of colors from previous image - it will now be different:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F59264%2F02c8242e88a3406d217ff2a80c9a9c1d%2FNotes_250208_092243.jpg?generation=1739003049901595&alt=media)\nSo we have our model prediction, from which we can draw binary mask, from which we can detect border - red line in image A. And we know centroids of individual particle pdfs - green and blue cross. Just using these centroids would give us good prediction but I got some smallish improvement from following trickery. We can split the red border to parts belonging to individual particles (image B). Now let's process them separately. If we randomly pick 4 points (4 points in real 3D problem, while in our 2D image it would be 3 points) and find their circumcenter (unique point equally distant from them), we get good candidate for particle centroid (image C). So I use this to obtain clouds of particle center candidates and then I find the final centroid prediction by averaging some dense core of this cluster (image D).\n\nThat's it. Sorry. I couldn't resist :-)\n\nThis whole KL magic improve my LB score by ~0.015 and it takes 90min to run on P100.", "3118646": "This is amazing, i'd love to see your Notebook regarding this if you'd publish it!", "3128478": "Hi Josef. Thanks again for participating and for your valuable report. I have a question regarding your post-processing. Do you calculate particle diameters after this step? If so, do you use that information to decide whether that particle belongs to the right class or not? For example, a lot of apoferritin particles cluster together. Without your postprocessing this cluster might seem like one big particle such as a VLP. I'm just curious if your method takes care of that kind of particle class confusion. Thank you so much.", "3130672": "Hello Reza, I apologize for the late reply, I was not monitoring this discussion.\nNo, I process prediction of each class separately. Particle diameter enters this calculation as a fixed parameter and my postprocessing is trying to do just single thing: to place single particle predictions over the NN output for that class.", "3130676": "Hi Josef, thanks for your response.", "3186952": "This is amazing, would love to see your notebook, especially the KL section.\nthanks for the explanation." }, "source": "meta" }