![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14984949%2Fca4fcfd62f0fdd1993651b7b8603496c%2FScreenshot%202025-02-08%20084151.png?generation=1739000551141841&alt=media\")
Code (training + inference, ready to submit): https://www.kaggle.com/code/jeroencottaar/czii-11th-place-solution
\nOur thanks to the competition organizers - this was a great introduction to 3D segmentation. And of course my own thanks to my teammate @davidlist !
\nIntroduction
\nThe goal of the competition was to identify various proteins in cryo-electron-tomography data (basically 3D maps of electron density), making this a 3D particle recognition problem. One key element was that the competition metric was heavily biased toward recall over precision - missing particles (false negatives) was far more expensive than predicting too many (false positives).
\nAs most solutions, our core is a UNet segmentation model. We have added some elements that aren’t commonly present in the other solutions:
\nWhat we really missed was to properly optimize our model on the public test dataset. Even though we used many submissions, those weren't really used in probing. We experimented quite a bit with UNet architectures and hyperparameters, but only in cross-validation, where it was almost impossible to draw any useful conclusions due to the small training set size.
\nIn this writeup, we'll very briefly discuss the outline of our solution, and then focus on these 3 elements in turn.
\nOutline of solution
\nThe core of our solution is a UNet:
\nTo go from the UNet segmentation map to particle locations:
\nBatch normalization
\nFor all normalization layers in the UNet, we use batch normalization. This means that rather than normalizing each instance, the normalization factors are based on running averages found during training (and fixed during inference). This significantly improves score and training stability. Why?
\nImagine we're doing inference with either the red or the blue window below, and consider what happens at location X if we use anything other than batch normalization
\n
If we're using the red window, the artifact on the bottom right has a large impact on our prediction at X, since the normalization is based on all data in the window. When we use the blue window, we suddenly get a very different prediction at X. While some type of long-range influence can be beneficial in a neural network, it should be clear that this one is entirely spurious. Note that this also means we get discontinuities at the transitions between inference windows (even when discarding edge pixels).
\nIn MONAI specifically you can easily change from instance to batch normalization by adding \"batch='NORM'\" to the constructor, though you may have to mess with the momentum as well to get stable training (see our code for details).
\n2D classifier
\nAs described in the competition paper, the hosts included classification based on 2D images. This inspired us to try the same.
\nWe train a 2D neural network on the true and false positives of the UNet model (summing over the Z axis). The simplest architecture worked best: simply 3 fully connected layers (with dropout). For beta-galactosidase we add a single convolution layer at the start (without activation). Typical augmentations are applied, and some work is done to balance the true and false positives (see code for details).
\nAs we can see below for beta-galactosidase, this allows us to filter out quite some false positives (compared to using cluster size alone as a feature). The decision boundary does need to be tuned on the leaderboard, which we unfortunately didn't do enough of.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14984949%2Fd85d7b33439526933bf287f5fc451c24%2FScreenshot%202025-02-08%20084030.png?generation=1739000440465360&alt=media\")
Improved ribosomes
\nThis is perhaps more of a 'Kaggle thing' (as in something you wouldn't typically add to a production machine learning solution)…
\nWe noticed that we were struggling with clusters of ribosomes, and single ribosomes surrounded by other stuff. This is because clusters tend to melt together. We solve this by predicting multiple ribosomes for large, elongated clusters, using K-means to split up the cluster
", "messages": [ { "id": "3118561", "postDate": "02/08/2025 07:43:03", "content": "Code (training + inference, ready to submit): https://www.kaggle.com/code/jeroencottaar/czii-11th-place-solution
\nOur thanks to the competition organizers - this was a great introduction to 3D segmentation. And of course my own thanks to my teammate @davidlist !
\nIntroduction
\nThe goal of the competition was to identify various proteins in cryo-electron-tomography data (basically 3D maps of electron density), making this a 3D particle recognition problem. One key element was that the competition metric was heavily biased toward recall over precision - missing particles (false negatives) was far more expensive than predicting too many (false positives).
\nAs most solutions, our core is a UNet segmentation model. We have added some elements that aren’t commonly present in the other solutions:
\nWhat we really missed was to properly optimize our model on the public test dataset. Even though we used many submissions, those weren't really used in probing. We experimented quite a bit with UNet architectures and hyperparameters, but only in cross-validation, where it was almost impossible to draw any useful conclusions due to the small training set size.
\nIn this writeup, we'll very briefly discuss the outline of our solution, and then focus on these 3 elements in turn.
\nOutline of solution
\nThe core of our solution is a UNet:
\nTo go from the UNet segmentation map to particle locations:
\nBatch normalization
\nFor all normalization layers in the UNet, we use batch normalization. This means that rather than normalizing each instance, the normalization factors are based on running averages found during training (and fixed during inference). This significantly improves score and training stability. Why?
\nImagine we're doing inference with either the red or the blue window below, and consider what happens at location X if we use anything other than batch normalization
\n
If we're using the red window, the artifact on the bottom right has a large impact on our prediction at X, since the normalization is based on all data in the window. When we use the blue window, we suddenly get a very different prediction at X. While some type of long-range influence can be beneficial in a neural network, it should be clear that this one is entirely spurious. Note that this also means we get discontinuities at the transitions between inference windows (even when discarding edge pixels).
\nIn MONAI specifically you can easily change from instance to batch normalization by adding \"batch='NORM'\" to the constructor, though you may have to mess with the momentum as well to get stable training (see our code for details).
\n2D classifier
\nAs described in the competition paper, the hosts included classification based on 2D images. This inspired us to try the same.
\nWe train a 2D neural network on the true and false positives of the UNet model (summing over the Z axis). The simplest architecture worked best: simply 3 fully connected layers (with dropout). For beta-galactosidase we add a single convolution layer at the start (without activation). Typical augmentations are applied, and some work is done to balance the true and false positives (see code for details).
\nAs we can see below for beta-galactosidase, this allows us to filter out quite some false positives (compared to using cluster size alone as a feature). The decision boundary does need to be tuned on the leaderboard, which we unfortunately didn't do enough of.
\n
Improved ribosomes
\nThis is perhaps more of a 'Kaggle thing' (as in something you wouldn't typically add to a production machine learning solution)…
\nWe noticed that we were struggling with clusters of ribosomes, and single ribosomes surrounded by other stuff. This is because clusters tend to melt together. We solve this by predicting multiple ribosomes for large, elongated clusters, using K-means to split up the cluster
", "rawMarkdown": "Code (training + inference, ready to submit): https://www.kaggle.com/code/jeroencottaar/czii-11th-place-solution\n\nOur thanks to the competition organizers - this was a great introduction to 3D segmentation. And of course my own thanks to my teammate @davidlist !\n\n**Introduction**\n\nThe goal of the competition was to identify various proteins in cryo-electron-tomography data (basically 3D maps of electron density), making this a 3D particle recognition problem. One key element was that the competition metric was heavily biased toward recall over precision - missing particles (false negatives) was far more expensive than predicting too many (false positives).\n\nAs most solutions, our core is a UNet segmentation model. We have added some elements that aren’t commonly present in the other solutions:\n\n- Batch normalization, stabilizing training and making inference consistent across window positions\n- A post filter based on a 2D classification neural network, removing false positives for beta-galactosidase and thyroglobulin\n- Predicting multiple ribosomes in close vicinity, allowing us to hit more ribosomes in confusing areas\n\nWhat we really missed was to properly optimize our model on the public test dataset. Even though we used many submissions, those weren't really used in probing. We experimented quite a bit with UNet architectures and hyperparameters, but only in cross-validation, where it was almost impossible to draw any useful conclusions due to the small training set size.\n\nIn this writeup, we'll very briefly discuss the outline of our solution, and then focus on these 3 elements in turn.\n\n**Outline of solution**\n\nThe core of our solution is a UNet:\n\n- 32 models ensembled, with a mixture of ResNet architectures. No selection on CV or LB; we just train 32 times with different seeds.\n- Models are ensembled by taking the mean of the heatmaps before softmax.\n- All normalization is BatchNorm (details below).\n- Training is always 2200 epochs with no early stopping, with learning rate reduced towards the end\n- Augmentation: 90-degree rotations around Z axis, flipping in Y axis. Others didn't help.\n- Loss: weighted combination of cross-entropy and Tversky loss (alpha=0.5, beta=8).\n- No test time augmentation.\n\nTo go from the UNet segmentation map to particle locations:\n\n- Apply thresholding and identify connected components.\n- Take the centroids of each cluster as a prediction. If a cluster is very big, make multiple predictions (details below).\n- Select which particles to include based on the connected cluster size. For beta-galactosidase and thyroglobulin, apply an additional filter based on a 2D neural network classifier (details below).\n\n**Batch normalization**\n\nFor all normalization layers in the UNet, we use batch normalization. This means that rather than normalizing each instance, the normalization factors are based on running averages found during training (and fixed during inference). This significantly improves score and training stability. Why?\n\nImagine we're doing inference with either the red or the blue window below, and consider what happens at location X if we use anything other than batch normalization\n\n\n\nIf we're using the red window, the artifact on the bottom right has a large impact on our prediction at X, since the normalization is based on all data in the window. When we use the blue window, we suddenly get a very different prediction at X. While some type of long-range influence can be beneficial in a neural network, it should be clear that this one is entirely spurious. Note that this also means we get discontinuities at the transitions between inference windows (even when discarding edge pixels).\n\nIn MONAI specifically you can easily change from instance to batch normalization by adding \"batch='NORM'\" to the constructor, though you may have to mess with the momentum as well to get stable training (see our code for details).\n\n**2D classifier**\n\nAs described in the competition paper, the hosts included classification based on 2D images. This inspired us to try the same.\n\nWe train a 2D neural network on the true and false positives of the UNet model (summing over the Z axis). The simplest architecture worked best: simply 3 fully connected layers (with dropout). For beta-galactosidase we add a single convolution layer at the start (without activation). Typical augmentations are applied, and some work is done to balance the true and false positives (see code for details).\n\nAs we can see below for beta-galactosidase, this allows us to filter out quite some false positives (compared to using cluster size alone as a feature). The decision boundary does need to be tuned on the leaderboard, which we unfortunately didn't do enough of.\n\n\n\n**Improved ribosomes**\n\nThis is perhaps more of a 'Kaggle thing' (as in something you wouldn't typically add to a production machine learning solution)...\n\nWe noticed that we were struggling with clusters of ribosomes, and single ribosomes surrounded by other stuff. This is because clusters tend to melt together. We solve this by predicting multiple ribosomes for large, elongated clusters, using K-means to split up the cluster", "votes": null }, { "id": "3119436", "postDate": "02/09/2025 09:18:37", "content": "Training and inference code is now available: https://www.kaggle.com/code/jeroencottaar/czii-11th-place-solution
", "rawMarkdown": "Training and inference code is now available: https://www.kaggle.com/code/jeroencottaar/czii-11th-place-solution", "votes": null }, { "id": "3121917", "postDate": "02/12/2025 05:50:08", "content": "Nice work @jeroencottaar . I really enjoy reading your solution. Could you please share how much 2D NN classifiers for beta-galactosidase and thyroglobulin boosted your score? I was experimenting with something similar, we were checking the mean probability of the center pixels of the predicted blob something like 2x2x2 cube of beta-galacotosidase and try to find a confidence score for that![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4362752%2F0c2893ef06c9fbfe018d1dceaf0d3eba%2Fimage.webp?generation=1739339191405300&alt=media\")
.
I ruled out classification option because I thought for the classifier to work it should have an accuracy of > 94%, which I thought was quite difficult for hard particles.
", "rawMarkdown": "Nice work @jeroencottaar . I really enjoy reading your solution. Could you please share how much 2D NN classifiers for beta-galactosidase and thyroglobulin boosted your score? I was experimenting with something similar, we were checking the mean probability of the center pixels of the predicted blob something like 2x2x2 cube of beta-galacotosidase and try to find a confidence score for that. \n\nI ruled out classification option because I thought for the classifier to work it should have an accuracy of > 94%, which I thought was quite difficult for hard particles.", "votes": null }, { "id": "3122381", "postDate": "02/12/2025 15:17:12", "content": "Thanks for your interest! It's hard to estimate the impact of the 2D classifier exactly, because turning it off requires reoptimizing other thresholds. But from what we saw throughout the competition, I think it's gaining 0.01 to 0.02.
\nNote that we don't try to select particles based on the 2D classifier score alone; we combine it with our existing selection mechanism (cluster size from the UNet).
", "rawMarkdown": "Thanks for your interest! It's hard to estimate the impact of the 2D classifier exactly, because turning it off requires reoptimizing other thresholds. But from what we saw throughout the competition, I think it's gaining 0.01 to 0.02.\n\nNote that we don't try to select particles based on the 2D classifier score alone; we combine it with our existing selection mechanism (cluster size from the UNet).", "votes": null } ], "comments": [ { "id": 3119436, "author_name": "jeroencottaar", "author_url": "", "post_date": "02/09/2025 09:18:37", "content": "Training and inference code is now available: https://www.kaggle.com/code/jeroencottaar/czii-11th-place-solution
", "votes": null, "replies": [] }, { "id": 3121917, "author_name": "iamparadox", "author_url": "", "post_date": "02/12/2025 05:50:08", "content": "Nice work @jeroencottaar . I really enjoy reading your solution. Could you please share how much 2D NN classifiers for beta-galactosidase and thyroglobulin boosted your score? I was experimenting with something similar, we were checking the mean probability of the center pixels of the predicted blob something like 2x2x2 cube of beta-galacotosidase and try to find a confidence score for that.
I ruled out classification option because I thought for the classifier to work it should have an accuracy of > 94%, which I thought was quite difficult for hard particles.
", "votes": null, "replies": [ { "id": 3122381, "author_name": "jeroencottaar", "author_url": "", "post_date": "02/12/2025 15:17:12", "content": "Thanks for your interest! It's hard to estimate the impact of the 2D classifier exactly, because turning it off requires reoptimizing other thresholds. But from what we saw throughout the competition, I think it's gaining 0.01 to 0.02.
\nNote that we don't try to select particles based on the 2D classifier score alone; we combine it with our existing selection mechanism (cluster size from the UNet).
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "3118561": "Code (training + inference, ready to submit): https://www.kaggle.com/code/jeroencottaar/czii-11th-place-solution\n\nOur thanks to the competition organizers - this was a great introduction to 3D segmentation. And of course my own thanks to my teammate @davidlist !\n\n**Introduction**\n\nThe goal of the competition was to identify various proteins in cryo-electron-tomography data (basically 3D maps of electron density), making this a 3D particle recognition problem. One key element was that the competition metric was heavily biased toward recall over precision - missing particles (false negatives) was far more expensive than predicting too many (false positives).\n\nAs most solutions, our core is a UNet segmentation model. We have added some elements that aren’t commonly present in the other solutions:\n\n- Batch normalization, stabilizing training and making inference consistent across window positions\n- A post filter based on a 2D classification neural network, removing false positives for beta-galactosidase and thyroglobulin\n- Predicting multiple ribosomes in close vicinity, allowing us to hit more ribosomes in confusing areas\n\nWhat we really missed was to properly optimize our model on the public test dataset. Even though we used many submissions, those weren't really used in probing. We experimented quite a bit with UNet architectures and hyperparameters, but only in cross-validation, where it was almost impossible to draw any useful conclusions due to the small training set size.\n\nIn this writeup, we'll very briefly discuss the outline of our solution, and then focus on these 3 elements in turn.\n\n**Outline of solution**\n\nThe core of our solution is a UNet:\n\n- 32 models ensembled, with a mixture of ResNet architectures. No selection on CV or LB; we just train 32 times with different seeds.\n- Models are ensembled by taking the mean of the heatmaps before softmax.\n- All normalization is BatchNorm (details below).\n- Training is always 2200 epochs with no early stopping, with learning rate reduced towards the end\n- Augmentation: 90-degree rotations around Z axis, flipping in Y axis. Others didn't help.\n- Loss: weighted combination of cross-entropy and Tversky loss (alpha=0.5, beta=8).\n- No test time augmentation.\n\nTo go from the UNet segmentation map to particle locations:\n\n- Apply thresholding and identify connected components.\n- Take the centroids of each cluster as a prediction. If a cluster is very big, make multiple predictions (details below).\n- Select which particles to include based on the connected cluster size. For beta-galactosidase and thyroglobulin, apply an additional filter based on a 2D neural network classifier (details below).\n\n**Batch normalization**\n\nFor all normalization layers in the UNet, we use batch normalization. This means that rather than normalizing each instance, the normalization factors are based on running averages found during training (and fixed during inference). This significantly improves score and training stability. Why?\n\nImagine we're doing inference with either the red or the blue window below, and consider what happens at location X if we use anything other than batch normalization\n\n\n\nIf we're using the red window, the artifact on the bottom right has a large impact on our prediction at X, since the normalization is based on all data in the window. When we use the blue window, we suddenly get a very different prediction at X. While some type of long-range influence can be beneficial in a neural network, it should be clear that this one is entirely spurious. Note that this also means we get discontinuities at the transitions between inference windows (even when discarding edge pixels).\n\nIn MONAI specifically you can easily change from instance to batch normalization by adding \"batch='NORM'\" to the constructor, though you may have to mess with the momentum as well to get stable training (see our code for details).\n\n**2D classifier**\n\nAs described in the competition paper, the hosts included classification based on 2D images. This inspired us to try the same.\n\nWe train a 2D neural network on the true and false positives of the UNet model (summing over the Z axis). The simplest architecture worked best: simply 3 fully connected layers (with dropout). For beta-galactosidase we add a single convolution layer at the start (without activation). Typical augmentations are applied, and some work is done to balance the true and false positives (see code for details).\n\nAs we can see below for beta-galactosidase, this allows us to filter out quite some false positives (compared to using cluster size alone as a feature). The decision boundary does need to be tuned on the leaderboard, which we unfortunately didn't do enough of.\n\n\n\n**Improved ribosomes**\n\nThis is perhaps more of a 'Kaggle thing' (as in something you wouldn't typically add to a production machine learning solution)...\n\nWe noticed that we were struggling with clusters of ribosomes, and single ribosomes surrounded by other stuff. This is because clusters tend to melt together. We solve this by predicting multiple ribosomes for large, elongated clusters, using K-means to split up the cluster", "3119436": "Training and inference code is now available: https://www.kaggle.com/code/jeroencottaar/czii-11th-place-solution", "3121917": "Nice work @jeroencottaar . I really enjoy reading your solution. Could you please share how much 2D NN classifiers for beta-galactosidase and thyroglobulin boosted your score? I was experimenting with something similar, we were checking the mean probability of the center pixels of the predicted blob something like 2x2x2 cube of beta-galacotosidase and try to find a confidence score for that. \n\nI ruled out classification option because I thought for the classifier to work it should have an accuracy of > 94%, which I thought was quite difficult for hard particles.", "3122381": "Thanks for your interest! It's hard to estimate the impact of the 2D classifier exactly, because turning it off requires reoptimizing other thresholds. But from what we saw throughout the competition, I think it's gaining 0.01 to 0.02.\n\nNote that we don't try to select particles based on the 2D classifier score alone; we combine it with our existing selection mechanism (cluster size from the UNet)." }, "source": "meta" }