{ "id": 561568, "title": "2nd Place Solution", "url": "/competitions/czii-cryo-et-object-identification/discussion/561568", "author_name": "AnnieGo", "post_date": "2025-02-06T18:53:56.557000", "votes": 44, "comment_count": 11, "views": 0, "content": "

Acknowledgments

\n

We sincerely appreciate Kaggle and the competition organizers for offering this invaluable opportunity. We also extend our gratitude to @hengck23 and @fnands for their significant contributions. fnands's notebook and hengck23's discussion provided a solid foundation for our approach. Finally, I am grateful to my teammate @luoziqian for the excellent collaboration.

\n

Summary

\n

Our approach is based on an ensemble of multiple lightweight segmentation models, with parameter sizes ranging from 873K to 14.2M. Following segmentation, we computed particle centroids using CC3D and filtered small clusters based on voxel count statistics.

\n

Overall Pipeline

\n

The overall pipeline is illustrated below:
\n

\n

Model Architecture

\n

We initiated our experiments using the MONAI UNet baseline model. Throughout our trials, we observed that models with a large number of parameters were prone to overfitting and often performed worse than lighter models. Consequently, we opted for lightweight architectures such as UNet3D, VoxResNet, VoxHRNet, SegResNet, DynUNet, and DenseVNet.

\n

However, while experimenting with VoxResNet, VoxHRNet, and DenseVNet, we encountered stability issues, including performance fluctuations and difficulties in convergence. Upon further analysis, we identified that MONAI UNet employs InstanceNorm3d and PReLU. By modifying the normalization and activation layers accordingly, we achieved more stable model performance.

\n

The model architecture of MONAI UNet is shown below:
\n\"\"

\n

The following figure compares the training performance of MONAI UNet with InstanceNorm3d and PReLU versus BatchNorm3d and ReLU across 5 experiments, demonstrating that the use of InstanceNorm3d and PReLU results in more stable training.

\n

\n

For the final ensemble, we selected models based on their public leaderboard scores:

\n\n

Training Strategy

\n

Our training configuration was designed to ensure stability and optimal performance. We found that the segmentation mask radius, loss function and data augmentation strategies played a crucial role in achieving reliable results.

\n

Segmentation Mask:

\n

We applied ground truth masks with a customized radius for each particle.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Particle TypesDefault RadiusLuoziqian's RadiusLion's Radius
apo-ferritin6060 x 0.580 x 0.4
beta-galactosidase9090 x 0.590 x 0.4
ribosome150150 x 0.5150 x 0.4
thyroglobulin130130 x 0.5120 x 0.4
virus-like-particle135135 x 0.5150 x 0.4
\n

Training Settings:

\n

Lion's models:

\n\n

Luoziqian's models

\n\n

Augmentations:

\n

Lion's models:

\n\n

Luoziqian's models

\n\n

Model Performance with 7 TTA Summary (Partial Selection)

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
NoModelDeveloperArchitectureParametersValid IDNormalizationActivationPublic LBPrivate LB
1epoch122-step2952-valid_loss0.3625-val_metric0.8367.ckptLionUNet3D1.1MTS_86_3InstanceNorm3dPReLU0.773790.76582
2epoch148-step3576-valid_loss1.1154-val_metric0.7722.ckptLionUNet3D1.1MTS_6_4InstanceNorm3dPReLU0.770210.76725
3epoch153-step3696-valid_loss0.3021-val_metric0.8900.ckptLionUNet3D1.6MTS_69_2InstanceNorm3dPReLU0.772050.76676
4epoch194-step4680-valid_loss1.0213-val_metric0.8788.ckptLionUNet3D1.1MTS_69_2InstanceNorm3dPReLU0.773900.76737
5epoch138-step3336-valid_loss0.3690-val_metric0.8476.ckptLionUNet3D1.1MTS_73_6InstanceNorm3dPReLU0.765430.76025
6epoch152-step3672-valid_loss0.4333-val_metric0.7929.ckptLionDenseVNet873KTS_6_6InstanceNorm3dPReLU0.765280.75417
7epoch195-step4704-valid_loss0.4258-val_metric0.7914.ckptLionVoxResNet7.0MTS_6_6InstanceNorm3dPReLU0.774570.76593
8epoch188-step4536-valid_loss0.4231-val_metric0.8659.ckptLionVoxHRNet1.4MTS_73_6InstanceNorm3dPReLU0.767380.75995
9epoch198-step4776-valid_loss0.3471-val_metric0.8730.ckptLionVoxHRNet1.4MTS_73_6InstanceNorm3dPReLU0.761350.75848
10epoch133-val_loss0.52-val_metric0.56-step3216.ckptLuoziqianUNet3D1.1MTS_6_4BatchNorm3dPReLU0.768440.76320
11epoch314-val_loss0.54-val_metric0.54-step7560.ckptLuoziqianSegResNet1.2MTS_6_4GroupNormReLU0.755210.74647
12epoch114-val_loss0.55-val_metric0.53-Step2760.ckptLuoziqianUNet2E3D14.2MTS_6_4BatchNorm3dReLU0.737580.72966
\n

Ensemble Strategy (Partial Selection)

\n

We select the models with the best public leaderboard scores for the final submission.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ModelsEnsemble StrategyCertainty ThresholdPublic LBPrivate LBSelected
[1, 2, 3, 4, 7, 8, 12]Average0.230.790940.78641❌
[1, 2, 3, 4, 7, 8, 12]Average0.180.792130.78630❌
[1, 2, 3, 4, 7, 8, 12]Average0.150.793070.78457❌
[1, 2, 3, 4, 7, 8, 12]weighted0.150.792470.78417❌
[1, 2, 3, 4, 6, 12]Average0.150.787010.78389❌
[1, 2, 3, 4, 5, 6, 7]Average0.150.791040.78277❌
[1, 2, 3, 4, 7, 9, 12]Average0.150.793910.78283✅
[1, 2, 3, 4, 6, 7, 9, 10, 11, 12]Average0.150.793550.78381✅
\n

Post-processing (Cluster Removal)

\n

We used CC3D to convert the segmentation results into particle clusters and applied the following strategies for cluster selection:

\n\n

Methods that Did Not Yield Improvements:

\n\n

Code

\n

Inference Notebook
\nLion's Training Code
\nLuoziqian's Training Code

\n

Reference

\n

Gubins, Ilja, et al. \"SHREC 2020: Classification in cryo-electron tomograms.\" Computers & Graphics 91 (2020): 279-289.

", "messages": [ { "id": 3117241, "postDate": "2025-02-06T18:53:56.557Z", "content": "

Acknowledgments

\n

We sincerely appreciate Kaggle and the competition organizers for offering this invaluable opportunity. We also extend our gratitude to @hengck23 and @fnands for their significant contributions. fnands's notebook and hengck23's discussion provided a solid foundation for our approach. Finally, I am grateful to my teammate @luoziqian for the excellent collaboration.

\n

Summary

\n

Our approach is based on an ensemble of multiple lightweight segmentation models, with parameter sizes ranging from 873K to 14.2M. Following segmentation, we computed particle centroids using CC3D and filtered small clusters based on voxel count statistics.

\n

Overall Pipeline

\n

The overall pipeline is illustrated below:
\n

\n

Model Architecture

\n

We initiated our experiments using the MONAI UNet baseline model. Throughout our trials, we observed that models with a large number of parameters were prone to overfitting and often performed worse than lighter models. Consequently, we opted for lightweight architectures such as UNet3D, VoxResNet, VoxHRNet, SegResNet, DynUNet, and DenseVNet.

\n

However, while experimenting with VoxResNet, VoxHRNet, and DenseVNet, we encountered stability issues, including performance fluctuations and difficulties in convergence. Upon further analysis, we identified that MONAI UNet employs InstanceNorm3d and PReLU. By modifying the normalization and activation layers accordingly, we achieved more stable model performance.

\n

The model architecture of MONAI UNet is shown below:
\n\"\"

\n

The following figure compares the training performance of MONAI UNet with InstanceNorm3d and PReLU versus BatchNorm3d and ReLU across 5 experiments, demonstrating that the use of InstanceNorm3d and PReLU results in more stable training.

\n

\n

For the final ensemble, we selected models based on their public leaderboard scores:

\n\n

Training Strategy

\n

Our training configuration was designed to ensure stability and optimal performance. We found that the segmentation mask radius, loss function and data augmentation strategies played a crucial role in achieving reliable results.

\n

Segmentation Mask:

\n

We applied ground truth masks with a customized radius for each particle.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Particle TypesDefault RadiusLuoziqian's RadiusLion's Radius
apo-ferritin6060 x 0.580 x 0.4
beta-galactosidase9090 x 0.590 x 0.4
ribosome150150 x 0.5150 x 0.4
thyroglobulin130130 x 0.5120 x 0.4
virus-like-particle135135 x 0.5150 x 0.4
\n

Training Settings:

\n

Lion's models:

\n\n

Luoziqian's models

\n\n

Augmentations:

\n

Lion's models:

\n\n

Luoziqian's models

\n\n

Model Performance with 7 TTA Summary (Partial Selection)

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
NoModelDeveloperArchitectureParametersValid IDNormalizationActivationPublic LBPrivate LB
1epoch122-step2952-valid_loss0.3625-val_metric0.8367.ckptLionUNet3D1.1MTS_86_3InstanceNorm3dPReLU0.773790.76582
2epoch148-step3576-valid_loss1.1154-val_metric0.7722.ckptLionUNet3D1.1MTS_6_4InstanceNorm3dPReLU0.770210.76725
3epoch153-step3696-valid_loss0.3021-val_metric0.8900.ckptLionUNet3D1.6MTS_69_2InstanceNorm3dPReLU0.772050.76676
4epoch194-step4680-valid_loss1.0213-val_metric0.8788.ckptLionUNet3D1.1MTS_69_2InstanceNorm3dPReLU0.773900.76737
5epoch138-step3336-valid_loss0.3690-val_metric0.8476.ckptLionUNet3D1.1MTS_73_6InstanceNorm3dPReLU0.765430.76025
6epoch152-step3672-valid_loss0.4333-val_metric0.7929.ckptLionDenseVNet873KTS_6_6InstanceNorm3dPReLU0.765280.75417
7epoch195-step4704-valid_loss0.4258-val_metric0.7914.ckptLionVoxResNet7.0MTS_6_6InstanceNorm3dPReLU0.774570.76593
8epoch188-step4536-valid_loss0.4231-val_metric0.8659.ckptLionVoxHRNet1.4MTS_73_6InstanceNorm3dPReLU0.767380.75995
9epoch198-step4776-valid_loss0.3471-val_metric0.8730.ckptLionVoxHRNet1.4MTS_73_6InstanceNorm3dPReLU0.761350.75848
10epoch133-val_loss0.52-val_metric0.56-step3216.ckptLuoziqianUNet3D1.1MTS_6_4BatchNorm3dPReLU0.768440.76320
11epoch314-val_loss0.54-val_metric0.54-step7560.ckptLuoziqianSegResNet1.2MTS_6_4GroupNormReLU0.755210.74647
12epoch114-val_loss0.55-val_metric0.53-Step2760.ckptLuoziqianUNet2E3D14.2MTS_6_4BatchNorm3dReLU0.737580.72966
\n

Ensemble Strategy (Partial Selection)

\n

We select the models with the best public leaderboard scores for the final submission.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ModelsEnsemble StrategyCertainty ThresholdPublic LBPrivate LBSelected
[1, 2, 3, 4, 7, 8, 12]Average0.230.790940.78641❌
[1, 2, 3, 4, 7, 8, 12]Average0.180.792130.78630❌
[1, 2, 3, 4, 7, 8, 12]Average0.150.793070.78457❌
[1, 2, 3, 4, 7, 8, 12]weighted0.150.792470.78417❌
[1, 2, 3, 4, 6, 12]Average0.150.787010.78389❌
[1, 2, 3, 4, 5, 6, 7]Average0.150.791040.78277❌
[1, 2, 3, 4, 7, 9, 12]Average0.150.793910.78283✅
[1, 2, 3, 4, 6, 7, 9, 10, 11, 12]Average0.150.793550.78381✅
\n

Post-processing (Cluster Removal)

\n

We used CC3D to convert the segmentation results into particle clusters and applied the following strategies for cluster selection:

\n\n

Methods that Did Not Yield Improvements:

\n\n

Code

\n

Inference Notebook
\nLion's Training Code
\nLuoziqian's Training Code

\n

Reference

\n

Gubins, Ilja, et al. \"SHREC 2020: Classification in cryo-electron tomograms.\" Computers & Graphics 91 (2020): 279-289.

", "rawMarkdown": "## Acknowledgments\n\nWe sincerely appreciate Kaggle and the competition organizers for offering this invaluable opportunity. We also extend our gratitude to @hengck23 and @fnands for their significant contributions. [**fnands's notebook**](https://www.kaggle.com/code/fnands/baseline-unet-train-submit/notebook) and [**hengck23's discussion**](https://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/547350#3056648) provided a solid foundation for our approach. Finally, I am grateful to my teammate @luoziqian for the excellent collaboration.\n\n## Summary\n\nOur approach is based on an ensemble of multiple lightweight segmentation models, with parameter sizes ranging from 873K to 14.2M. Following segmentation, we computed particle centroids using CC3D and filtered small clusters based on voxel count statistics.\n\n## Overall Pipeline\n\nThe overall pipeline is illustrated below:\n\n\n\n## Model Architecture\n\nWe initiated our experiments using the MONAI UNet baseline model. Throughout our trials, we observed that models with a large number of parameters were prone to overfitting and often performed worse than lighter models. Consequently, we opted for lightweight architectures such as UNet3D, VoxResNet, VoxHRNet, SegResNet, DynUNet, and DenseVNet.\n\nHowever, while experimenting with VoxResNet, VoxHRNet, and DenseVNet, we encountered stability issues, including performance fluctuations and difficulties in convergence. Upon further analysis, we identified that MONAI UNet employs `InstanceNorm3d` and `PReLU`. By modifying the normalization and activation layers accordingly, we achieved more stable model performance.\n\nThe model architecture of MONAI UNet is shown below:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5483160%2F35bb85a0777083a30ffc41461aa2c154%2Fmonai_unet.png?generation=1738867977514788&alt=media)\n\nThe following figure compares the training performance of MONAI UNet with InstanceNorm3d and PReLU versus BatchNorm3d and ReLU across 5 experiments, demonstrating that the use of InstanceNorm3d and PReLU results in more stable training.\n\n\n\nFor the final ensemble, we selected models based on their public leaderboard scores:\n\n- UNet3D\n- VoxResNet\n- VoxHRNet\n- SegResNet\n- DenseVNet\n- UNet2E3D\n\n## Training Strategy\n\nOur training configuration was designed to ensure stability and optimal performance. We found that the **segmentation mask radius**, **loss function** and **data augmentation strategies** played a crucial role in achieving reliable results.\n\n#### Segmentation Mask:\n\nWe applied ground truth masks with a customized radius for each particle.\n\n| **Particle Types** | **Default Radius** |**Luoziqian's Radius** | **Lion's Radius** |\n| ------------------- | ------------------ | ----------------------- | ---------------- |\n| apo-ferritin | 60 | 60 x 0.5 | 80 x 0.4 | \n| beta-galactosidase | 90 | 90 x 0.5 | 90 x 0.4 |\n| ribosome | 150 | 150 x 0.5 | 150 x 0.4 |\n| thyroglobulin | 130 | 130 x 0.5 |120 x 0.4 |\n| virus-like-particle | 135 | 135 x 0.5 | 150 x 0.4 | \n\n#### Training Settings:\n\n**Lion's** models:\n- patch size `[128, 256, 256]` or `[128, 384, 384]`\n- `200` epochs\n- 7 classes\n- Optimizer: AdamW with a learning rate of `0.001` (no learning rate scheduler)\n- Batch size: `2` or `4`\n- Drop path rate: `0.1` or `0.3`\n- Model selection based on the F-beta metric, save top-5 best model\n- Loss functions:\n - Tversky Loss\n - Cross-Entropy Loss with weights `[1.0, 1.0, 0.0, 2.0, 1.0, 2.0, 1.0]`\n\n**Luoziqian's** models\n- patch size `[128, 200, 200]` or `[128, 256, 256]`\n- `100` or `300` epochs\n- 6 classes\n- Early stopping with `patience=20`\n- Batch size: `1` or `2`\n- Model selection is based on the evaluation loss. The last 100 models are saved and their F-beta scores are verified\n- Loss functions:\n - Dice Loss\n - Tversky Loss\n - Cross-Entropy Loss, scaled to 0.05 or 0.1\n\n\n#### Augmentations:\n\n**Lion's** models:\n\n- RandCropByLabelClassesd with ratios `[1, 1, 1, 1, 2, 1, 2]` for balanced sampling\n- RandRotate90d\n- RandFlipd\n- RandAffined\n- RandGaussianNoised\n\n**Luoziqian's** models\n- RandCropByLabelClassesd\n- RandRotate90d\n- RandFlipd\n- RandShiftIntensityd\n\n\n#### Model Performance with 7 TTA Summary (Partial Selection)\n\n| **No** | **Model** | **Developer** | **Architecture** | **Parameters** | **Valid ID** | **Normalization** | **Activation** | **Public LB** | **Private LB** |\n| ------ | -------------------------------------------------------- | ----------------| ---------------- | -------------- | ----------------- | ----------------- | -------------- | ------------- | -------------- |\n| 1 | epoch122-step2952-valid_loss0.3625-val_metric0.8367.ckpt | Lion | UNet3D | 1.1M | TS_86_3 | InstanceNorm3d | PReLU | 0.77379 | 0.76582 |\n| 2 | epoch148-step3576-valid_loss1.1154-val_metric0.7722.ckpt | Lion | UNet3D | 1.1M | TS_6_4 | InstanceNorm3d | PReLU | 0.77021 | 0.76725 |\n| 3 | epoch153-step3696-valid_loss0.3021-val_metric0.8900.ckpt | Lion | UNet3D | 1.6M | TS_69_2 | InstanceNorm3d | PReLU | 0.77205 | 0.76676 |\n| 4 | epoch194-step4680-valid_loss1.0213-val_metric0.8788.ckpt | Lion | UNet3D | 1.1M | TS_69_2 | InstanceNorm3d | PReLU | 0.77390 | 0.76737 |\n| 5 | epoch138-step3336-valid_loss0.3690-val_metric0.8476.ckpt | Lion | UNet3D | 1.1M | TS_73_6 | InstanceNorm3d | PReLU | 0.76543 | 0.76025 |\n| 6 | epoch152-step3672-valid_loss0.4333-val_metric0.7929.ckpt | Lion | DenseVNet | 873K | TS_6_6 | InstanceNorm3d | PReLU | 0.76528 | 0.75417 |\n| 7 | epoch195-step4704-valid_loss0.4258-val_metric0.7914.ckpt | Lion | VoxResNet | 7.0M | TS_6_6 | InstanceNorm3d | PReLU | 0.77457 | 0.76593 |\n| 8 | epoch188-step4536-valid_loss0.4231-val_metric0.8659.ckpt | Lion | VoxHRNet | 1.4M | TS_73_6 | InstanceNorm3d | PReLU | 0.76738 | 0.75995 |\n| 9 | epoch198-step4776-valid_loss0.3471-val_metric0.8730.ckpt | Lion | VoxHRNet | 1.4M | TS_73_6 | InstanceNorm3d | PReLU | 0.76135 | 0.75848 |\n| 10 | epoch133-val_loss0.52-val_metric0.56-step3216.ckpt | Luoziqian | UNet3D | 1.1M | TS_6_4 | BatchNorm3d | PReLU | 0.76844 | 0.76320 |\n| 11 | epoch314-val_loss0.54-val_metric0.54-step7560.ckpt | Luoziqian | SegResNet | 1.2M | TS_6_4 | GroupNorm | ReLU | 0.75521 | 0.74647 |\n| 12 | epoch114-val_loss0.55-val_metric0.53-Step2760.ckpt | Luoziqian | UNet2E3D | 14.2M | TS_6_4 | BatchNorm3d | ReLU | 0.73758 | 0.72966 |\n\n\n## Ensemble Strategy (Partial Selection)\nWe select the models with the best public leaderboard scores for the final submission.\n\n| **Models** | **Ensemble Strategy** | **Certainty Threshold** | **Public LB** | **Private LB** | **Selected** |\n| --------------------------------- | --------------------- | --------------------- | ------------- | -------------- | ----------------- |\n| [1, 2, 3, 4, 7, 8, 12] | Average | 0.23 | 0.79094 | 0.78641 | ❌ |\n| [1, 2, 3, 4, 7, 8, 12] | Average | 0.18 | 0.79213 | 0.78630 | ❌ |\n| [1, 2, 3, 4, 7, 8, 12] | Average | 0.15 | 0.79307 | 0.78457 | ❌ |\n| [1, 2, 3, 4, 7, 8, 12] | weighted | 0.15 | 0.79247 | 0.78417 | ❌ |\n| [1, 2, 3, 4, 6, 12] | Average | 0.15 | 0.78701 | 0.78389 | ❌ |\n| [1, 2, 3, 4, 5, 6, 7] | Average | 0.15 | 0.79104 | 0.78277 | ❌ |\n| [1, 2, 3, 4, 7, 9, 12] | Average | 0.15 | 0.79391 | 0.78283 | ✅ |\n| [1, 2, 3, 4, 6, 7, 9, 10, 11, 12] | Average | 0.15 | 0.79355 | 0.78381 | ✅ |\n\n## Post-processing (Cluster Removal)\n\nWe used `CC3D` to convert the segmentation results into particle clusters and applied the following strategies for cluster selection:\n\n- Threshold-based clustering (the threshold is set to 0.15 for all classses to improve recall)\n- Cluster size filtering\n\n## Methods that Did Not Yield Improvements:\n\n- Synthetic data\n- Second-stage classification\n- Heavy-weight models (transformer-based models)\n\n## Code\n\n[Inference Notebook](https://www.kaggle.com/code/luoziqian/7-model-6tta-t4x2-patch-norm-v2?scriptVersionId=220452693)\n[Lion's Training Code](https://github.com/GWwangshuo/Kaggle-2024-CZII-Pub)\n[Luoziqian's Training Code](https://github.com/luoziqianX/CZII-CryoET-Object-Identification-2nd-luoziqian)\n\n## Reference\n\nGubins, Ilja, et al. \"SHREC 2020: Classification in cryo-electron tomograms.\" *Computers & Graphics* 91 (2020): 279-289.", "votes": 44 }, { "id": 3117309, "postDate": "2025-02-06T20:36:24.863Z", "content": "

Wow, you have very strong single models. Focus on stabilizing the training might be the trick(s) we missed. Congratulations on the gold and the prize!

", "rawMarkdown": "Wow, you have very strong single models. Focus on stabilizing the training might be the trick(s) we missed. Congratulations on the gold and the prize!", "votes": 2 }, { "id": 3120885, "postDate": "2025-02-11T03:15:04.657Z", "content": "

Congratulations !
\nI've realized that there are a lot of room for improvement even if segmentation + cc3d approach and lack of my skills to design experiments.
\nIn Luoziqian's Training Code, I cannot find output comp metrics in training code. Did you choose best model by other way ?

", "rawMarkdown": "Congratulations !\nI've realized that there are a lot of room for improvement even if segmentation + cc3d approach and lack of my skills to design experiments.\nIn Luoziqian's Training Code, I cannot find output comp metrics in training code. Did you choose best model by other way ?", "replies": [ { "id": 3120907, "postDate": "2025-02-11T03:47:15.863Z", "content": "

You can change the path_dir in compute-cv-7TTA.ipynb and run it to calculate the F-beta4 score of ckpts after training. I think the Tversky Loss for each class is enough for me to initially rule out less successful experiments. Since measuring F-beta4 during training would significantly increase the experiment time and my limited computing resources, I did not calculate it during training.

", "rawMarkdown": "You can change the path_dir in [compute-cv-7TTA.ipynb](https://github.com/luoziqianX/CZII-CryoET-Object-Identification-2nd-luoziqian/blob/main/compute-cv-7TTA.ipynb) and run it to calculate the F-beta4 score of ckpts after training. I think the Tversky Loss for each class is enough for me to initially rule out less successful experiments. Since measuring F-beta4 during training would significantly increase the experiment time and my limited computing resources, I did not calculate it during training.", "votes": 4 } ] }, { "id": 3119199, "postDate": "2025-02-09T02:21:27.833Z", "content": "

Congratulations! @sjtuwangshuo .

\n

Which sample do you use to validate?

", "rawMarkdown": "Congratulations! @sjtuwangshuo .\n\nWhich sample do you use to validate? ", "replies": [ { "id": 3119239, "postDate": "2025-02-09T04:05:20.560Z", "content": "

Initial Strategy:

\n\n

Refinement:
\nSubsequently, Luoziqian observed that models validated on ['TS_5_4', 'TS_6_4', 'TS_69_2'] tend to achieve better public leaderboard performance, though not consistently. Based on this observation, I evaluate model performance using 7× TTA on these specific folds and submit the model with the highest F-beta score.

\n

For details on the validation samples of the selected models used in the final ensemble, please refer to the \"Model Performance with 7× TTA Summary (Partial Selection)\" section of this post.

", "rawMarkdown": "Initial Strategy:\n\n- Split tomograms into seven folds (e.g., TS_5_4, TS_6_4, TS_69_2, etc.).\n- Train each fold and save the models with the top five highest F-beta scores.\n- Evaluate each of the top five models using 7× Test-Time Augmentation (TTA) on the same validation sample and compute the F-beta score.\n- Select the model with the highest F-beta score after 7× TTA for submission.\n\nRefinement:\nSubsequently, `Luoziqian` observed that models validated on ['TS_5_4', 'TS_6_4', 'TS_69_2'] tend to achieve better public leaderboard performance, though not consistently. Based on this observation, I evaluate model performance using 7× TTA on these specific folds and submit the model with the highest F-beta score.\n\nFor details on the validation samples of the selected models used in the final ensemble, please refer to the `\"Model Performance with 7× TTA Summary (Partial Selection)\"` section of this post." } ] }, { "id": 3118525, "postDate": "2025-02-08T06:52:59.977Z", "content": "

Congratulations! thank you for sharing your solution

\n

I have a question :why experiment no.2 and no.10/11/12 have such a huge different valid score(0.7x vs 0.5x,but lb very close)？

", "rawMarkdown": "Congratulations! thank you for sharing your solution\n\nI have a question :why experiment no.2 and no.10/11/12 have such a huge different valid score(0.7x vs 0.5x,but lb very close)？\n\n", "replies": [ { "id": 3118546, "postDate": "2025-02-08T07:30:02.710Z", "content": "

Could you please clarify what is meant by “0.7x vs. 0.5x, but lb very close”? Please note that the experiments presented thus far represent only a subset of the complete experimental results. Additionally, my colleague, Luoziqian, has developed several models based on the MONAI UNet architecture that achieve a public leaderboard score exceeding 0.760; however, these models do not exhibit improved performance when ensembled.

", "rawMarkdown": "Could you please clarify what is meant by “0.7x vs. 0.5x, but lb very close”? Please note that the experiments presented thus far represent only a subset of the complete experimental results. Additionally, my colleague, Luoziqian, has developed several models based on the MONAI UNet architecture that achieve a public leaderboard score exceeding 0.760; however, these models do not exhibit improved performance when ensembled.", "replies": [ { "id": 3118558, "postDate": "2025-02-08T07:42:32.960Z", "content": "

噢是这样的，我看你实验的model列里面有val_metric这一项，no.2和no.10,no.11,no.12都是用TS_6_4作为验证集，好像这几个的验证集指标差异挺大的，想问问看是不是有什么特殊处理我错过了

", "rawMarkdown": "噢是这样的，我看你实验的model列里面有val_metric这一项，no.2和no.10,no.11,no.12都是用TS_6_4作为验证集，好像这几个的验证集指标差异挺大的，想问问看是不是有什么特殊处理我错过了" }, { "id": 3118572, "postDate": "2025-02-08T08:01:55.547Z", "content": "

Our experiments indicated that several factors significantly affect performance, including spatial size, mask radius, data augmentation strategies, loss functions, normalization layers, and activation layers.

\n

In the comparison between Experiment No.2 and Experiment No.10, the following differences were observed matter:

\n\n

Furthermore, when comparing Experiment No.2 with Experiments No.11 and No.12, we found that the MONAI UNet outperformed both SegResNet and UNet2E3D. This improved performance can be attributed to several architectural features of the MONAI UNet:

\n\n

For both SegResNet and UNet2E3D, modifications were implemented based on our experimental investigations to facilitate training convergence. Specifically, we:

\n\n

Despite these modifications, neither SegResNet nor UNet2E3D outperformed the MONAI UNet.

", "rawMarkdown": "Our experiments indicated that several factors significantly affect performance, including spatial size, mask radius, data augmentation strategies, loss functions, normalization layers, and activation layers.\n\nIn the comparison between Experiment `No.2` and Experiment `No.10`, the following differences were observed matter:\n- Patch spatial size\n- Mask radius\n- Data augmentation strategies\n- Loss functions\n- Number of classes\n\nFurthermore, when comparing Experiment `No.2` with Experiments `No.11` and `No.12`, we found that the MONAI UNet outperformed both SegResNet and UNet2E3D. This improved performance can be attributed to several architectural features of the MONAI UNet:\n\n- The use of `InstanceNorm3d` in conjunction with `PReLU activation`.\n- An encoder downsampling factor of `1/4` as opposed to `1/8`, `1/16`, or `1/32`.\n- The implementation of a deconvolution layer using `ConvTransposed3D` instead of conventional upsampling.\n- The incorporation of `residual blocks`, which enhances parameter efficiency and reduce t the likelihood of overfitting.\n\nFor both SegResNet and UNet2E3D, modifications were implemented based on our experimental investigations to facilitate training convergence. Specifically, we:\n- Revised the normalization and activation layers.\n- Replaced the decoder with a deconvolution layer.\n- Reduced the number of decoder channels to create a more lightweight model.\n\nDespite these modifications, neither SegResNet nor UNet2E3D outperformed the MONAI UNet.", "votes": 1 } ] } ] }, { "id": 3117336, "postDate": "2025-02-06T21:33:08.730Z", "content": "

Congrats! Glad the notebook helped you get started!

", "rawMarkdown": "Congrats! Glad the notebook helped you get started!" }, { "id": 3118220, "postDate": "2025-02-07T18:57:19.297Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 3117309, "author_name": "Sinan Calisir", "author_url": "", "post_date": "2025-02-06T20:36:24.863000", "content": "

Wow, you have very strong single models. Focus on stabilizing the training might be the trick(s) we missed. Congratulations on the gold and the prize!

", "votes": 2, "replies": [] }, { "id": 3120885, "author_name": "Aurora_blue", "author_url": "", "post_date": "2025-02-11T03:15:04.657000", "content": "

Congratulations !
\nI've realized that there are a lot of room for improvement even if segmentation + cc3d approach and lack of my skills to design experiments.
\nIn Luoziqian's Training Code, I cannot find output comp metrics in training code. Did you choose best model by other way ?

", "votes": 0, "replies": [ { "id": 3120907, "author_name": "LuoZiqian", "author_url": "", "post_date": "2025-02-11T03:47:15.863000", "content": "

You can change the path_dir in compute-cv-7TTA.ipynb and run it to calculate the F-beta4 score of ckpts after training. I think the Tversky Loss for each class is enough for me to initially rule out less successful experiments. Since measuring F-beta4 during training would significantly increase the experiment time and my limited computing resources, I did not calculate it during training.

", "votes": 4, "replies": [] } ] }, { "id": 3119199, "author_name": "ForcewithMe", "author_url": "", "post_date": "2025-02-09T02:21:27.833000", "content": "

Congratulations! @sjtuwangshuo .

\n

Which sample do you use to validate?

", "votes": 0, "replies": [ { "id": 3119239, "author_name": "AnnieGo", "author_url": "", "post_date": "2025-02-09T04:05:20.560000", "content": "

Initial Strategy:

\n\n

Refinement:
\nSubsequently, Luoziqian observed that models validated on ['TS_5_4', 'TS_6_4', 'TS_69_2'] tend to achieve better public leaderboard performance, though not consistently. Based on this observation, I evaluate model performance using 7× TTA on these specific folds and submit the model with the highest F-beta score.

\n

For details on the validation samples of the selected models used in the final ensemble, please refer to the \"Model Performance with 7× TTA Summary (Partial Selection)\" section of this post.

", "votes": 0, "replies": [] } ] }, { "id": 3118525, "author_name": "FatLiuyun", "author_url": "", "post_date": "2025-02-08T06:52:59.977000", "content": "

Congratulations! thank you for sharing your solution

\n

I have a question :why experiment no.2 and no.10/11/12 have such a huge different valid score(0.7x vs 0.5x,but lb very close)？

", "votes": 0, "replies": [ { "id": 3118546, "author_name": "AnnieGo", "author_url": "", "post_date": "2025-02-08T07:30:02.710000", "content": "

Could you please clarify what is meant by “0.7x vs. 0.5x, but lb very close”? Please note that the experiments presented thus far represent only a subset of the complete experimental results. Additionally, my colleague, Luoziqian, has developed several models based on the MONAI UNet architecture that achieve a public leaderboard score exceeding 0.760; however, these models do not exhibit improved performance when ensembled.

", "votes": 0, "replies": [ { "id": 3118558, "author_name": "FatLiuyun", "author_url": "", "post_date": "2025-02-08T07:42:32.960000", "content": "

噢是这样的，我看你实验的model列里面有val_metric这一项，no.2和no.10,no.11,no.12都是用TS_6_4作为验证集，好像这几个的验证集指标差异挺大的，想问问看是不是有什么特殊处理我错过了

", "votes": 0, "replies": [] }, { "id": 3118572, "author_name": "AnnieGo", "author_url": "", "post_date": "2025-02-08T08:01:55.547000", "content": "

Our experiments indicated that several factors significantly affect performance, including spatial size, mask radius, data augmentation strategies, loss functions, normalization layers, and activation layers.

\n

In the comparison between Experiment No.2 and Experiment No.10, the following differences were observed matter:

\n\n

Furthermore, when comparing Experiment No.2 with Experiments No.11 and No.12, we found that the MONAI UNet outperformed both SegResNet and UNet2E3D. This improved performance can be attributed to several architectural features of the MONAI UNet:

\n\n

For both SegResNet and UNet2E3D, modifications were implemented based on our experimental investigations to facilitate training convergence. Specifically, we:

\n\n

Despite these modifications, neither SegResNet nor UNet2E3D outperformed the MONAI UNet.

", "votes": 1, "replies": [] } ] } ] }, { "id": 3117336, "author_name": "fnands", "author_url": "", "post_date": "2025-02-06T21:33:08.730000", "content": "

Congrats! Glad the notebook helped you get started!

", "votes": 0, "replies": [] }, { "id": 3118220, "author_name": "", "author_url": "", "post_date": "2025-02-07T18:57:19.297000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "3117241": "## Acknowledgments\n\nWe sincerely appreciate Kaggle and the competition organizers for offering this invaluable opportunity. We also extend our gratitude to @hengck23 and @fnands for their significant contributions. [**fnands's notebook**](https://www.kaggle.com/code/fnands/baseline-unet-train-submit/notebook) and [**hengck23's discussion**](https://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/547350#3056648) provided a solid foundation for our approach. Finally, I am grateful to my teammate @luoziqian for the excellent collaboration.\n\n## Summary\n\nOur approach is based on an ensemble of multiple lightweight segmentation models, with parameter sizes ranging from 873K to 14.2M. Following segmentation, we computed particle centroids using CC3D and filtered small clusters based on voxel count statistics.\n\n## Overall Pipeline\n\nThe overall pipeline is illustrated below:\n\n\n\n## Model Architecture\n\nWe initiated our experiments using the MONAI UNet baseline model. Throughout our trials, we observed that models with a large number of parameters were prone to overfitting and often performed worse than lighter models. Consequently, we opted for lightweight architectures such as UNet3D, VoxResNet, VoxHRNet, SegResNet, DynUNet, and DenseVNet.\n\nHowever, while experimenting with VoxResNet, VoxHRNet, and DenseVNet, we encountered stability issues, including performance fluctuations and difficulties in convergence. Upon further analysis, we identified that MONAI UNet employs `InstanceNorm3d` and `PReLU`. By modifying the normalization and activation layers accordingly, we achieved more stable model performance.\n\nThe model architecture of MONAI UNet is shown below:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5483160%2F35bb85a0777083a30ffc41461aa2c154%2Fmonai_unet.png?generation=1738867977514788&alt=media)\n\nThe following figure compares the training performance of MONAI UNet with InstanceNorm3d and PReLU versus BatchNorm3d and ReLU across 5 experiments, demonstrating that the use of InstanceNorm3d and PReLU results in more stable training.\n\n\n\nFor the final ensemble, we selected models based on their public leaderboard scores:\n\n- UNet3D\n- VoxResNet\n- VoxHRNet\n- SegResNet\n- DenseVNet\n- UNet2E3D\n\n## Training Strategy\n\nOur training configuration was designed to ensure stability and optimal performance. We found that the **segmentation mask radius**, **loss function** and **data augmentation strategies** played a crucial role in achieving reliable results.\n\n#### Segmentation Mask:\n\nWe applied ground truth masks with a customized radius for each particle.\n\n| **Particle Types** | **Default Radius** |**Luoziqian's Radius** | **Lion's Radius** |\n| ------------------- | ------------------ | ----------------------- | ---------------- |\n| apo-ferritin | 60 | 60 x 0.5 | 80 x 0.4 | \n| beta-galactosidase | 90 | 90 x 0.5 | 90 x 0.4 |\n| ribosome | 150 | 150 x 0.5 | 150 x 0.4 |\n| thyroglobulin | 130 | 130 x 0.5 |120 x 0.4 |\n| virus-like-particle | 135 | 135 x 0.5 | 150 x 0.4 | \n\n#### Training Settings:\n\n**Lion's** models:\n- patch size `[128, 256, 256]` or `[128, 384, 384]`\n- `200` epochs\n- 7 classes\n- Optimizer: AdamW with a learning rate of `0.001` (no learning rate scheduler)\n- Batch size: `2` or `4`\n- Drop path rate: `0.1` or `0.3`\n- Model selection based on the F-beta metric, save top-5 best model\n- Loss functions:\n - Tversky Loss\n - Cross-Entropy Loss with weights `[1.0, 1.0, 0.0, 2.0, 1.0, 2.0, 1.0]`\n\n**Luoziqian's** models\n- patch size `[128, 200, 200]` or `[128, 256, 256]`\n- `100` or `300` epochs\n- 6 classes\n- Early stopping with `patience=20`\n- Batch size: `1` or `2`\n- Model selection is based on the evaluation loss. The last 100 models are saved and their F-beta scores are verified\n- Loss functions:\n - Dice Loss\n - Tversky Loss\n - Cross-Entropy Loss, scaled to 0.05 or 0.1\n\n\n#### Augmentations:\n\n**Lion's** models:\n\n- RandCropByLabelClassesd with ratios `[1, 1, 1, 1, 2, 1, 2]` for balanced sampling\n- RandRotate90d\n- RandFlipd\n- RandAffined\n- RandGaussianNoised\n\n**Luoziqian's** models\n- RandCropByLabelClassesd\n- RandRotate90d\n- RandFlipd\n- RandShiftIntensityd\n\n\n#### Model Performance with 7 TTA Summary (Partial Selection)\n\n| **No** | **Model** | **Developer** | **Architecture** | **Parameters** | **Valid ID** | **Normalization** | **Activation** | **Public LB** | **Private LB** |\n| ------ | -------------------------------------------------------- | ----------------| ---------------- | -------------- | ----------------- | ----------------- | -------------- | ------------- | -------------- |\n| 1 | epoch122-step2952-valid_loss0.3625-val_metric0.8367.ckpt | Lion | UNet3D | 1.1M | TS_86_3 | InstanceNorm3d | PReLU | 0.77379 | 0.76582 |\n| 2 | epoch148-step3576-valid_loss1.1154-val_metric0.7722.ckpt | Lion | UNet3D | 1.1M | TS_6_4 | InstanceNorm3d | PReLU | 0.77021 | 0.76725 |\n| 3 | epoch153-step3696-valid_loss0.3021-val_metric0.8900.ckpt | Lion | UNet3D | 1.6M | TS_69_2 | InstanceNorm3d | PReLU | 0.77205 | 0.76676 |\n| 4 | epoch194-step4680-valid_loss1.0213-val_metric0.8788.ckpt | Lion | UNet3D | 1.1M | TS_69_2 | InstanceNorm3d | PReLU | 0.77390 | 0.76737 |\n| 5 | epoch138-step3336-valid_loss0.3690-val_metric0.8476.ckpt | Lion | UNet3D | 1.1M | TS_73_6 | InstanceNorm3d | PReLU | 0.76543 | 0.76025 |\n| 6 | epoch152-step3672-valid_loss0.4333-val_metric0.7929.ckpt | Lion | DenseVNet | 873K | TS_6_6 | InstanceNorm3d | PReLU | 0.76528 | 0.75417 |\n| 7 | epoch195-step4704-valid_loss0.4258-val_metric0.7914.ckpt | Lion | VoxResNet | 7.0M | TS_6_6 | InstanceNorm3d | PReLU | 0.77457 | 0.76593 |\n| 8 | epoch188-step4536-valid_loss0.4231-val_metric0.8659.ckpt | Lion | VoxHRNet | 1.4M | TS_73_6 | InstanceNorm3d | PReLU | 0.76738 | 0.75995 |\n| 9 | epoch198-step4776-valid_loss0.3471-val_metric0.8730.ckpt | Lion | VoxHRNet | 1.4M | TS_73_6 | InstanceNorm3d | PReLU | 0.76135 | 0.75848 |\n| 10 | epoch133-val_loss0.52-val_metric0.56-step3216.ckpt | Luoziqian | UNet3D | 1.1M | TS_6_4 | BatchNorm3d | PReLU | 0.76844 | 0.76320 |\n| 11 | epoch314-val_loss0.54-val_metric0.54-step7560.ckpt | Luoziqian | SegResNet | 1.2M | TS_6_4 | GroupNorm | ReLU | 0.75521 | 0.74647 |\n| 12 | epoch114-val_loss0.55-val_metric0.53-Step2760.ckpt | Luoziqian | UNet2E3D | 14.2M | TS_6_4 | BatchNorm3d | ReLU | 0.73758 | 0.72966 |\n\n\n## Ensemble Strategy (Partial Selection)\nWe select the models with the best public leaderboard scores for the final submission.\n\n| **Models** | **Ensemble Strategy** | **Certainty Threshold** | **Public LB** | **Private LB** | **Selected** |\n| --------------------------------- | --------------------- | --------------------- | ------------- | -------------- | ----------------- |\n| [1, 2, 3, 4, 7, 8, 12] | Average | 0.23 | 0.79094 | 0.78641 | ❌ |\n| [1, 2, 3, 4, 7, 8, 12] | Average | 0.18 | 0.79213 | 0.78630 | ❌ |\n| [1, 2, 3, 4, 7, 8, 12] | Average | 0.15 | 0.79307 | 0.78457 | ❌ |\n| [1, 2, 3, 4, 7, 8, 12] | weighted | 0.15 | 0.79247 | 0.78417 | ❌ |\n| [1, 2, 3, 4, 6, 12] | Average | 0.15 | 0.78701 | 0.78389 | ❌ |\n| [1, 2, 3, 4, 5, 6, 7] | Average | 0.15 | 0.79104 | 0.78277 | ❌ |\n| [1, 2, 3, 4, 7, 9, 12] | Average | 0.15 | 0.79391 | 0.78283 | ✅ |\n| [1, 2, 3, 4, 6, 7, 9, 10, 11, 12] | Average | 0.15 | 0.79355 | 0.78381 | ✅ |\n\n## Post-processing (Cluster Removal)\n\nWe used `CC3D` to convert the segmentation results into particle clusters and applied the following strategies for cluster selection:\n\n- Threshold-based clustering (the threshold is set to 0.15 for all classses to improve recall)\n- Cluster size filtering\n\n## Methods that Did Not Yield Improvements:\n\n- Synthetic data\n- Second-stage classification\n- Heavy-weight models (transformer-based models)\n\n## Code\n\n[Inference Notebook](https://www.kaggle.com/code/luoziqian/7-model-6tta-t4x2-patch-norm-v2?scriptVersionId=220452693)\n[Lion's Training Code](https://github.com/GWwangshuo/Kaggle-2024-CZII-Pub)\n[Luoziqian's Training Code](https://github.com/luoziqianX/CZII-CryoET-Object-Identification-2nd-luoziqian)\n\n## Reference\n\nGubins, Ilja, et al. \"SHREC 2020: Classification in cryo-electron tomograms.\" *Computers & Graphics* 91 (2020): 279-289.", "3117309": "Wow, you have very strong single models. Focus on stabilizing the training might be the trick(s) we missed. Congratulations on the gold and the prize!", "3120885": "Congratulations !\nI've realized that there are a lot of room for improvement even if segmentation + cc3d approach and lack of my skills to design experiments.\nIn Luoziqian's Training Code, I cannot find output comp metrics in training code. Did you choose best model by other way ?", "3119199": "Congratulations! @sjtuwangshuo .\n\nWhich sample do you use to validate? ", "3118525": "Congratulations! thank you for sharing your solution\n\nI have a question :why experiment no.2 and no.10/11/12 have such a huge different valid score(0.7x vs 0.5x,but lb very close)？\n\n", "3117336": "Congrats! Glad the notebook helped you get started!", "3118220": "" } }