![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10254700%2Fd2d3ba10849048534fcc91ff265ac800%2F3d_heatmap.png?generation=1749212341383311&alt=media\")
Many thanks to the competition host and Kaggle for another engaging challenge - and big congratulations to all the participants!
\nThank you to every one who actively shared insights and notebooks to the community, especially @brendanartley for the external dataset.
As always, I had a great time learning throughout the competition and was fortunate enough to finish in the gold zone. I'm really happy to share a few thoughts here.
\n3D Resnext50 (2xTTA), 3D Densenet121 (2xTTA), 3D X3D-M (2xTTA), 2D MaxViT (2xTTA) and 2D CoaT (1xTTA)(224, 448, 448) while 2D use (3, 896, 896) patch size to incorporate enough global context.stride=16 and sigma=200A for uncertainty modelingWhat not work
\nA simple 4-StratifiedKfold by number of motors was conducted on train dataset. Validation samples are augmented by Rotate90+Flip (16 posible combinations) to increase the number of samples, making validation metrics more stable and trustworthy to track.
\nExperiment progress was tracked using mainly F2 score and additionally Partial Average Precision, cutoff at a specific Recall, e.g > 0.9. The second is not a binary metric so more stable and less sentitive to threshold, strong correlate with F2. Single-fold local CV was the main factor to select models and hyperparameters.
\nTrain on all data, validate on LB: However, when my pipeline are bug-free and going better (when CV > 0.96 and LB > 0.8), CV is just saturated and not correlate to LB well. So at the later stage, CV is just used to guide a reasonable value of hyperparameter (mainly learning rate). In last week, all models were trained on all dataset without local validation and are submitted to LB.
\nI resample tomogram to a fixed voxel spacing of 16.0, use sliding windows (patching) technique, split tomogram to 224x448x448 (possibly overlap) patches as input to models to obtain 16x downscaled prediction heatmap of shape 14x28x28.
\nThe labeling coordinate is a little bit noisy/uncertain, also the tolerance radius is high, so using high heatmap resolution seem to be redundant, significantly increase the computation. Radius of 1000A == 62.5 voxels in input volume == 4 voxels in prediction heatmap. I've experimented with prediction heatmap of 8x and 32x downscaled heatmap and found that 8x ~ 16x >> 32x. To the end, I use 16x downscaled heatmap for both 3D and 2D models.
Optimal voxel spacing is ~16.0
\nFor this task, resampling to a suitable voxel spacing is important. Using small voxel spacing keep the very small details but computation intensive due to high resolution, in contrast large voxel spacing could erase out all the textures needed for detection. Experiment results confirm that very small detail/textures are matter:
The mean original voxel spacing in LB/PB is 13.1 ???
\nHaha, just my stupid guess work and there're no way I could confirm. I trained model with target_spacing=16.0 and submit to LB with different of assumtions about the original spacing. 13.1 brings the highest LB. Given that test samples have higher resolution (~400x1400x1050 in average) and 44.7% of train samples have voxel_spacing=13.1, I safely assume that the average test voxel spacing is ~13.1, and resample to 16.0 for all experiments
\n\nI tried to quickly train a voxel spacing prediction model on the last day, but failed, metric is about
\nMAE=5.1, can't predict any useful thing :D
Each sample is dominated by negative voxels, so I try to control the number of positive/negative patches ratio in the training progress. For 3D approach, the positive/negative samples ratio is 20/1.
\nFor each epoch, all positive samples are ensured to be augmented with all 16 transformations composed of Rotate90+Flip, go after a \"heavy\" MONAI augmentation pipeline. Keypoints passed to MONAI dictionary transform as well so that we get the Affine-transformed keypoints to render the heatmap with larger stride (16 in this case). I really like MONAI's Lazy Resampling, which make augmentation and dataloader become fast and efficient.
\nVarious MONAI 3D augmentations were used:
RandZoom, RandAffinedRandShiftIntensityd, RandStdShiftIntensityd, RandScaleIntensityFixedMeand, RandScaleIntensityd, RandAdjustContrastd, RandHistogramShiftdRandSimulateLowResolutiondRandCoarseDropoutKeepKeypointsMixup and Cutmix were tried but bring no clear improvement so be ignored in latter experiments.
\n monai transforms T\n monai_custom CT\n\nT.Compose(\n [\n \n T.Spacingd(target_spacing=, p=),\n CT.CustomCropBySlicesd(),\n \n T.RandZoomd(min_zoom=(, , ), max_zoom=(, , ), p=),\n T.OneOf(\n [\n \n T.RandAffined(\n rotate_range=((, ), (, ), (, )),\n scale_range=((-, ), (-, ), (-, )),\n p=,\n ),\n \n T.RandAffined(\n rotate_range=((-, ), (-, ), (, )),\n shear_range=((-, ), (-, ), (-, )),\n scale_range=((-, ), (-, ), (-, )),\n p=,\n ),\n ],\n p=,\n ),\n \n T.OneOf(\n [\n T.Compose(\n [\n \n T.OneOf(\n [\n T.RandShiftIntensityd(offset=(-, ), p=),\n T.RandStdShiftIntensityd(factors=(-, ), p=),\n ],\n p=,\n ),\n \n T.OneOf(\n [\n \n T.RandScaleIntensityFixedMeand(factors=(-, ), p=),\n \n T.RandScaleIntensityFixedMeand(factors=(, ), p=),\n ],\n p=,\n ),\n ],\n p=,\n ),\n T.RandScaleIntensityd(factors=(-, ), channel_wise=, p=),\n T.RandAdjustContrastd(gamma=(, ), p=),\n T.RandHistogramShiftd(num_control_points=(, ), p=),\n ],\n p=,\n ),\n \n \n CT.ApplyTransformToNormalDistributionsd(keys=[]),\n \n T.RandSimulateLowResolutiond(\n zoom_range=(, ), downsample_mode=, upsample_mode=, p=\n ),\n CT.RandCoarseDropoutWithKeypointsd(\n keypoints_key=, holes=, spatial_size=(, , ),\n dropout_holes=, fill_value=(, ), max_holes=,\n max_spatial_size=(, , ), prob=\n )\n ]\n)\nFor a task where real distance/scale could be measured, e.g voxel spacing is known, I prefer a fixed sigma measured in real-life distance. Some advances in bottom-up human pose estimation have introduced techniques to adaptively rescale each keypoint's heatmap. Intuitively, these methods adjust the heatmap’s sigma values based on the size of each person, assigning larger/smaller sigmas (higher/lower uncertainty) to larger/smaller ones (relative to their pixel count/box area).
\nI use a sigma value of 200A (0.2 times the radius). Note that going through Spatial augmentations like Spacing, Affine or Zoom, the voxel spacing is changed so the value of sigma used in rendering the heatmap is not a fixed number of voxels, but rather 0.2 * 1000 / cur_voxel_spacing.
\nA custom implementation of ApplyTransformToPoints was implemented to apply Affine Transform to a Normal distribution, result in another Normal distribution to render on the target heatmap volume. Hence, instead of a ball-like shape, each keypoint looks like a rotated ellipsoid.
Does this make sense? Maybe not that significant. It's just an assumption that the likelihood of a motor strictly follows a 3D Normal distribution in the original tomogram, which is not likely for this task. Later, I decided to use a ball-like-shape Gaussian heatmap, with diagonal covariance matrix and single sigma for all 3 dimensions ZYX to make life easier for my model (just need to predict the mean and scale, intuitively).
\n3D-UNet with various 3D/2.5D Encoder was tried and be used in final submision. Taking input as a 3D volume of shape (Z, Y, X) = (224, 448, 448), all model predict the output heatmap of shape (Z/16, Y/16, X/16) = (14, 28, 28).
\n3D-UNet Decoder borrowed from segmentation_models_pytorch_3d was tweak to output multiscale heatmaps for ultilizing deep supervision. Since for this task, finest heatmap has stride 16 right after the coarsest one of 32 (not too deep), deep supervision seem to be redundant experimentally, later I just supervise the 1/16 heatmap only.
I mainly tweak and use models from either timm3d or SlowFast. By default, video models like X3D and I3D in SlowFast did not use temporal pooling, keeping the temporal dim unchanged to keep as much as temporal resolution as posible. For this task, this is unnecessary, I make change so that depth (temporal in video term) is progressively downsampled by 2 at each stage, synchronized with spatial downsampling too. Now, all 3D models have the same interface, output at least 4 (usually 5) levels of feature map at stride 1/4, 1/8, 1/16, 1/32.
\nAs discuss above, we need high resolution (small voxel spacing) for keeping very little details/textures. Standard UNet ultilize and fuse low-level/fine features via skip connection to the same output level. However, UNet with output stride of 16 is not actually U-shape now, hence there're no shorter path to propagate the low-level/fine features (rich texture/localization) to the high-level/coarse output heatmap (rich semantic). Motivated by this, I add necks to fuse low-level features to the coarse output heatmap head, efficiently improve information propagation through the network. Two types of necks were tried:
It's clear that Simple FPN is a \"more straight\" way to achieve our goal. Not putting much time on this, but my experiment results also confirm that Simple FPN is usually better than PAN. So all my final models were additionally equipped with this Simple FPN neck, illustrated below.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10254700%2F2f2cab849715f0dd2bf2b2a93fe04e8a%2F3d_unet_fpn.jpg?generation=1749212388256232&alt=media\")
I'm also working with 2.5D encoder, combined with FPN neck and 3D UNet decoder too. The idea is very simple. First we sample a list of 2D 3-channels image by sliding over the depth (Z) dimension with stride of 2, effectively treat one volume as a batch of 2D images. At each stage of 2D backbone, spatial resolution is downsampled by 2 by default, we also down sample the depth (now treated as batch) by a factor of 2 too by either AvgPool3d((2, 1, 1)) or Conv3d(stride=(2, 1, 1)). The first one was better and easier to train since it keep tensor statistics unchanged, better leveraging transfer learning from pretrained 2D backbone. It shows good score if no external dataset was used, significantly reduce overhead at middle/latter stages, being lighweight and helps model effectively \"see\" all 2D slices of a volume. It's okay for 2D convolution-only backbone such as EfficientNet or ConvNeXT which the heavy computation goes in middle or latter stages, but not for backbone which use Transformer in early stage cause of the high resolution and very large \"fake batch size\"
Also played with somethings like injecting LSTM layers after obtained 2D features, CV may go up but LB just unchanged. Perhap Conv3d in FPN neck and UNet Decoder was better fusing slicewise features, additional mixing is not necessary anymore.
\nI report LB/PB score of single model (single checkpoint). These models are:
\nI start using external data a little bit late, so just 1 last checkpoint of them was submitted to obtain LB score with very limitted tuning. Each model's runtime is not exceed 3h.
\n| Model | \nParams | \nPrivate LB | \nPublic LB | \n
|---|---|---|---|
| 3D RESNEXT50 | \n49.9M | \n86.2 | \n85.8 | \n
| 3D DENSENET121 | \n27.6M | \n85.7 | \n86.0 | \n
| X3DM + FPN | \n10.4M | \n84.6 | \n84.8 | \n
| X3DM + PAN + MIXUP | \n6.9M | \n84.6 | \n84.5 | \n
| X3DL | \n12.8M | \n83.9 | \n85.2 | \n
| 3D RESNET50 | \n70.3M | \n83.0 | \n85.4 | \n
| 2.5D CONVNEXT_TINY | \n40.3M | \n82.4 | \n83.7 | \n
| 2D COAT_LITE_MEDIUM | \n99.6M | \n83.3 | \n84.8 | \n
| 2D MAXVIT_TINY | \n86.1M | \n82.5 | \n85.4 | \n
2D approach was in my plan, but it becomes urgent since I feel that may be many high score public LB may using 2D approaches, especially YOLO. 2 weeks before competition deadline, I started working on 2D modeling and inherrit many details from the 3D one. Training 2D models was much more resource efficient and much faster compared to 3D.
\nThe same target voxel spacing of 16.0 was used too, i.e I resize each 2D slice so that XY spacing is 16A. Of course augmentations could scale this spacing later, but same as 3D approach, sigma of each keypoint is rescaled accordingly so that final sigma is fixed at a real distance of 200A.
\nIn inference stage, I load the full volume on RAM then perform LINEAR interpolation to fixed voxel spacing of (Z, Y, X) = (32, 16, 16), that say sampling along Z (depth) is more sparse. This helps reduce inference cost without too much performance decregation. I use 3 channels image of size (896, 896) as input to 2D-UNet model.
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10254700%2Fa69bde60b2bcc2eff12633f2a3e5ea08%2F2d_pipeline.jpg?generation=1749212467165435&alt=media\")
In training stage, different from 3D pipeline where I load whole volume to RAM then using MONAI's lazy resampling (using F.grid_sample internally), 2D pipeline just sample a specific value of Z (following truncated normal distribution with scale of 1 sigma) then load 3 nearby images per sample (spacing~32A) from disk, which is pretty efficient. Although 2D approach is more sample efficient to reach a medium score, controlling the positive/negative samples ratio is still important so that model could see more negative slices over all data domain/distribution rather than just the \"motor’s neighbor\" slices. I started experimenting with negative/positive ratio equal to 0.25, latter found that CV improved quite significantly when set it to 0.75, but not enough submissions to confirm that on LB.
\nAlbumentations was used to create augmentation pipeline, with ability to handle and keep track of keypoints scaling as well, which is pretty neat.
\n
TARGET_SPACING = \nPATCH_SIZE = (, )\n\n ():\n \n params\n data (data[]) > :\n kpt = random.choice(data[])\n \n (kpt) == \n x, y = kpt[:]\n scale = kpt[]\n [\n x - margin_xy[] * scale,\n y - margin_xy[] * scale,\n x + margin_xy[] * scale,\n y + margin_xy[] * scale,\n ]\n :\n \n\n\naugment_transform = A.Compose(\n [\n \n A.HorizontalFlip(p=),\n A.VerticalFlip(p=),\n \n A.OneOf(\n [\n A.GaussNoise(\n var_limit=(, ),\n mean=,\n per_channel=,\n noise_scale_factor=,\n p=,\n ),\n A.MultiplicativeNoise(\n multiplier=(, ),\n per_channel=,\n elementwise=,\n p=,\n ),\n ],\n p=,\n ),\n \n A.OneOf(\n [\n \n A.OneOf(\n [\n A.Downscale(\n scale_range=(, ),\n interpolation_pair={\n : cv2.INTER_LANCZOS4,\n : cv2.INTER_AREA,\n },\n p=,\n ),\n A.Downscale(\n scale_range=(, ),\n interpolation_pair={\n : cv2.INTER_LINEAR,\n : cv2.INTER_AREA,\n },\n p=,\n ),\n A.Downscale(\n scale_range=(, ),\n interpolation_pair={\n : cv2.INTER_LINEAR,\n : cv2.INTER_LINEAR,\n },\n p=,\n ),\n ],\n p=,\n ),\n A.ImageCompression(\n compression_type=, quality_range=(, ), p=\n ),\n A.Posterize(num_bits=(, ), p=),\n ],\n p=,\n ),\n \n A.OneOf(\n [\n \n A.Emboss(alpha=(, ), strength=(, ), p=),\n A.Sharpen(alpha=(, ), lightness=(, ), p=),\n A.CLAHE(clip_limit=, tile_grid_size=(, ), p=),\n ],\n p=,\n ),\n \n A.OneOf(\n [\n A.OneOf(\n [\n A.RandomBrightnessContrast(\n brightness_limit=(-, ),\n contrast_limit=(-, ),\n brightness_by_max=,\n p=,\n ),\n A.RandomBrightnessContrast(\n brightness_limit=(, ),\n contrast_limit=(-, ),\n brightness_by_max=,\n p=,\n ),\n A.RandomBrightnessContrast(\n brightness_limit=(-, ),\n contrast_limit=(, ),\n brightness_by_max=,\n p=,\n ),\n A.RandomBrightnessContrast(\n brightness_limit=(, ),\n contrast_limit=(, ),\n brightness_by_max=,\n p=,\n ),\n ],\n p=,\n ),\n A.RandomToneCurve(scale=, per_channel=, p=),\n A.RandomGamma(gamma_limit=(, ), p=),\n ],\n p=,\n ),\n \n A.OneOf(\n [\n \n A.Affine(\n scale={: (, ), : (, )},\n translate_percent=,\n rotate=(, ),\n shear={: (-, ), : (-, )},\n interpolation=cv2.INTER_LINEAR,\n cval=,\n mode=cv2.BORDER_CONSTANT,\n fit_output=,\n keep_ratio=,\n balanced_scale=,\n p=,\n ),\n \n A.Affine(\n scale={: (, ), : (, )},\n translate_percent=,\n rotate=(, ),\n shear={: (-, ), : (-, )},\n interpolation=cv2.INTER_LINEAR,\n cval=,\n mode=cv2.BORDER_CONSTANT,\n fit_output=,\n keep_ratio=,\n balanced_scale=,\n p=,\n ),\n A.Perspective(\n scale=(, ),\n keep_size=,\n pad_mode=cv2.BORDER_CONSTANT,\n pad_val=,\n fit_output=,\n interpolation=cv2.INTER_LINEAR,\n p=,\n ),\n ],\n p=,\n ),\n \n AC.CustomRandomSizedBBoxSafeCrop(\n crop_size=PATCH_SIZE,\n scale=(, ), \n ratio=(, ), \n get_bbox_func=partial(\n _byu_get_safe_bbox,\n margin_xy=(\n / TARGET_SPACING,\n / TARGET_SPACING,\n ),\n ),\n retry=,\n p=,\n ),\n A.PadIfNeeded(\n *PATCH_SIZE,\n position=,\n border_mode=cv2.BORDER_CONSTANT,\n value=,\n p=,\n ),\n ],\n p=,\n)\nThere're many ways to render target heatmap. I do a sweep over 4 kinds of heatmap type with different parameters, and found that the best CV was Gaussian heatmap with peak confident and radius scales smaller when goes far from the Z groudtruth coordinate, same as if we render 3D Gaussian heatmap then take a 2D slice over. adaptive scaling > min=0, max=1 > segment > point, which support more that we need to modeling uncertainty better.
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10254700%2Fd53a89634e4065110b61d729a2736fda%2F2d_heatmap_types_compressed.jpg?generation=1749212650075518&alt=media\")
The model used for 2D modeling was adapted from this Google Contrails solution writeup, thank authors very much for public their source code. It contains an image encoder, a simple FPN neck to fuse low-level features to /16 feature map, and an UNet Decoder with Pixel Shuffle block.
\nI mainly experiment with different types of 2D encoders including EfficientNet, ConvNeXt, CoaT and MaxViT. No supprising, the Hierachical Convolution-Transformer backbones were perform much better than convolution only, indicated that global context matter and we need a sufficient spatial size for obtaining enough context. In selected final submission, I use 2 2D-UNet models with timm backbones: maxvit_tiny_tf_512.in1k and coat_lite_medium_384.in1k.
Again, I must thank @brendanartley for the external dataset, especially in early stage of the competition, and @tatamikenn for his note on wrong-quantized tomograms.
\nI started diving into that external dataset to train my final submission a week ago, after ensured that my training/inference pipeline is good enough and bug-free (no?). Without external data, I have a simple ensemble of 3D/2.5D UNet with PB 0.851 and LB 0.852.
@brendanartley's dataset was good, but seem not to be directly usable for my pipeline because of slicing and resizing to a fixed shape and losing voxel spacing information. So I tweak the download script to get my own new version, improve it by pseudo labeling+manual review:
\nF.interpolate() to target_spacing=max(16.0, ori_spacing) to save disk space and reduce dataloader overhead in the future. This results in a new 180GB external dataset.0.05. Comparing model prediction results with converted GT to identify TP/TN/FP/FN motors.Adding so many external data change the game significantly. Training are more stable and allow for much longer training without overfiting. Though more tunings on new setup are needed, however, the deadline is quickly approaching its final days..
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10254700%2F715d399576ca0db1d6ff68d78b522fd1%2Fcvat.png?generation=1749212703333308&alt=media\")
As usual, I'm not use a fixed threshold since it's largely change between submitted models to the LB. Instead, just use quantile thresholding at 55% (55% lowest threshold samples are predicted as negative, the remain 45% ones predicted as positive) which is adaptively changed based on model's confident distribution, so just one LB submission is enough to estimate the best F2 score.
\nThe postprocessing process:
\nconf_mode='avg'I didn't conduct many experiments with ensemble methods before, and decided to use WBF on the two last days, just believe that it could survive some very strong peak signal from one single model, thus improve Recall.
\nThanks for your attention !
", "messages": [ { "id": "3218612", "postDate": "06/06/2025 12:31:22", "content": "Many thanks to the competition host and Kaggle for another engaging challenge - and big congratulations to all the participants!
\nThank you to every one who actively shared insights and notebooks to the community, especially @brendanartley for the external dataset.
As always, I had a great time learning throughout the competition and was fortunate enough to finish in the gold zone. I'm really happy to share a few thoughts here.
\n3D Resnext50 (2xTTA), 3D Densenet121 (2xTTA), 3D X3D-M (2xTTA), 2D MaxViT (2xTTA) and 2D CoaT (1xTTA)(224, 448, 448) while 2D use (3, 896, 896) patch size to incorporate enough global context.stride=16 and sigma=200A for uncertainty modelingWhat not work
\nA simple 4-StratifiedKfold by number of motors was conducted on train dataset. Validation samples are augmented by Rotate90+Flip (16 posible combinations) to increase the number of samples, making validation metrics more stable and trustworthy to track.
\nExperiment progress was tracked using mainly F2 score and additionally Partial Average Precision, cutoff at a specific Recall, e.g > 0.9. The second is not a binary metric so more stable and less sentitive to threshold, strong correlate with F2. Single-fold local CV was the main factor to select models and hyperparameters.
\nTrain on all data, validate on LB: However, when my pipeline are bug-free and going better (when CV > 0.96 and LB > 0.8), CV is just saturated and not correlate to LB well. So at the later stage, CV is just used to guide a reasonable value of hyperparameter (mainly learning rate). In last week, all models were trained on all dataset without local validation and are submitted to LB.
\nI resample tomogram to a fixed voxel spacing of 16.0, use sliding windows (patching) technique, split tomogram to 224x448x448 (possibly overlap) patches as input to models to obtain 16x downscaled prediction heatmap of shape 14x28x28.
\nThe labeling coordinate is a little bit noisy/uncertain, also the tolerance radius is high, so using high heatmap resolution seem to be redundant, significantly increase the computation. Radius of 1000A == 62.5 voxels in input volume == 4 voxels in prediction heatmap. I've experimented with prediction heatmap of 8x and 32x downscaled heatmap and found that 8x ~ 16x >> 32x. To the end, I use 16x downscaled heatmap for both 3D and 2D models.
Optimal voxel spacing is ~16.0
\nFor this task, resampling to a suitable voxel spacing is important. Using small voxel spacing keep the very small details but computation intensive due to high resolution, in contrast large voxel spacing could erase out all the textures needed for detection. Experiment results confirm that very small detail/textures are matter:
The mean original voxel spacing in LB/PB is 13.1 ???
\nHaha, just my stupid guess work and there're no way I could confirm. I trained model with target_spacing=16.0 and submit to LB with different of assumtions about the original spacing. 13.1 brings the highest LB. Given that test samples have higher resolution (~400x1400x1050 in average) and 44.7% of train samples have voxel_spacing=13.1, I safely assume that the average test voxel spacing is ~13.1, and resample to 16.0 for all experiments
\n\nI tried to quickly train a voxel spacing prediction model on the last day, but failed, metric is about
\nMAE=5.1, can't predict any useful thing :D
Each sample is dominated by negative voxels, so I try to control the number of positive/negative patches ratio in the training progress. For 3D approach, the positive/negative samples ratio is 20/1.
\nFor each epoch, all positive samples are ensured to be augmented with all 16 transformations composed of Rotate90+Flip, go after a \"heavy\" MONAI augmentation pipeline. Keypoints passed to MONAI dictionary transform as well so that we get the Affine-transformed keypoints to render the heatmap with larger stride (16 in this case). I really like MONAI's Lazy Resampling, which make augmentation and dataloader become fast and efficient.
\nVarious MONAI 3D augmentations were used:
RandZoom, RandAffinedRandShiftIntensityd, RandStdShiftIntensityd, RandScaleIntensityFixedMeand, RandScaleIntensityd, RandAdjustContrastd, RandHistogramShiftdRandSimulateLowResolutiondRandCoarseDropoutKeepKeypointsMixup and Cutmix were tried but bring no clear improvement so be ignored in latter experiments.
\n monai transforms T\n monai_custom CT\n\nT.Compose(\n [\n \n T.Spacingd(target_spacing=, p=),\n CT.CustomCropBySlicesd(),\n \n T.RandZoomd(min_zoom=(, , ), max_zoom=(, , ), p=),\n T.OneOf(\n [\n \n T.RandAffined(\n rotate_range=((, ), (, ), (, )),\n scale_range=((-, ), (-, ), (-, )),\n p=,\n ),\n \n T.RandAffined(\n rotate_range=((-, ), (-, ), (, )),\n shear_range=((-, ), (-, ), (-, )),\n scale_range=((-, ), (-, ), (-, )),\n p=,\n ),\n ],\n p=,\n ),\n \n T.OneOf(\n [\n T.Compose(\n [\n \n T.OneOf(\n [\n T.RandShiftIntensityd(offset=(-, ), p=),\n T.RandStdShiftIntensityd(factors=(-, ), p=),\n ],\n p=,\n ),\n \n T.OneOf(\n [\n \n T.RandScaleIntensityFixedMeand(factors=(-, ), p=),\n \n T.RandScaleIntensityFixedMeand(factors=(, ), p=),\n ],\n p=,\n ),\n ],\n p=,\n ),\n T.RandScaleIntensityd(factors=(-, ), channel_wise=, p=),\n T.RandAdjustContrastd(gamma=(, ), p=),\n T.RandHistogramShiftd(num_control_points=(, ), p=),\n ],\n p=,\n ),\n \n \n CT.ApplyTransformToNormalDistributionsd(keys=[]),\n \n T.RandSimulateLowResolutiond(\n zoom_range=(, ), downsample_mode=, upsample_mode=, p=\n ),\n CT.RandCoarseDropoutWithKeypointsd(\n keypoints_key=, holes=, spatial_size=(, , ),\n dropout_holes=, fill_value=(, ), max_holes=,\n max_spatial_size=(, , ), prob=\n )\n ]\n)\nFor a task where real distance/scale could be measured, e.g voxel spacing is known, I prefer a fixed sigma measured in real-life distance. Some advances in bottom-up human pose estimation have introduced techniques to adaptively rescale each keypoint's heatmap. Intuitively, these methods adjust the heatmap’s sigma values based on the size of each person, assigning larger/smaller sigmas (higher/lower uncertainty) to larger/smaller ones (relative to their pixel count/box area).
\nI use a sigma value of 200A (0.2 times the radius). Note that going through Spatial augmentations like Spacing, Affine or Zoom, the voxel spacing is changed so the value of sigma used in rendering the heatmap is not a fixed number of voxels, but rather 0.2 * 1000 / cur_voxel_spacing.
\nA custom implementation of ApplyTransformToPoints was implemented to apply Affine Transform to a Normal distribution, result in another Normal distribution to render on the target heatmap volume. Hence, instead of a ball-like shape, each keypoint looks like a rotated ellipsoid.
Does this make sense? Maybe not that significant. It's just an assumption that the likelihood of a motor strictly follows a 3D Normal distribution in the original tomogram, which is not likely for this task. Later, I decided to use a ball-like-shape Gaussian heatmap, with diagonal covariance matrix and single sigma for all 3 dimensions ZYX to make life easier for my model (just need to predict the mean and scale, intuitively).
\n3D-UNet with various 3D/2.5D Encoder was tried and be used in final submision. Taking input as a 3D volume of shape (Z, Y, X) = (224, 448, 448), all model predict the output heatmap of shape (Z/16, Y/16, X/16) = (14, 28, 28).
\n3D-UNet Decoder borrowed from segmentation_models_pytorch_3d was tweak to output multiscale heatmaps for ultilizing deep supervision. Since for this task, finest heatmap has stride 16 right after the coarsest one of 32 (not too deep), deep supervision seem to be redundant experimentally, later I just supervise the 1/16 heatmap only.
I mainly tweak and use models from either timm3d or SlowFast. By default, video models like X3D and I3D in SlowFast did not use temporal pooling, keeping the temporal dim unchanged to keep as much as temporal resolution as posible. For this task, this is unnecessary, I make change so that depth (temporal in video term) is progressively downsampled by 2 at each stage, synchronized with spatial downsampling too. Now, all 3D models have the same interface, output at least 4 (usually 5) levels of feature map at stride 1/4, 1/8, 1/16, 1/32.
\nAs discuss above, we need high resolution (small voxel spacing) for keeping very little details/textures. Standard UNet ultilize and fuse low-level/fine features via skip connection to the same output level. However, UNet with output stride of 16 is not actually U-shape now, hence there're no shorter path to propagate the low-level/fine features (rich texture/localization) to the high-level/coarse output heatmap (rich semantic). Motivated by this, I add necks to fuse low-level features to the coarse output heatmap head, efficiently improve information propagation through the network. Two types of necks were tried:
It's clear that Simple FPN is a \"more straight\" way to achieve our goal. Not putting much time on this, but my experiment results also confirm that Simple FPN is usually better than PAN. So all my final models were additionally equipped with this Simple FPN neck, illustrated below.
\n
I'm also working with 2.5D encoder, combined with FPN neck and 3D UNet decoder too. The idea is very simple. First we sample a list of 2D 3-channels image by sliding over the depth (Z) dimension with stride of 2, effectively treat one volume as a batch of 2D images. At each stage of 2D backbone, spatial resolution is downsampled by 2 by default, we also down sample the depth (now treated as batch) by a factor of 2 too by either AvgPool3d((2, 1, 1)) or Conv3d(stride=(2, 1, 1)). The first one was better and easier to train since it keep tensor statistics unchanged, better leveraging transfer learning from pretrained 2D backbone. It shows good score if no external dataset was used, significantly reduce overhead at middle/latter stages, being lighweight and helps model effectively \"see\" all 2D slices of a volume. It's okay for 2D convolution-only backbone such as EfficientNet or ConvNeXT which the heavy computation goes in middle or latter stages, but not for backbone which use Transformer in early stage cause of the high resolution and very large \"fake batch size\"
Also played with somethings like injecting LSTM layers after obtained 2D features, CV may go up but LB just unchanged. Perhap Conv3d in FPN neck and UNet Decoder was better fusing slicewise features, additional mixing is not necessary anymore.
\nI report LB/PB score of single model (single checkpoint). These models are:
\nI start using external data a little bit late, so just 1 last checkpoint of them was submitted to obtain LB score with very limitted tuning. Each model's runtime is not exceed 3h.
\n| Model | \nParams | \nPrivate LB | \nPublic LB | \n
|---|---|---|---|
| 3D RESNEXT50 | \n49.9M | \n86.2 | \n85.8 | \n
| 3D DENSENET121 | \n27.6M | \n85.7 | \n86.0 | \n
| X3DM + FPN | \n10.4M | \n84.6 | \n84.8 | \n
| X3DM + PAN + MIXUP | \n6.9M | \n84.6 | \n84.5 | \n
| X3DL | \n12.8M | \n83.9 | \n85.2 | \n
| 3D RESNET50 | \n70.3M | \n83.0 | \n85.4 | \n
| 2.5D CONVNEXT_TINY | \n40.3M | \n82.4 | \n83.7 | \n
| 2D COAT_LITE_MEDIUM | \n99.6M | \n83.3 | \n84.8 | \n
| 2D MAXVIT_TINY | \n86.1M | \n82.5 | \n85.4 | \n
2D approach was in my plan, but it becomes urgent since I feel that may be many high score public LB may using 2D approaches, especially YOLO. 2 weeks before competition deadline, I started working on 2D modeling and inherrit many details from the 3D one. Training 2D models was much more resource efficient and much faster compared to 3D.
\nThe same target voxel spacing of 16.0 was used too, i.e I resize each 2D slice so that XY spacing is 16A. Of course augmentations could scale this spacing later, but same as 3D approach, sigma of each keypoint is rescaled accordingly so that final sigma is fixed at a real distance of 200A.
\nIn inference stage, I load the full volume on RAM then perform LINEAR interpolation to fixed voxel spacing of (Z, Y, X) = (32, 16, 16), that say sampling along Z (depth) is more sparse. This helps reduce inference cost without too much performance decregation. I use 3 channels image of size (896, 896) as input to 2D-UNet model.
In training stage, different from 3D pipeline where I load whole volume to RAM then using MONAI's lazy resampling (using F.grid_sample internally), 2D pipeline just sample a specific value of Z (following truncated normal distribution with scale of 1 sigma) then load 3 nearby images per sample (spacing~32A) from disk, which is pretty efficient. Although 2D approach is more sample efficient to reach a medium score, controlling the positive/negative samples ratio is still important so that model could see more negative slices over all data domain/distribution rather than just the \"motor’s neighbor\" slices. I started experimenting with negative/positive ratio equal to 0.25, latter found that CV improved quite significantly when set it to 0.75, but not enough submissions to confirm that on LB.
\nAlbumentations was used to create augmentation pipeline, with ability to handle and keep track of keypoints scaling as well, which is pretty neat.
\n
TARGET_SPACING = \nPATCH_SIZE = (, )\n\n ():\n \n params\n data (data[]) > :\n kpt = random.choice(data[])\n \n (kpt) == \n x, y = kpt[:]\n scale = kpt[]\n [\n x - margin_xy[] * scale,\n y - margin_xy[] * scale,\n x + margin_xy[] * scale,\n y + margin_xy[] * scale,\n ]\n :\n \n\n\naugment_transform = A.Compose(\n [\n \n A.HorizontalFlip(p=),\n A.VerticalFlip(p=),\n \n A.OneOf(\n [\n A.GaussNoise(\n var_limit=(, ),\n mean=,\n per_channel=,\n noise_scale_factor=,\n p=,\n ),\n A.MultiplicativeNoise(\n multiplier=(, ),\n per_channel=,\n elementwise=,\n p=,\n ),\n ],\n p=,\n ),\n \n A.OneOf(\n [\n \n A.OneOf(\n [\n A.Downscale(\n scale_range=(, ),\n interpolation_pair={\n : cv2.INTER_LANCZOS4,\n : cv2.INTER_AREA,\n },\n p=,\n ),\n A.Downscale(\n scale_range=(, ),\n interpolation_pair={\n : cv2.INTER_LINEAR,\n : cv2.INTER_AREA,\n },\n p=,\n ),\n A.Downscale(\n scale_range=(, ),\n interpolation_pair={\n : cv2.INTER_LINEAR,\n : cv2.INTER_LINEAR,\n },\n p=,\n ),\n ],\n p=,\n ),\n A.ImageCompression(\n compression_type=, quality_range=(, ), p=\n ),\n A.Posterize(num_bits=(, ), p=),\n ],\n p=,\n ),\n \n A.OneOf(\n [\n \n A.Emboss(alpha=(, ), strength=(, ), p=),\n A.Sharpen(alpha=(, ), lightness=(, ), p=),\n A.CLAHE(clip_limit=, tile_grid_size=(, ), p=),\n ],\n p=,\n ),\n \n A.OneOf(\n [\n A.OneOf(\n [\n A.RandomBrightnessContrast(\n brightness_limit=(-, ),\n contrast_limit=(-, ),\n brightness_by_max=,\n p=,\n ),\n A.RandomBrightnessContrast(\n brightness_limit=(, ),\n contrast_limit=(-, ),\n brightness_by_max=,\n p=,\n ),\n A.RandomBrightnessContrast(\n brightness_limit=(-, ),\n contrast_limit=(, ),\n brightness_by_max=,\n p=,\n ),\n A.RandomBrightnessContrast(\n brightness_limit=(, ),\n contrast_limit=(, ),\n brightness_by_max=,\n p=,\n ),\n ],\n p=,\n ),\n A.RandomToneCurve(scale=, per_channel=, p=),\n A.RandomGamma(gamma_limit=(, ), p=),\n ],\n p=,\n ),\n \n A.OneOf(\n [\n \n A.Affine(\n scale={: (, ), : (, )},\n translate_percent=,\n rotate=(, ),\n shear={: (-, ), : (-, )},\n interpolation=cv2.INTER_LINEAR,\n cval=,\n mode=cv2.BORDER_CONSTANT,\n fit_output=,\n keep_ratio=,\n balanced_scale=,\n p=,\n ),\n \n A.Affine(\n scale={: (, ), : (, )},\n translate_percent=,\n rotate=(, ),\n shear={: (-, ), : (-, )},\n interpolation=cv2.INTER_LINEAR,\n cval=,\n mode=cv2.BORDER_CONSTANT,\n fit_output=,\n keep_ratio=,\n balanced_scale=,\n p=,\n ),\n A.Perspective(\n scale=(, ),\n keep_size=,\n pad_mode=cv2.BORDER_CONSTANT,\n pad_val=,\n fit_output=,\n interpolation=cv2.INTER_LINEAR,\n p=,\n ),\n ],\n p=,\n ),\n \n AC.CustomRandomSizedBBoxSafeCrop(\n crop_size=PATCH_SIZE,\n scale=(, ), \n ratio=(, ), \n get_bbox_func=partial(\n _byu_get_safe_bbox,\n margin_xy=(\n / TARGET_SPACING,\n / TARGET_SPACING,\n ),\n ),\n retry=,\n p=,\n ),\n A.PadIfNeeded(\n *PATCH_SIZE,\n position=,\n border_mode=cv2.BORDER_CONSTANT,\n value=,\n p=,\n ),\n ],\n p=,\n)\nThere're many ways to render target heatmap. I do a sweep over 4 kinds of heatmap type with different parameters, and found that the best CV was Gaussian heatmap with peak confident and radius scales smaller when goes far from the Z groudtruth coordinate, same as if we render 3D Gaussian heatmap then take a 2D slice over. adaptive scaling > min=0, max=1 > segment > point, which support more that we need to modeling uncertainty better.
The model used for 2D modeling was adapted from this Google Contrails solution writeup, thank authors very much for public their source code. It contains an image encoder, a simple FPN neck to fuse low-level features to /16 feature map, and an UNet Decoder with Pixel Shuffle block.
\nI mainly experiment with different types of 2D encoders including EfficientNet, ConvNeXt, CoaT and MaxViT. No supprising, the Hierachical Convolution-Transformer backbones were perform much better than convolution only, indicated that global context matter and we need a sufficient spatial size for obtaining enough context. In selected final submission, I use 2 2D-UNet models with timm backbones: maxvit_tiny_tf_512.in1k and coat_lite_medium_384.in1k.
Again, I must thank @brendanartley for the external dataset, especially in early stage of the competition, and @tatamikenn for his note on wrong-quantized tomograms.
\nI started diving into that external dataset to train my final submission a week ago, after ensured that my training/inference pipeline is good enough and bug-free (no?). Without external data, I have a simple ensemble of 3D/2.5D UNet with PB 0.851 and LB 0.852.
@brendanartley's dataset was good, but seem not to be directly usable for my pipeline because of slicing and resizing to a fixed shape and losing voxel spacing information. So I tweak the download script to get my own new version, improve it by pseudo labeling+manual review:
\nF.interpolate() to target_spacing=max(16.0, ori_spacing) to save disk space and reduce dataloader overhead in the future. This results in a new 180GB external dataset.0.05. Comparing model prediction results with converted GT to identify TP/TN/FP/FN motors.Adding so many external data change the game significantly. Training are more stable and allow for much longer training without overfiting. Though more tunings on new setup are needed, however, the deadline is quickly approaching its final days..
\n
As usual, I'm not use a fixed threshold since it's largely change between submitted models to the LB. Instead, just use quantile thresholding at 55% (55% lowest threshold samples are predicted as negative, the remain 45% ones predicted as positive) which is adaptively changed based on model's confident distribution, so just one LB submission is enough to estimate the best F2 score.
\nThe postprocessing process:
\nconf_mode='avg'I didn't conduct many experiments with ensemble methods before, and decided to use WBF on the two last days, just believe that it could survive some very strong peak signal from one single model, thus improve Recall.
\nThanks for your attention !
", "rawMarkdown": "Many thanks to the competition host and Kaggle for another engaging challenge - and big congratulations to all the participants!\nThank you to every one who actively shared insights and notebooks to the community, especially @brendanartley for [the external dataset](https://www.kaggle.com/competitions/byu-locating-bacterial-flagellar-motors-2025/discussion/569921).\n\nAs always, I had a great time learning throughout the competition and was fortunate enough to finish in the gold zone. I'm really happy to share a few thoughts here.\n \n# TLDR;\n- I'm focus on the 3D/2.5D/2D UNet approaches. The final submission PB 86.6 was an ensemble of 5 models: `3D Resnext50 (2xTTA)`, `3D Densenet121 (2xTTA)`, `3D X3D-M (2xTTA)`, `2D MaxViT (2xTTA)` and `2D CoaT (1xTTA)`\n- 3D model use patch size of `(224, 448, 448)` while 2D use `(3, 896, 896)` patch size to incorporate enough global context.\n- Gaussian heatmap with `stride=16` and `sigma=200A` for uncertainty modeling\n- Simple FPN neck to fuse low-level features into stride-16-feature map\n- All final selected models are trained on combine of train dataset + external dataset without local validation\n- Heavy augmentations allowed to train for more epochs and be more robust to domain/distribution change.\n- Pseudo labeling on train+external dataset, with manual review\n- Resample all tomograms to 16A voxel spacing + Sliding window patches inference\n- Simple BCE loss works best\n- Weighted Box Fusion (WBF) to ensemble multiple model/TTA ’s predictions\n\n\n**What not work**\n- Other losses: MSE, L1, Weighted BCE, Focal, Tversky, Combine multiple losses\n\n\n\n\n# Table of Contents\n\n\n- [Validation strategy](#validation-strategy)\n- [3D-UNet Approach](#3d-unet-approach)\n - [Data processing](#data-processing)\n - [3D augmentation](#3d-augmentation)\n - [3D target heatmap](#3d-target-heatmap)\n - [3D modeling](#3d-modeling)\n - [3D Encoder](#3d-encoder)\n - [2.5D Encoder](#25d-encoder)\n - [Some results](#some-results)\n- [2D-UNet Approach](#2d-unet-approach)\n - [Data processing](#data-processing-1)\n - [2D augmentation](#2d-augmentation)\n - [2D target heatmap](#2d-target-heatmap)\n - [2D modeling](#2d-modeling)\n- [External dataset & Pseudo label](#external-dataset--pseudo-label)\n- [Postprocessing](#postprocessing)\n- [Source code](#source-code)\n\n\n---\n\n# Validation strategy\nA simple 4-StratifiedKfold by number of motors was conducted on train dataset. Validation samples are augmented by Rotate90+Flip (16 posible combinations) to increase the number of samples, making validation metrics more stable and trustworthy to track.\n\nExperiment progress was tracked using mainly F2 score and additionally Partial Average Precision, cutoff at a specific Recall, e.g > 0.9. The second is not a binary metric so more stable and less sentitive to threshold, strong correlate with F2. Single-fold local CV was the main factor to select models and hyperparameters.\n\n**Train on all data, validate on LB:** However, when my pipeline are bug-free and going better (when CV > 0.96 and LB > 0.8), CV is just saturated and not correlate to LB well. So at the later stage, CV is just used to guide a reasonable value of hyperparameter (mainly learning rate). In last week, all models were trained on all dataset without local validation and are submitted to LB.\n\n\n# 3D-UNet Approach\n\n\n## Data processing\n\nI resample tomogram to a fixed voxel spacing of 16.0, use sliding windows (patching) technique, split tomogram to `224x448x448` (possibly overlap) patches as input to models to obtain **16x** downscaled prediction heatmap of shape `14x28x28`. \nThe labeling coordinate is a little bit noisy/uncertain, also the tolerance radius is high, so using high heatmap resolution seem to be redundant, significantly increase the computation. `Radius of 1000A` == `62.5 voxels in input volume` == `4 voxels in prediction heatmap`. I've experimented with prediction heatmap of `8x` and `32x` downscaled heatmap and found that `8x` ~ `16x` >> `32x`. To the end, I use `16x` downscaled heatmap for both 3D and 2D models.\n\n**Optimal voxel spacing is ~16.0** \nFor this task, resampling to a suitable voxel spacing is important. Using small voxel spacing keep the very small details but computation intensive due to high resolution, in contrast large voxel spacing could erase out all the textures needed for detection. Experiment results confirm that very small detail/textures are matter:\n\n- Tried various target voxel spacing 12, 16, 20, 24, 32, 64; found that CV degrade significantly when spacing > 32. LB also degrade much faster right after spacing > 20. I found 16.0 is the best balance value, matching with the typical [median spacing](https://github.com/MIC-DKFZ/nnUNet/issues/2701) default practice of nnUnet (median spacing is 15.6 for the train dataset).\n- Resampling interpolation mode (LINEAR, AREA, NEAREST) also could significantly affect performance. All tomograms are also noisy (Gaussian noise?) so resize/interpolation could erase thing in an unexpected manner.\n\n**The mean original voxel spacing in LB/PB is 13.1 ???** \nHaha, just my stupid guess work and there're no way I could confirm. I trained model with `target_spacing=16.0` and submit to LB with different of assumtions about the original spacing. `13.1` brings the highest LB. Given that test samples have higher resolution (`~400x1400x1050` in average) and 44.7% of train samples have `voxel_spacing=13.1`, **I safely assume that the average test voxel spacing is ~13.1, and resample to 16.0 for all experiments**\n> I tried to quickly train a voxel spacing prediction model on the last day, but failed, metric is about `MAE=5.1`, can't predict any useful thing :D\n\nEach sample is dominated by negative voxels, so I try to control the number of positive/negative patches ratio in the training progress. For 3D approach, the positive/negative samples ratio is 20/1.\n\n\n## 3D augmentation\n\nFor each epoch, all positive samples are ensured to be augmented with all 16 transformations composed of Rotate90+Flip, go after a \"heavy\" MONAI augmentation pipeline. Keypoints passed to MONAI dictionary transform as well so that we get the Affine-transformed keypoints to render the heatmap with larger stride (16 in this case). I really like MONAI's [Lazy Resampling](https://docs.monai.io/en/stable/lazy_resampling.html), which make augmentation and dataloader become fast and efficient.\nVarious MONAI 3D augmentations were used:\n- Spatial: `RandZoom`, `RandAffined`\n- Intensity: `RandShiftIntensityd`, `RandStdShiftIntensityd`, `RandScaleIntensityFixedMeand`, `RandScaleIntensityd`, `RandAdjustContrastd`, `RandHistogramShiftd`\n- Reduce quality: `RandSimulateLowResolutiond`\n- Dropout: `RandCoarseDropoutKeepKeypoints`\n\nMixup and Cutmix were tried but bring no clear improvement so be ignored in latter experiments.\n\ncongrats, thanks for your sharing, can't image so many works behind.
", "rawMarkdown": "congrats, thanks for your sharing, can't image so many works behind.", "votes": null }, { "id": "3223247", "postDate": "06/13/2025 04:16:15", "content": "Congratulations and thank you for sharing details of your work. I believe success is even better when achieved alone. I look forward to the publication of your notebooks to learn more about medical applications.
\nAll the best!
", "rawMarkdown": "Congratulations and thank you for sharing details of your work. I believe success is even better when achieved alone. I look forward to the publication of your notebooks to learn more about medical applications.\n\nAll the best!", "votes": null }, { "id": "3240519", "postDate": "07/03/2025 22:41:16", "content": "@dangnh0611 I am attempting to recreate your solution and am unable to setup the environment. I decomposed the pip install commands of the dockerfile into its separate components and the one that fails is pip install mamba-ssm
\nIt’s strange, did you use the same base image pytorch/pytorch:2.6.0-cuda12.6-cudnn9-devel ?
\nAnyway, you can skip installing mamba-ssm caused my solution did not use it. Sorry for the confusion
This is one of the best posts I've read. Great write up, good graphics!
", "rawMarkdown": "This is one of the best posts I've read. Great write up, good graphics!", "votes": null }, { "id": "3245116", "postDate": "07/09/2025 00:14:03", "content": "I ended up getting the docker environment working after a while. However, when running the script to download extra data, the processing workers freeze up. I am going to try and train your model without the extra data until I can find a workaround
", "rawMarkdown": "I ended up getting the docker environment working after a while. However, when running the script to download extra data, the processing workers freeze up. I am going to try and train your model without the extra data until I can find a workaround", "votes": null }, { "id": "3245250", "postDate": "07/09/2025 05:50:17", "content": "The download script runs fine on my machine, but I haven’t tested it in other environments, so there may be issues I’m not yet aware of.
\nIn case of failure, you can modify the script a little bit to download continously, skipping the processed ones.
\nNote that some tomograms can take a long time to process due to their high resolution. On my setup, a single tomogram can take up to 20 minutes, however there're not so many ones like that.
\nIf you run into any problems, feel free to DM me directly on a chat platform. I’m happy to help, just email me at dangnh0611@gmail.com to coordinate a DM.
\nThanks for your patience.
congrats, thanks for your sharing, can't image so many works behind.
", "votes": null, "replies": [] }, { "id": 3223247, "author_name": "guillermoperezg", "author_url": "", "post_date": "06/13/2025 04:16:15", "content": "Congratulations and thank you for sharing details of your work. I believe success is even better when achieved alone. I look forward to the publication of your notebooks to learn more about medical applications.
\nAll the best!
", "votes": null, "replies": [] }, { "id": 3240519, "author_name": "andrewjdarley", "author_url": "", "post_date": "07/03/2025 22:41:16", "content": "@dangnh0611 I am attempting to recreate your solution and am unable to setup the environment. I decomposed the pip install commands of the dockerfile into its separate components and the one that fails is pip install mamba-ssm
\nIt’s strange, did you use the same base image pytorch/pytorch:2.6.0-cuda12.6-cudnn9-devel ?
\nAnyway, you can skip installing mamba-ssm caused my solution did not use it. Sorry for the confusion
I ended up getting the docker environment working after a while. However, when running the script to download extra data, the processing workers freeze up. I am going to try and train your model without the extra data until I can find a workaround
", "votes": null, "replies": [ { "id": 3245250, "author_name": "dangnh0611", "author_url": "", "post_date": "07/09/2025 05:50:17", "content": "The download script runs fine on my machine, but I haven’t tested it in other environments, so there may be issues I’m not yet aware of.
\nIn case of failure, you can modify the script a little bit to download continously, skipping the processed ones.
\nNote that some tomograms can take a long time to process due to their high resolution. On my setup, a single tomogram can take up to 20 minutes, however there're not so many ones like that.
\nIf you run into any problems, feel free to DM me directly on a chat platform. I’m happy to help, just email me at dangnh0611@gmail.com to coordinate a DM.
\nThanks for your patience.
This is one of the best posts I've read. Great write up, good graphics!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3218612": "Many thanks to the competition host and Kaggle for another engaging challenge - and big congratulations to all the participants!\nThank you to every one who actively shared insights and notebooks to the community, especially @brendanartley for [the external dataset](https://www.kaggle.com/competitions/byu-locating-bacterial-flagellar-motors-2025/discussion/569921).\n\nAs always, I had a great time learning throughout the competition and was fortunate enough to finish in the gold zone. I'm really happy to share a few thoughts here.\n \n# TLDR;\n- I'm focus on the 3D/2.5D/2D UNet approaches. The final submission PB 86.6 was an ensemble of 5 models: `3D Resnext50 (2xTTA)`, `3D Densenet121 (2xTTA)`, `3D X3D-M (2xTTA)`, `2D MaxViT (2xTTA)` and `2D CoaT (1xTTA)`\n- 3D model use patch size of `(224, 448, 448)` while 2D use `(3, 896, 896)` patch size to incorporate enough global context.\n- Gaussian heatmap with `stride=16` and `sigma=200A` for uncertainty modeling\n- Simple FPN neck to fuse low-level features into stride-16-feature map\n- All final selected models are trained on combine of train dataset + external dataset without local validation\n- Heavy augmentations allowed to train for more epochs and be more robust to domain/distribution change.\n- Pseudo labeling on train+external dataset, with manual review\n- Resample all tomograms to 16A voxel spacing + Sliding window patches inference\n- Simple BCE loss works best\n- Weighted Box Fusion (WBF) to ensemble multiple model/TTA ’s predictions\n\n\n**What not work**\n- Other losses: MSE, L1, Weighted BCE, Focal, Tversky, Combine multiple losses\n\n\n\n\n# Table of Contents\n\n\n- [Validation strategy](#validation-strategy)\n- [3D-UNet Approach](#3d-unet-approach)\n - [Data processing](#data-processing)\n - [3D augmentation](#3d-augmentation)\n - [3D target heatmap](#3d-target-heatmap)\n - [3D modeling](#3d-modeling)\n - [3D Encoder](#3d-encoder)\n - [2.5D Encoder](#25d-encoder)\n - [Some results](#some-results)\n- [2D-UNet Approach](#2d-unet-approach)\n - [Data processing](#data-processing-1)\n - [2D augmentation](#2d-augmentation)\n - [2D target heatmap](#2d-target-heatmap)\n - [2D modeling](#2d-modeling)\n- [External dataset & Pseudo label](#external-dataset--pseudo-label)\n- [Postprocessing](#postprocessing)\n- [Source code](#source-code)\n\n\n---\n\n# Validation strategy\nA simple 4-StratifiedKfold by number of motors was conducted on train dataset. Validation samples are augmented by Rotate90+Flip (16 posible combinations) to increase the number of samples, making validation metrics more stable and trustworthy to track.\n\nExperiment progress was tracked using mainly F2 score and additionally Partial Average Precision, cutoff at a specific Recall, e.g > 0.9. The second is not a binary metric so more stable and less sentitive to threshold, strong correlate with F2. Single-fold local CV was the main factor to select models and hyperparameters.\n\n**Train on all data, validate on LB:** However, when my pipeline are bug-free and going better (when CV > 0.96 and LB > 0.8), CV is just saturated and not correlate to LB well. So at the later stage, CV is just used to guide a reasonable value of hyperparameter (mainly learning rate). In last week, all models were trained on all dataset without local validation and are submitted to LB.\n\n\n# 3D-UNet Approach\n\n\n## Data processing\n\nI resample tomogram to a fixed voxel spacing of 16.0, use sliding windows (patching) technique, split tomogram to `224x448x448` (possibly overlap) patches as input to models to obtain **16x** downscaled prediction heatmap of shape `14x28x28`. \nThe labeling coordinate is a little bit noisy/uncertain, also the tolerance radius is high, so using high heatmap resolution seem to be redundant, significantly increase the computation. `Radius of 1000A` == `62.5 voxels in input volume` == `4 voxels in prediction heatmap`. I've experimented with prediction heatmap of `8x` and `32x` downscaled heatmap and found that `8x` ~ `16x` >> `32x`. To the end, I use `16x` downscaled heatmap for both 3D and 2D models.\n\n**Optimal voxel spacing is ~16.0** \nFor this task, resampling to a suitable voxel spacing is important. Using small voxel spacing keep the very small details but computation intensive due to high resolution, in contrast large voxel spacing could erase out all the textures needed for detection. Experiment results confirm that very small detail/textures are matter:\n\n- Tried various target voxel spacing 12, 16, 20, 24, 32, 64; found that CV degrade significantly when spacing > 32. LB also degrade much faster right after spacing > 20. I found 16.0 is the best balance value, matching with the typical [median spacing](https://github.com/MIC-DKFZ/nnUNet/issues/2701) default practice of nnUnet (median spacing is 15.6 for the train dataset).\n- Resampling interpolation mode (LINEAR, AREA, NEAREST) also could significantly affect performance. All tomograms are also noisy (Gaussian noise?) so resize/interpolation could erase thing in an unexpected manner.\n\n**The mean original voxel spacing in LB/PB is 13.1 ???** \nHaha, just my stupid guess work and there're no way I could confirm. I trained model with `target_spacing=16.0` and submit to LB with different of assumtions about the original spacing. `13.1` brings the highest LB. Given that test samples have higher resolution (`~400x1400x1050` in average) and 44.7% of train samples have `voxel_spacing=13.1`, **I safely assume that the average test voxel spacing is ~13.1, and resample to 16.0 for all experiments**\n> I tried to quickly train a voxel spacing prediction model on the last day, but failed, metric is about `MAE=5.1`, can't predict any useful thing :D\n\nEach sample is dominated by negative voxels, so I try to control the number of positive/negative patches ratio in the training progress. For 3D approach, the positive/negative samples ratio is 20/1.\n\n\n## 3D augmentation\n\nFor each epoch, all positive samples are ensured to be augmented with all 16 transformations composed of Rotate90+Flip, go after a \"heavy\" MONAI augmentation pipeline. Keypoints passed to MONAI dictionary transform as well so that we get the Affine-transformed keypoints to render the heatmap with larger stride (16 in this case). I really like MONAI's [Lazy Resampling](https://docs.monai.io/en/stable/lazy_resampling.html), which make augmentation and dataloader become fast and efficient.\nVarious MONAI 3D augmentations were used:\n- Spatial: `RandZoom`, `RandAffined`\n- Intensity: `RandShiftIntensityd`, `RandStdShiftIntensityd`, `RandScaleIntensityFixedMeand`, `RandScaleIntensityd`, `RandAdjustContrastd`, `RandHistogramShiftd`\n- Reduce quality: `RandSimulateLowResolutiond`\n- Dropout: `RandCoarseDropoutKeepKeypoints`\n\nMixup and Cutmix were tried but bring no clear improvement so be ignored in latter experiments.\n\n