Currently, I am developing ensemble functions for object localization. Here, I’d like to share some brief insights on the methods I’ve explored so far:
\nHDBSCAN is an effective tool for handling clusters with varying shapes and densities. I initially tried it and am now conducting deeper experiments on its properties for object localization.
\nthreshold from your model, min_cluster_size, and min_samples from HDBSCAN provide flexible control over prediction strength. Particularly effective for models trained with Gaussian balls or other irregularly shaped labels (e.g., keypoints annotated on motors or bacteria). I find it works especially well in 3D models, where the additional z-dimension improves clustering.| methods | \nworse case LB | \nbest case LB | \n
|---|---|---|
| average | \n0.69 | \n0.8 | \n
| maximum | \n0.81 | \n0.84 | \n
| clustering | \n0.76 | \n0.84 | \n
After reviewing the YOLO predictions using the maximum function and hdbscan, I found that YOLO actually produces many peak signals on the hidden test set (including almost all public notebooks and my best local model). This suggests that many high scores are the result of lucky predictions.
", "messages": [ { "id": 3198136, "postDate": "2025-05-09T05:12:26.843Z", "content": "Currently, I am developing ensemble functions for object localization. Here, I’d like to share some brief insights on the methods I’ve explored so far:
\nHDBSCAN is an effective tool for handling clusters with varying shapes and densities. I initially tried it and am now conducting deeper experiments on its properties for object localization.
\nthreshold from your model, min_cluster_size, and min_samples from HDBSCAN provide flexible control over prediction strength. Particularly effective for models trained with Gaussian balls or other irregularly shaped labels (e.g., keypoints annotated on motors or bacteria). I find it works especially well in 3D models, where the additional z-dimension improves clustering.| methods | \nworse case LB | \nbest case LB | \n
|---|---|---|
| average | \n0.69 | \n0.8 | \n
| maximum | \n0.81 | \n0.84 | \n
| clustering | \n0.76 | \n0.84 | \n
After reviewing the YOLO predictions using the maximum function and hdbscan, I found that YOLO actually produces many peak signals on the hidden test set (including almost all public notebooks and my best local model). This suggests that many high scores are the result of lucky predictions.
", "rawMarkdown": "Currently, I am developing ensemble functions for object localization. Here, I’d like to share some brief insights on the methods I’ve explored so far:\n\n# Average & Weighted Sum\n* Works well when your models have low diversity.\n* Performance may degrade if your models are highly diverse.\n* Weight selection is done through brute-force search.\n\n# Maximum Function\n* Ensures that object presence is not missed.\n* Crucially, it allows you to detect false positives from individual models — if the overall score drops after adding another model, it suggests that the new model is predicting false positives.\n\n# Clustering (HDBSCAN)\nHDBSCAN is an effective tool for handling clusters with varying shapes and densities. I initially tried it and am now conducting deeper experiments on its properties for object localization.\n* If a model predicts multiple keypoints in an area, there is a high possibility that an object is present.\n* Suppresses peak signals, reducing false positives. It also helps verify whether your current score improvements rely on peak signals, which can cause instability and shake-up.\n* Parameters like `threshold` from your model, `min_cluster_size`, and `min_samples` from HDBSCAN provide flexible control over prediction strength. Particularly effective for models trained with Gaussian balls or other irregularly shaped labels (e.g., keypoints annotated on motors or bacteria). I find it works especially well in 3D models, where the additional z-dimension improves clustering.\n* Helps address prediction diversity across multiple models.\n\n| methods | worse case LB | best case LB |\n| --- | --- |\n| average |0.69|0.8|\n| maximum |0.81|0.84|\n| clustering |0.76|0.84|\n\n# Notes\nAfter reviewing the YOLO predictions using the maximum function and hdbscan, I found that YOLO actually produces many peak signals on the hidden test set (including almost all public notebooks and my best local model). This suggests that many high scores are the result of lucky predictions.", "votes": 37 }, { "id": 3206242, "postDate": "2025-05-21T04:34:01.907Z", "content": "Additional tips in HDBSCAN:
\nAssume we have two predictions from two models, with each model predicting a single keypoint in 3D volume. Setting min_samples=1 and min_cluster_size=2 allows HDBSCAN to preserve similar predictions made by both models without needing prediction multiple points at the target.
| Setting | \nValue | \nBehavior | \n
|---|---|---|
min_samples | \n1 | \nAlmost every point can be a core point → high recall, permissive clustering | \n
min_cluster_size | \n5 | \nClusters must have ≥5 points → filters out small/noisy clusters | \n
If we can ensemble a large number of models, we gain more flexibility to control the recall rate, making the clustering process more robust.
", "rawMarkdown": "Additional tips in HDBSCAN:\n\n### Preserving basic predictions\nAssume we have two predictions from two models, with each model predicting a single keypoint in 3D volume. Setting `min_samples=1` and `min_cluster_size=2` allows HDBSCAN to preserve similar predictions made by both models without needing prediction multiple points at the target.\n\n### One neighbor as core point with five minimum cluster capacity:\n| Setting | Value | Behavior |\n| --- | --- |\n| `min_samples` | 1 | Almost every point can be a core point → high recall, permissive clustering |\n| `min_cluster_size` | 5 | Clusters must have ≥5 points → filters out small/noisy clusters |\n\nIf we can ensemble a large number of models, we gain more flexibility to control the recall rate, making the clustering process more robust.", "votes": 1, "replies": [ { "id": 3206403, "postDate": "2025-05-21T08:54:20.330Z", "content": "\n\nAssume we have two predictions from two models, with each model predicting a single keypoint in 3D volume. Setting min_samples=1 and min_cluster_size=2
\n
For both models you take the highest confidence prediction in 3d then ? This config means if both predictions are close it's motor else it's noise. If I understood correctly this will lead to miss true positives
\n\n\nIf we can ensemble a large number of models, we gain more flexibility to control the recall rate, making the clustering process more robust.
\n
It doesn't mean we will be able to catch more motors. Depending on your models, the noise clusters might actually get bigger :/
\nI noticed that it's not easy to tune hdbscan/dbscan and I think that there is a high risk of overfitting to the public lb
For both models you take highest confidence prediction d ? This config means both predictions are s noise. If I understood correctly this will lead miss positives\nYou can duplicate your prediction and plus 1e-4. This secure your prediction.
\nIt doesn\nI noticed that it\nYeah the biggest problem is tunning the hyperparameter. We need to balance FP and FN through the parameter tunning. I prefer accepting more FP since the metric is fbeta score with beta=2.
\nFurthermore, developing how to select the cluster is also a challenge, currently I was trying confidence-based selection, but it turns out that cluster-sized-based selection is better. This needs further development.
", "rawMarkdown": "@victorvannobel \n\n```\nFor both models you take the highest confidence prediction in 3d then ? This config means if both predictions are close it's motor else it's noise. If I understood correctly this will lead to miss true positives\n```\nYou can duplicate your prediction and plus 1e-4. This secure your prediction.\n\n```\nIt doesn't mean we will be able to catch more motors. Depending on your models, the noise clusters might actually get bigger :/\nI noticed that it's not easy to tune hdbscan/dbscan and I think that there is a high risk of overfitting to the public lb\n```\nYeah the biggest problem is tunning the hyperparameter. We need to balance FP and FN through the parameter tunning. I prefer accepting more FP since the metric is fbeta score with beta=2.\n\nFurthermore, developing how to select the cluster is also a challenge, currently I was trying confidence-based selection, but it turns out that cluster-sized-based selection is better. This needs further development." }, { "id": 3207061, "postDate": "2025-05-22T07:02:49.313Z", "content": "What is the image size for your model? I tried using DBSCAN but got only 0.765
", "rawMarkdown": "What is the image size for your model? I tried using DBSCAN but got only 0.765" } ] } ] }, { "id": 3200048, "postDate": "2025-05-12T00:13:24.490Z", "content": "Thanks for your insights! I'm a beginner, so I have some basic questions.
\nDo you do the ensembling before or after 3D NMS?
\nI was just ensembling by taking all the predictions of all the models and treating that as the effectively single model and then doing 3D NMS. But I have much less control than the methods you mentioned as a result…. Is that the same as the Maximum method?
\nAnd how do you predict worst case and best case LB?
\nAlso maximum seems is working just as well , if not better than clustering … I would guess that because clustering will also take the Z axis into account better, I'll expect it to have better performance but it isn't the case ?
\nAnd what is the Gaussian ball model ? I have been trying 3D models and keypoint detectors, but they seem to be much harder to train and to learn something meaningful compared to the templated yolo model.
@iamudit The 3D NMS implemented in the public notebook is effectively equivalent to simply taking the maximum. The filtering step is redundant because they directly select final_detection = detection[0].
If you're doing clustering, you need to ensure that there are enough detected points; otherwise, clustering won’t work. Clustering is actually better than taking the maximum when you have 10+ ensemble models. Selecting the cluster with the largest size is similar to majority voting.
\nI understand it's quite challenging to complete predictions with 10+ models within 12 hours. That’s why I developed a two-stage method: the ensemble operates in the second stage, and the input at that stage is much smaller compared to the raw volume.
\nAlso, you can’t effectively train a 3D model by providing only a single point—there's too much semantic ambiguity. A simple improvement for the labels is to annotate all points within 1000 angstroms of the target point as positive. Using a Gaussian ball follows the same idea but instead of assigning hard labels, it gives a soft, continuous score—like applying a 3D Gaussian function.
", "rawMarkdown": "@iamudit The 3D NMS implemented in the public notebook is effectively equivalent to simply taking the maximum. The filtering step is redundant because they directly select `final_detection = detection[0]`.\n\nIf you're doing clustering, you need to ensure that there are enough detected points; otherwise, clustering won’t work. Clustering is actually better than taking the maximum when you have 10+ ensemble models. Selecting the cluster with the largest size is similar to majority voting.\n\nI understand it's quite challenging to complete predictions with 10+ models within 12 hours. That’s why I developed a two-stage method: the ensemble operates in the second stage, and the input at that stage is much smaller compared to the raw volume.\n\nAlso, you can’t effectively train a 3D model by providing only a single point—there's too much semantic ambiguity. A simple improvement for the labels is to annotate all points within 1000 angstroms of the target point as positive. Using a Gaussian ball follows the same idea but instead of assigning hard labels, it gives a soft, continuous score—like applying a 3D Gaussian function.\n", "votes": 4, "replies": [ { "id": 3201130, "postDate": "2025-05-13T12:55:55.673Z", "content": "@tom99763 Thank you for your explanation~!
\nI see, 10+ ensemble models is impressive and at that point with the high number of detections,clustering would be much better than a simple maximum.
\nI'm doing a similar staging as you and still, currently fitting in more than 2 models within the time limit seems challenging, I guess lowering the image resolution would increase the speed and therefore the raw number of models, (apart from discarding many slices till multiple models fit) l (but also compromise the prediction accuracies ) so there is a LOT of experimentation to be done to determine the best configuration !!!
Thanks for explaining the 3D model and the Gaussian ball. Do you think it has the potential to beat the 2 stage model ? where the first stage has a (comparatively)simple task and 2nd stage has a simple task (both in lower dimensionality) . These 3D models suffer from the curse of dimensionality.
\nBut I completely agree that if trained well enough and made light enough, it would be a viable candidate in the ensemble pipeline.
@iamudit My solution is two-stage + 3d (I have published the initial idea of my solution), but without using traditional 3D models like UNet or SegResNet. I found an effective way to replicate the functionality of those models while keeping the architecture extremely lightweight. Even with over 10 models in the pipeline, the total inference time remains around 7 hours which is the same as running a single model.
\nI'm still finding more models to stack, thinking that 20+ models is possible.
", "rawMarkdown": "@iamudit My solution is two-stage + 3d (I have published [the initial idea of my solution](https://www.kaggle.com/code/tom99763/first-idea-to-extract-keypoints-byu)), but without using traditional 3D models like UNet or SegResNet. I found an effective way to replicate the functionality of those models while keeping the architecture extremely lightweight. Even with over 10 models in the pipeline, the total inference time remains around 7 hours which is the same as running a single model.\n\nI'm still finding more models to stack, thinking that 20+ models is possible.", "votes": 5 }, { "id": 3202231, "postDate": "2025-05-15T04:31:52.157Z", "content": "Thanks!! It looks like a very interesting solution. Looking forward to your solution writeup! :)
", "rawMarkdown": "Thanks!! It looks like a very interesting solution. Looking forward to your solution writeup! :)", "votes": 2 } ] } ] }, { "id": 3201017, "postDate": "2025-05-13T10:55:45.087Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 3206242, "author_name": "Tom", "author_url": "", "post_date": "2025-05-21T04:34:01.907000", "content": "Additional tips in HDBSCAN:
\nAssume we have two predictions from two models, with each model predicting a single keypoint in 3D volume. Setting min_samples=1 and min_cluster_size=2 allows HDBSCAN to preserve similar predictions made by both models without needing prediction multiple points at the target.
| Setting | \nValue | \nBehavior | \n
|---|---|---|
min_samples | \n1 | \nAlmost every point can be a core point → high recall, permissive clustering | \n
min_cluster_size | \n5 | \nClusters must have ≥5 points → filters out small/noisy clusters | \n
If we can ensemble a large number of models, we gain more flexibility to control the recall rate, making the clustering process more robust.
", "votes": 1, "replies": [ { "id": 3206403, "author_name": "Victor", "author_url": "", "post_date": "2025-05-21T08:54:20.330000", "content": "\n\nAssume we have two predictions from two models, with each model predicting a single keypoint in 3D volume. Setting min_samples=1 and min_cluster_size=2
\n
For both models you take the highest confidence prediction in 3d then ? This config means if both predictions are close it's motor else it's noise. If I understood correctly this will lead to miss true positives
\n\n\nIf we can ensemble a large number of models, we gain more flexibility to control the recall rate, making the clustering process more robust.
\n
It doesn't mean we will be able to catch more motors. Depending on your models, the noise clusters might actually get bigger :/
\nI noticed that it's not easy to tune hdbscan/dbscan and I think that there is a high risk of overfitting to the public lb
For both models you take highest confidence prediction d ? This config means both predictions are s noise. If I understood correctly this will lead miss positives\nYou can duplicate your prediction and plus 1e-4. This secure your prediction.
\nIt doesn\nI noticed that it\nYeah the biggest problem is tunning the hyperparameter. We need to balance FP and FN through the parameter tunning. I prefer accepting more FP since the metric is fbeta score with beta=2.
\nFurthermore, developing how to select the cluster is also a challenge, currently I was trying confidence-based selection, but it turns out that cluster-sized-based selection is better. This needs further development.
", "votes": 0, "replies": [] }, { "id": 3207061, "author_name": "Rustam Bazarbayev", "author_url": "", "post_date": "2025-05-22T07:02:49.313000", "content": "What is the image size for your model? I tried using DBSCAN but got only 0.765
", "votes": 0, "replies": [] } ] } ] }, { "id": 3200048, "author_name": "Udit Jain", "author_url": "", "post_date": "2025-05-12T00:13:24.490000", "content": "Thanks for your insights! I'm a beginner, so I have some basic questions.
\nDo you do the ensembling before or after 3D NMS?
\nI was just ensembling by taking all the predictions of all the models and treating that as the effectively single model and then doing 3D NMS. But I have much less control than the methods you mentioned as a result…. Is that the same as the Maximum method?
\nAnd how do you predict worst case and best case LB?
\nAlso maximum seems is working just as well , if not better than clustering … I would guess that because clustering will also take the Z axis into account better, I'll expect it to have better performance but it isn't the case ?
\nAnd what is the Gaussian ball model ? I have been trying 3D models and keypoint detectors, but they seem to be much harder to train and to learn something meaningful compared to the templated yolo model.
@iamudit The 3D NMS implemented in the public notebook is effectively equivalent to simply taking the maximum. The filtering step is redundant because they directly select final_detection = detection[0].
If you're doing clustering, you need to ensure that there are enough detected points; otherwise, clustering won’t work. Clustering is actually better than taking the maximum when you have 10+ ensemble models. Selecting the cluster with the largest size is similar to majority voting.
\nI understand it's quite challenging to complete predictions with 10+ models within 12 hours. That’s why I developed a two-stage method: the ensemble operates in the second stage, and the input at that stage is much smaller compared to the raw volume.
\nAlso, you can’t effectively train a 3D model by providing only a single point—there's too much semantic ambiguity. A simple improvement for the labels is to annotate all points within 1000 angstroms of the target point as positive. Using a Gaussian ball follows the same idea but instead of assigning hard labels, it gives a soft, continuous score—like applying a 3D Gaussian function.
", "votes": 4, "replies": [ { "id": 3201130, "author_name": "Udit Jain", "author_url": "", "post_date": "2025-05-13T12:55:55.673000", "content": "@tom99763 Thank you for your explanation~!
\nI see, 10+ ensemble models is impressive and at that point with the high number of detections,clustering would be much better than a simple maximum.
\nI'm doing a similar staging as you and still, currently fitting in more than 2 models within the time limit seems challenging, I guess lowering the image resolution would increase the speed and therefore the raw number of models, (apart from discarding many slices till multiple models fit) l (but also compromise the prediction accuracies ) so there is a LOT of experimentation to be done to determine the best configuration !!!
Thanks for explaining the 3D model and the Gaussian ball. Do you think it has the potential to beat the 2 stage model ? where the first stage has a (comparatively)simple task and 2nd stage has a simple task (both in lower dimensionality) . These 3D models suffer from the curse of dimensionality.
\nBut I completely agree that if trained well enough and made light enough, it would be a viable candidate in the ensemble pipeline.
@iamudit My solution is two-stage + 3d (I have published the initial idea of my solution), but without using traditional 3D models like UNet or SegResNet. I found an effective way to replicate the functionality of those models while keeping the architecture extremely lightweight. Even with over 10 models in the pipeline, the total inference time remains around 7 hours which is the same as running a single model.
\nI'm still finding more models to stack, thinking that 20+ models is possible.
", "votes": 5, "replies": [] }, { "id": 3202231, "author_name": "Udit Jain", "author_url": "", "post_date": "2025-05-15T04:31:52.157000", "content": "Thanks!! It looks like a very interesting solution. Looking forward to your solution writeup! :)
", "votes": 2, "replies": [] } ] } ] }, { "id": 3201017, "author_name": "", "author_url": "", "post_date": "2025-05-13T10:55:45.087000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "3198136": "Currently, I am developing ensemble functions for object localization. Here, I’d like to share some brief insights on the methods I’ve explored so far:\n\n# Average & Weighted Sum\n* Works well when your models have low diversity.\n* Performance may degrade if your models are highly diverse.\n* Weight selection is done through brute-force search.\n\n# Maximum Function\n* Ensures that object presence is not missed.\n* Crucially, it allows you to detect false positives from individual models — if the overall score drops after adding another model, it suggests that the new model is predicting false positives.\n\n# Clustering (HDBSCAN)\nHDBSCAN is an effective tool for handling clusters with varying shapes and densities. I initially tried it and am now conducting deeper experiments on its properties for object localization.\n* If a model predicts multiple keypoints in an area, there is a high possibility that an object is present.\n* Suppresses peak signals, reducing false positives. It also helps verify whether your current score improvements rely on peak signals, which can cause instability and shake-up.\n* Parameters like `threshold` from your model, `min_cluster_size`, and `min_samples` from HDBSCAN provide flexible control over prediction strength. Particularly effective for models trained with Gaussian balls or other irregularly shaped labels (e.g., keypoints annotated on motors or bacteria). I find it works especially well in 3D models, where the additional z-dimension improves clustering.\n* Helps address prediction diversity across multiple models.\n\n| methods | worse case LB | best case LB |\n| --- | --- |\n| average |0.69|0.8|\n| maximum |0.81|0.84|\n| clustering |0.76|0.84|\n\n# Notes\nAfter reviewing the YOLO predictions using the maximum function and hdbscan, I found that YOLO actually produces many peak signals on the hidden test set (including almost all public notebooks and my best local model). This suggests that many high scores are the result of lucky predictions.", "3206242": "Additional tips in HDBSCAN:\n\n### Preserving basic predictions\nAssume we have two predictions from two models, with each model predicting a single keypoint in 3D volume. Setting `min_samples=1` and `min_cluster_size=2` allows HDBSCAN to preserve similar predictions made by both models without needing prediction multiple points at the target.\n\n### One neighbor as core point with five minimum cluster capacity:\n| Setting | Value | Behavior |\n| --- | --- |\n| `min_samples` | 1 | Almost every point can be a core point → high recall, permissive clustering |\n| `min_cluster_size` | 5 | Clusters must have ≥5 points → filters out small/noisy clusters |\n\nIf we can ensemble a large number of models, we gain more flexibility to control the recall rate, making the clustering process more robust.", "3200048": "Thanks for your insights! I'm a beginner, so I have some basic questions. \nDo you do the ensembling before or after 3D NMS? \nI was just ensembling by taking all the predictions of all the models and treating that as the effectively single model and then doing 3D NMS. But I have much less control than the methods you mentioned as a result.... Is that the same as the Maximum method?\nAnd how do you predict worst case and best case LB? \nAlso maximum seems is working just as well , if not better than clustering ... I would guess that because clustering will also take the Z axis into account better, I'll expect it to have better performance but it isn't the case ? \nAnd what is the Gaussian ball model ? I have been trying 3D models and keypoint detectors, but they seem to be much harder to train and to learn something meaningful compared to the templated yolo model. ", "3201017": "" } }