i make some submission and do some simple test, it is obviously that:
\n(1) your model give has lower lb then cv because of less detection (not more FP). your lb improve if you lower your threshold.
\n(2) despite domain shift, most model are good at predicting negative label (background image)
(3) if the false accept rate is low or zero, then you can do test probing. e.g. if you flip the image intensity and if there is some \"lb score\", you can conclude some image have intensity flip (like the ones in the train). you can proceed to test for larger scale, smaller scale, blur, etc.. in short you can probe test domain shift.
\nI recall a kaggle competition for coral reef object detection where is hidden test label has \"bigger aspect box label\" (test and train labels are differently). while many kagglers are puzzled why their lb is worse than lb, some kagglers find/probe this in the competition
\n(3) normally if your model works robust for image +noise, it is resilient to domain shift. try to use noise augmentation.
\n(4) what is fatal to model:
\ngood luck!
\nif you are developing AI commercial product, you imagine your customer problem and think of how to develop AI tech to help them.
\nIf you are writing AI paper, you imagine who others will use your work to advance even more and you think of solving fundamental problems
\nif you are taking part in AI competition, you imagine how the hidden test data will look like and think of how that can help you to be better than others
", "messages": [ { "id": 3176187, "postDate": "2025-04-11T03:51:33.477Z", "content": "i make some submission and do some simple test, it is obviously that:
\n(1) your model give has lower lb then cv because of less detection (not more FP). your lb improve if you lower your threshold.
\n(2) despite domain shift, most model are good at predicting negative label (background image)
(3) if the false accept rate is low or zero, then you can do test probing. e.g. if you flip the image intensity and if there is some \"lb score\", you can conclude some image have intensity flip (like the ones in the train). you can proceed to test for larger scale, smaller scale, blur, etc.. in short you can probe test domain shift.
\nI recall a kaggle competition for coral reef object detection where is hidden test label has \"bigger aspect box label\" (test and train labels are differently). while many kagglers are puzzled why their lb is worse than lb, some kagglers find/probe this in the competition
\n(3) normally if your model works robust for image +noise, it is resilient to domain shift. try to use noise augmentation.
\n(4) what is fatal to model:
\ngood luck!
\nif you are developing AI commercial product, you imagine your customer problem and think of how to develop AI tech to help them.
\nIf you are writing AI paper, you imagine who others will use your work to advance even more and you think of solving fundamental problems
\nif you are taking part in AI competition, you imagine how the hidden test data will look like and think of how that can help you to be better than others
", "rawMarkdown": "i make some submission and do some simple test, it is obviously that:\n(1) your model give has lower lb then cv because of less detection (not more FP). your lb improve if you lower your threshold.\n(2) despite domain shift, most model are good at predicting negative label (background image)\n- you can prove this by: set very low threshold. make submission for object only for <0.1, then for <0.2 ..... we want to find the point when lb stop giving zero (lb is zero when both precision and recall are zero). compare this with your training.\n\n(3) if the false accept rate is low or zero, then you can do test probing. e.g. if you flip the image intensity and if there is some \"lb score\", you can conclude some image have intensity flip (like the ones in the train). you can proceed to test for larger scale, smaller scale, blur, etc.. in short you can probe test domain shift.\n\nI recall a kaggle competition for coral reef object detection where is hidden test label has \"bigger aspect box label\" (test and train labels are differently). while many kagglers are puzzled why their lb is worse than lb, some kagglers find/probe this in the competition\n\n(3) normally if your model works robust for image +noise, it is resilient to domain shift. try to use noise augmentation.\n\n(4) what is fatal to model:\n- when test scale is different from train\n- when test intensity level is different from train (different histogram distribution)\n- when test noise level is different from train\n- artifacts in test\n- different object structure/appearance in test\n\ngood luck!\n\n\n----\n\nif you are developing AI commercial product, you imagine your customer problem and think of how to develop AI tech to help them.\n\nIf you are writing AI paper, you imagine who others will use your work to advance even more and you think of solving fundamental problems\n\nif you are taking part in AI competition, you imagine how the hidden test data will look like and think of how that can help you to be better than others", "votes": 49 }, { "id": 3177086, "postDate": "2025-04-12T08:25:41.217Z", "content": "In my case, my model development is based on keypoint detection (like orb in opencv, I think it ignores domain shift cause it is not a ml model). After probing many times, I found that my current model detects too much false positives on hidden testset. But its recall is great on my local test, almost cannot miss any motor.
", "rawMarkdown": "In my case, my model development is based on keypoint detection (like orb in opencv, I think it ignores domain shift cause it is not a ml model). After probing many times, I found that my current model detects too much false positives on hidden testset. But its recall is great on my local test, almost cannot miss any motor.", "votes": 3, "replies": [ { "id": 3178100, "postDate": "2025-04-13T17:58:37.853Z", "content": "Yeah from what I have heard about your solutions for this comp so far I am super excited to see your work this time around!
", "rawMarkdown": "Yeah from what I have heard about your solutions for this comp so far I am super excited to see your work this time around! " } ] }, { "id": 3177501, "postDate": "2025-04-12T21:24:15.173Z", "content": "Super insightful! Love the practical tricks—especially the probing methods and the mindset tips. Really sharp advice . Thanks for sharing!
", "rawMarkdown": "Super insightful! Love the practical tricks—especially the probing methods and the mindset tips. Really sharp advice . Thanks for sharing!", "votes": 1 }, { "id": 3176255, "postDate": "2025-04-11T06:38:32.080Z", "content": "This is more evident in my current Lb. I tried lowering the threshold by 0.05, lb 0.610 -> 0.725, but I know there is still a lot more I need to do.
\n\n", "rawMarkdown": "This is more evident in my current Lb. I tried lowering the threshold by 0.05, lb 0.610 -> 0.725, but I know there is still a lot more I need to do.\n>(1) your model give has lower lb then cv because of less detection (not more FP). your lb improve if you lower your threshold.", "votes": 1 }, { "id": 3180011, "postDate": "2025-04-16T03:17:51.563Z", "content": "(1) your model give has lower lb then cv because of less detection (not more FP). your lb improve if you lower your threshold.
\n
maybe you know the % of pos samples in the test. then you can do dynamic thresholding
", "rawMarkdown": "maybe you know the % of pos samples in the test. then you can do dynamic thresholding", "votes": 2 }, { "id": 3180062, "postDate": "2025-04-16T04:35:44.103Z", "content": "Amazing and informative
", "rawMarkdown": "Amazing and informative " }, { "id": 3176416, "postDate": "2025-04-11T10:21:21.783Z", "content": "I've been experimenting with the conf thresholds for several day by far. I think I found a way to get the optimal LB threshold based on cv only. Tested that on 4 different pipelines (for each one i submitted several thresholds and the one my cv says is optimal) and it gave best results in all. But choosing which model is better based on cv is still not solved for me. So i think we need to depend on lb a bit :)
", "rawMarkdown": "I've been experimenting with the conf thresholds for several day by far. I think I found a way to get the optimal LB threshold based on cv only. Tested that on 4 different pipelines (for each one i submitted several thresholds and the one my cv says is optimal) and it gave best results in all. But choosing which model is better based on cv is still not solved for me. So i think we need to depend on lb a bit :)", "replies": [ { "id": 3176437, "postDate": "2025-04-11T10:50:00.710Z", "content": "wokring on threshold will not win competition.
\nmaking model insensitive/robust against threshold will
In my case, my model development is based on keypoint detection (like orb in opencv, I think it ignores domain shift cause it is not a ml model). After probing many times, I found that my current model detects too much false positives on hidden testset. But its recall is great on my local test, almost cannot miss any motor.
", "votes": 3, "replies": [ { "id": 3178100, "author_name": "Cody_Null", "author_url": "", "post_date": "2025-04-13T17:58:37.853000", "content": "Yeah from what I have heard about your solutions for this comp so far I am super excited to see your work this time around!
", "votes": 0, "replies": [] } ] }, { "id": 3177501, "author_name": "MEHUL MRIDUL", "author_url": "", "post_date": "2025-04-12T21:24:15.173000", "content": "Super insightful! Love the practical tricks—especially the probing methods and the mindset tips. Really sharp advice . Thanks for sharing!
", "votes": 1, "replies": [] }, { "id": 3176255, "author_name": "colt", "author_url": "", "post_date": "2025-04-11T06:38:32.080000", "content": "This is more evident in my current Lb. I tried lowering the threshold by 0.05, lb 0.610 -> 0.725, but I know there is still a lot more I need to do.
\n\n", "votes": 1, "replies": [] }, { "id": 3180011, "author_name": "hengck23", "author_url": "", "post_date": "2025-04-16T03:17:51.563000", "content": "(1) your model give has lower lb then cv because of less detection (not more FP). your lb improve if you lower your threshold.
\n
maybe you know the % of pos samples in the test. then you can do dynamic thresholding
", "votes": 2, "replies": [] }, { "id": 3180062, "author_name": "Memoona Qaiser", "author_url": "", "post_date": "2025-04-16T04:35:44.103000", "content": "Amazing and informative
", "votes": 0, "replies": [] }, { "id": 3176416, "author_name": "Mohamed Eltayeb", "author_url": "", "post_date": "2025-04-11T10:21:21.783000", "content": "I've been experimenting with the conf thresholds for several day by far. I think I found a way to get the optimal LB threshold based on cv only. Tested that on 4 different pipelines (for each one i submitted several thresholds and the one my cv says is optimal) and it gave best results in all. But choosing which model is better based on cv is still not solved for me. So i think we need to depend on lb a bit :)
", "votes": 0, "replies": [ { "id": 3176437, "author_name": "hengck23", "author_url": "", "post_date": "2025-04-11T10:50:00.710000", "content": "wokring on threshold will not win competition.
\nmaking model insensitive/robust against threshold will