Hi Everyone,
\nIt seems there's a lot of emphasis on the threshold in this competition. I'm very new to segmentation problems, so I'm wondering how people have been choosing theirs.
\nThe strategy I have taken is to always set the mask with >= 0.5 as positive (1) and < 0.5 as negative (0). Intuitively this makes sense to me because the output of my final layer is sigmoidal, so the center point of that function seems like a reasonable set point. Is the idea to change the threshold to something else based on the thought that if your model is prone to false negatives, you want relax the constraints on the model and if your model is prone to false positives, you want to be more restrictive?
\nHow have you approached determining robust thresholds?
", "messages": [ { "id": "2618726", "postDate": "01/25/2024 00:10:54", "content": "Hi Everyone,
\nIt seems there's a lot of emphasis on the threshold in this competition. I'm very new to segmentation problems, so I'm wondering how people have been choosing theirs.
\nThe strategy I have taken is to always set the mask with >= 0.5 as positive (1) and < 0.5 as negative (0). Intuitively this makes sense to me because the output of my final layer is sigmoidal, so the center point of that function seems like a reasonable set point. Is the idea to change the threshold to something else based on the thought that if your model is prone to false negatives, you want relax the constraints on the model and if your model is prone to false positives, you want to be more restrictive?
\nHow have you approached determining robust thresholds?
", "rawMarkdown": "Hi Everyone,\n\nIt seems there's a lot of emphasis on the threshold in this competition. I'm very new to segmentation problems, so I'm wondering how people have been choosing theirs.\n\nThe strategy I have taken is to always set the mask with >= 0.5 as positive (1) and < 0.5 as negative (0). Intuitively this makes sense to me because the output of my final layer is sigmoidal, so the center point of that function seems like a reasonable set point. Is the idea to change the threshold to something else based on the thought that if your model is prone to false negatives, you want relax the constraints on the model and if your model is prone to false positives, you want to be more restrictive?\n\nHow have you approached determining robust thresholds?", "votes": null }, { "id": "2618949", "postDate": "01/25/2024 05:42:43", "content": "If you believe in lb, you can choose the same threshold as lb, or you can filter by proportion like this notebook
\nTH=[x.flatten().numpy() for x in output]\nTH=np.concatenate(TH)\nindex = -int(len(TH) * CFG.th_percentile)\nTH:int = np.partition(TH, index)[index]
\nhttps://www.kaggle.com/code/misakimatsutomo/inference-1024-should-have-a-percentile-of-0-00149.
My method has similar proportions of threshold in public test and private test.
\nIf you believe in cv, you can choose the same threshold as cv.
\nCertainly, there is also an element of chance in play as well. Wishing you the best of luck.😀
Thanks @tanxxx :) best of luck to you too! Looks like you have a good chance of placing well if there isn't too much shakeup.
\nAh, so it's a matter of \"which so you trust\". That makes sense, thanks for the advice!
\nI don't think I completely understand how the thresholding code above works. I'll have to read it more closely this afternoon.
", "rawMarkdown": "Thanks @tanxxx :) best of luck to you too! Looks like you have a good chance of placing well if there isn't too much shakeup.\n\nAh, so it's a matter of \"which so you trust\". That makes sense, thanks for the advice!\n\nI don't think I completely understand how the thresholding code above works. I'll have to read it more closely this afternoon.", "votes": null } ], "comments": [ { "id": 2618949, "author_name": "tanxxx", "author_url": "", "post_date": "01/25/2024 05:42:43", "content": "If you believe in lb, you can choose the same threshold as lb, or you can filter by proportion like this notebook
\nTH=[x.flatten().numpy() for x in output]\nTH=np.concatenate(TH)\nindex = -int(len(TH) * CFG.th_percentile)\nTH:int = np.partition(TH, index)[index]
\nhttps://www.kaggle.com/code/misakimatsutomo/inference-1024-should-have-a-percentile-of-0-00149.
My method has similar proportions of threshold in public test and private test.
\nIf you believe in cv, you can choose the same threshold as cv.
\nCertainly, there is also an element of chance in play as well. Wishing you the best of luck.😀
Thanks @tanxxx :) best of luck to you too! Looks like you have a good chance of placing well if there isn't too much shakeup.
\nAh, so it's a matter of \"which so you trust\". That makes sense, thanks for the advice!
\nI don't think I completely understand how the thresholding code above works. I'll have to read it more closely this afternoon.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2618726": "Hi Everyone,\n\nIt seems there's a lot of emphasis on the threshold in this competition. I'm very new to segmentation problems, so I'm wondering how people have been choosing theirs.\n\nThe strategy I have taken is to always set the mask with >= 0.5 as positive (1) and < 0.5 as negative (0). Intuitively this makes sense to me because the output of my final layer is sigmoidal, so the center point of that function seems like a reasonable set point. Is the idea to change the threshold to something else based on the thought that if your model is prone to false negatives, you want relax the constraints on the model and if your model is prone to false positives, you want to be more restrictive?\n\nHow have you approached determining robust thresholds?", "2618949": "If you believe in lb, you can choose the same threshold as lb, or you can filter by proportion like this notebook \n`TH=[x.flatten().numpy() for x in output]\nTH=np.concatenate(TH)\nindex = -int(len(TH) * CFG.th_percentile)\nTH:int = np.partition(TH, index)[index]`\n[https://www.kaggle.com/code/misakimatsutomo/inference-1024-should-have-a-percentile-of-0-00149.](url)\n\nMy method has similar proportions of threshold in public test and private test. \nIf you believe in cv, you can choose the same threshold as cv.\nCertainly, there is also an element of chance in play as well. Wishing you the best of luck.😀", "2619440": "Thanks @tanxxx :) best of luck to you too! Looks like you have a good chance of placing well if there isn't too much shakeup.\n\nAh, so it's a matter of \"which so you trust\". That makes sense, thanks for the advice!\n\nI don't think I completely understand how the thresholding code above works. I'll have to read it more closely this afternoon." }, "source": "meta" }