The conf threshold I am currently using for the LB maximum (0.547, YOLOv5) is 0.13.
\nEven though F2score emphasizes Recall, I'm wondering if the number in this conf is too low. Overfitting to PublicLB?
The conf threshold I am currently using for the LB maximum (0.547, YOLOv5) is 0.13.
\nEven though F2score emphasizes Recall, I'm wondering if the number in this conf is too low. Overfitting to PublicLB?
The confidence threshold value itself isn’t a problem; it depends on the model (low conf threshold is no problem if the model can distinguish positive and negative bounding box well).
\nThe problem is, the parameter fitting your validation set has generally applicable to public/private test set.
For me, the discussion below has much insight:
\nhttps://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/298427
For other strategy, though it’s computationally intense, nested cross validation for fitting hyper parameter might reduce overfitting.
", "rawMarkdown": "The confidence threshold value itself isn’t a problem; it depends on the model (low conf threshold is no problem if the model can distinguish positive and negative bounding box well).\nThe problem is, the parameter fitting your validation set has generally applicable to public/private test set.\n\nFor me, the discussion below has much insight:\nhttps://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/298427\n\nFor other strategy, though it’s computationally intense, nested cross validation for fitting hyper parameter might reduce overfitting.", "votes": null }, { "id": "1639110", "postDate": "01/05/2022 11:39:42", "content": "the f2 formula is:
\ndef f_beta(tp, fp, fn, beta=2):\n f2 = (1 + beta ** 2) * tp / ((1 + beta ** 2) * tp + beta ** 2 * fn + fp)\n # f2 = 5 * tp / (5 * tp + 4 * fn + fp)\n\nreturn f2\nfor example, if i lower the conf and i get an improvement of one tp (and a corresponding reduction of one fn) but a degradation one more fp, i still gain in f2.
\nthis is because of the beta multiplier in the formula, one improvement of tp becomes 5x tp.
\nbut if lower the conf leads to e.g 5x more fp than tp, the f2 score degrades.
\n(the actual number may not be 5x, you should use python to make a a graph f2 vs tp,trp,fn)
for your trained model, you should have a graph of conf vs fp, conf vs tp , conf vs tn.
\nthen you should estimate be able to estimate the performance of your model in test (since you know how many test images and test objects)
Thanks for the interesting comments.
\nI will read the discussion you shared.
thank you.
\nWhen evaluating the validation data, I would like to examine the relationship with conf.
in theory, when detection is more important than fp, you can modify your binary cross entropy loss to weigh more on positive class.
\nalternatively, you can increase the population of positive objects by cut and paster / mosaic augmentation for one image.
\nthen the optimal confidence threshold should increase in value.
", "rawMarkdown": "in theory, when detection is more important than fp, you can modify your binary cross entropy loss to weigh more on positive class.\n\nalternatively, you can increase the population of positive objects by cut and paster / mosaic augmentation for one image.\n\nthen the optimal confidence threshold should increase in value.", "votes": null } ], "comments": [ { "id": 1638947, "author_name": "tatamikenn", "author_url": "", "post_date": "01/05/2022 07:31:31", "content": "The confidence threshold value itself isn’t a problem; it depends on the model (low conf threshold is no problem if the model can distinguish positive and negative bounding box well).
\nThe problem is, the parameter fitting your validation set has generally applicable to public/private test set.
For me, the discussion below has much insight:
\nhttps://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/298427
For other strategy, though it’s computationally intense, nested cross validation for fitting hyper parameter might reduce overfitting.
", "votes": null, "replies": [ { "id": 1639411, "author_name": "syurenuko", "author_url": "", "post_date": "01/05/2022 16:02:16", "content": "Thanks for the interesting comments.
\nI will read the discussion you shared.
the f2 formula is:
\ndef f_beta(tp, fp, fn, beta=2):\n f2 = (1 + beta ** 2) * tp / ((1 + beta ** 2) * tp + beta ** 2 * fn + fp)\n # f2 = 5 * tp / (5 * tp + 4 * fn + fp)\n\nreturn f2\nfor example, if i lower the conf and i get an improvement of one tp (and a corresponding reduction of one fn) but a degradation one more fp, i still gain in f2.
\nthis is because of the beta multiplier in the formula, one improvement of tp becomes 5x tp.
\nbut if lower the conf leads to e.g 5x more fp than tp, the f2 score degrades.
\n(the actual number may not be 5x, you should use python to make a a graph f2 vs tp,trp,fn)
for your trained model, you should have a graph of conf vs fp, conf vs tp , conf vs tn.
\nthen you should estimate be able to estimate the performance of your model in test (since you know how many test images and test objects)
thank you.
\nWhen evaluating the validation data, I would like to examine the relationship with conf.
in theory, when detection is more important than fp, you can modify your binary cross entropy loss to weigh more on positive class.
\nalternatively, you can increase the population of positive objects by cut and paster / mosaic augmentation for one image.
\nthen the optimal confidence threshold should increase in value.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1638833": "The conf threshold I am currently using for the LB maximum (0.547, YOLOv5) is 0.13.\nEven though F2score emphasizes Recall, I'm wondering if the number in this conf is too low. Overfitting to PublicLB?", "1638947": "The confidence threshold value itself isn’t a problem; it depends on the model (low conf threshold is no problem if the model can distinguish positive and negative bounding box well).\nThe problem is, the parameter fitting your validation set has generally applicable to public/private test set.\n\nFor me, the discussion below has much insight:\nhttps://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/298427\n\nFor other strategy, though it’s computationally intense, nested cross validation for fitting hyper parameter might reduce overfitting.", "1639110": "the f2 formula is:\n\n```\ndef f_beta(tp, fp, fn, beta=2):\n f2 = (1 + beta ** 2) * tp / ((1 + beta ** 2) * tp + beta ** 2 * fn + fp)\n # f2 = 5 * tp / (5 * tp + 4 * fn + fp)\n\nreturn f2\n\n```\n\nfor example, if i lower the conf and i get an improvement of one tp (and a corresponding reduction of one fn) but a degradation one more fp, i still gain in f2. \n\nthis is because of the beta multiplier in the formula, one improvement of tp becomes 5x tp.\n\nbut if lower the conf leads to e.g 5x more fp than tp, the f2 score degrades.\n(the actual number may not be 5x, you should use python to make a a graph f2 vs tp,trp,fn)\n\nfor your trained model, you should have a graph of conf vs fp, conf vs tp , conf vs tn.\nthen you should estimate be able to estimate the performance of your model in test (since you know how many test images and test objects)", "1639411": "Thanks for the interesting comments.\nI will read the discussion you shared.", "1639417": "thank you.\nWhen evaluating the validation data, I would like to examine the relationship with conf.", "1639550": "in theory, when detection is more important than fp, you can modify your binary cross entropy loss to weigh more on positive class.\n\nalternatively, you can increase the population of positive objects by cut and paster / mosaic augmentation for one image.\n\nthen the optimal confidence threshold should increase in value." }, "source": "meta" }