I added 10,000 FP bbox to one frame, and submit this model. After calculated LB, the score is dropped 0.006 (0.601 -> 0.595). Given the total frame number is around 3,500 (0.25 * 14,000), and number of GT label is ~2450[1], the score drop is apparently too little.
\nOn the contrary, if I add one FP bbox on every frame, the score is dropped as expected (0.601 -> 0.426).
\n\nWe have now highly rational explanation on this issue.
\nIt seems that the box threshold per image is set to ~129 to avoid system load.
Details are explained in this thread:
\nhttps://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/302632#1661980
These are clarified by Kaggle staff:
\n\n\nNo, the F2 score is calculated globally.
\n
\n There is a cutoff of 100 predictions per image. Is this low enough that it's relevant for your actual model outputs?
https://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/302632#1663027
\nThe test code I used is as below:
\nP_KEEP = 1.0\nCONF = 0.50\nFP_COUNT = 10_000\ndef drop_pred(bboxes, confs, p_keep):\n '''\n randomly drop prediction for the probability (1 - p_keep).\n '''\n if p_keep == 1:\n return bboxes, confs\n bboxes = bboxes.copy()\n confs = confs.copy()\n assert len(bboxes) == len(confs)\n pp = np.random.uniform(size=len(bboxes))\n bboxes = bboxes[pp <= p_keep]\n confs = confs[pp <= p_keep]\n return bboxes, confs\n\ndef add_fp(bboxes, confs, fp_count=0, fp_conf=1e-4):\n '''\n add false negative to the frame\n '''\n if fp_count == 0:\n return bboxes, confs\n bboxes = bboxes.copy()\n confs = confs.copy()\n assert len(bboxes) == len(confs)\n FPs = [[i % 1280, i // 1280, 1, 1] for i in range(fp_count)]\n bboxes = np.append(bboxes, FPs, axis=0)\n assert bboxes.dtype == int\n\n # we need to set sufficiently small confidence value as for not to affect TP\n conf_FPs = [fp_conf] * fp_count\n confs = np.append(confs, conf_FPs)\n assert confs.dtype == float\n return bboxes, confs \n\n\ndef predict(model, img, size=768, augment=False, p_keep=1.0):\n height, width = img.shape[:2]\n results = model(img, size=size, augment=augment) # custom inference size\n preds = results.pandas().xyxy[0]\n bboxes = preds[['xmin','ymin','xmax','ymax']].values\n if len(bboxes):\n bboxes = voc2coco(bboxes, height, width).astype(int)\n confs = preds.confidence.values\n\n bboxes, confs = drop_pred(bboxes, confs, p_keep=p_keep)\n return bboxes, confs\n else:\n bboxes, confs = np.zeros((0, 4), dtype=int), np.zeros((0), dtype=int)\n return bboxes, confs\n\ndef format_prediction(bboxes, confs):\n annot = ''\n if len(bboxes) > 0:\n for idx in range(len(bboxes)):\n xmin, ymin, w, h = bboxes[idx]\n conf = confs[idx]\n annot += f'{conf:.8f} {xmin} {ymin} {w} {h}'\n annot +=' '\n annot = annot.strip(' ')\n return annot\nfor idx, (img, pred_df) in enumerate(tqdm(iter_test)):\n bboxes, confs = predict(model, img, size=IMG_SIZE, augment=AUGMENT, p_keep=P_KEEP)\n\n # idx=5200 is possible to be a public test frame\n # - ref: https://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/302057\n fp_add_frame = 5200\n if idx == fp_add_frame:\n bboxes, confs = add_fp(bboxes, confs, fp_count=FP_COUNT)\n print(f\"{len(bboxes)}, {len(confs)}\")\n print(f\"{bboxes[0]}, {confs[0]}\")\n\n annot = format_prediction(bboxes, confs) \n pred_df['annotations'] = annot\n env.predict(pred_df)\n if idx < 3:\n display(show_img(img, bboxes, bbox_format='coco'))\n[1] LB probing result: GT Labels per Frame in the Public LB
\nI added 10,000 FP bbox to one frame, and submit this model. After calculated LB, the score is dropped 0.006 (0.601 -> 0.595). Given the total frame number is around 3,500 (0.25 * 14,000), and number of GT label is ~2450[1], the score drop is apparently too little.
\nOn the contrary, if I add one FP bbox on every frame, the score is dropped as expected (0.601 -> 0.426).
\n\nWe have now highly rational explanation on this issue.
\nIt seems that the box threshold per image is set to ~129 to avoid system load.
Details are explained in this thread:
\nhttps://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/302632#1661980
These are clarified by Kaggle staff:
\n\n\nNo, the F2 score is calculated globally.
\n
\n There is a cutoff of 100 predictions per image. Is this low enough that it's relevant for your actual model outputs?
https://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/302632#1663027
\nThe test code I used is as below:
\nP_KEEP = 1.0\nCONF = 0.50\nFP_COUNT = 10_000\ndef drop_pred(bboxes, confs, p_keep):\n '''\n randomly drop prediction for the probability (1 - p_keep).\n '''\n if p_keep == 1:\n return bboxes, confs\n bboxes = bboxes.copy()\n confs = confs.copy()\n assert len(bboxes) == len(confs)\n pp = np.random.uniform(size=len(bboxes))\n bboxes = bboxes[pp <= p_keep]\n confs = confs[pp <= p_keep]\n return bboxes, confs\n\ndef add_fp(bboxes, confs, fp_count=0, fp_conf=1e-4):\n '''\n add false negative to the frame\n '''\n if fp_count == 0:\n return bboxes, confs\n bboxes = bboxes.copy()\n confs = confs.copy()\n assert len(bboxes) == len(confs)\n FPs = [[i % 1280, i // 1280, 1, 1] for i in range(fp_count)]\n bboxes = np.append(bboxes, FPs, axis=0)\n assert bboxes.dtype == int\n\n # we need to set sufficiently small confidence value as for not to affect TP\n conf_FPs = [fp_conf] * fp_count\n confs = np.append(confs, conf_FPs)\n assert confs.dtype == float\n return bboxes, confs \n\n\ndef predict(model, img, size=768, augment=False, p_keep=1.0):\n height, width = img.shape[:2]\n results = model(img, size=size, augment=augment) # custom inference size\n preds = results.pandas().xyxy[0]\n bboxes = preds[['xmin','ymin','xmax','ymax']].values\n if len(bboxes):\n bboxes = voc2coco(bboxes, height, width).astype(int)\n confs = preds.confidence.values\n\n bboxes, confs = drop_pred(bboxes, confs, p_keep=p_keep)\n return bboxes, confs\n else:\n bboxes, confs = np.zeros((0, 4), dtype=int), np.zeros((0), dtype=int)\n return bboxes, confs\n\ndef format_prediction(bboxes, confs):\n annot = ''\n if len(bboxes) > 0:\n for idx in range(len(bboxes)):\n xmin, ymin, w, h = bboxes[idx]\n conf = confs[idx]\n annot += f'{conf:.8f} {xmin} {ymin} {w} {h}'\n annot +=' '\n annot = annot.strip(' ')\n return annot\nfor idx, (img, pred_df) in enumerate(tqdm(iter_test)):\n bboxes, confs = predict(model, img, size=IMG_SIZE, augment=AUGMENT, p_keep=P_KEEP)\n\n # idx=5200 is possible to be a public test frame\n # - ref: https://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/302057\n fp_add_frame = 5200\n if idx == fp_add_frame:\n bboxes, confs = add_fp(bboxes, confs, fp_count=FP_COUNT)\n print(f\"{len(bboxes)}, {len(confs)}\")\n print(f\"{bboxes[0]}, {confs[0]}\")\n\n annot = format_prediction(bboxes, confs) \n pred_df['annotations'] = annot\n env.predict(pred_df)\n if idx < 3:\n display(show_img(img, bboxes, bbox_format='coco'))\n[1] LB probing result: GT Labels per Frame in the Public LB
\nI think they just cut-off too many predicted boxes per image to not overload the system. The drop in your score would be smaller if it would be local metric. The only explanation I can see is that it is cut-off. Or maybe you have duplicate boxes they drop, I did not check your code. All this is just a guess.
", "rawMarkdown": "I think they just cut-off too many predicted boxes per image to not overload the system. The drop in your score would be smaller if it would be local metric. The only explanation I can see is that it is cut-off. Or maybe you have duplicate boxes they drop, I did not check your code. All this is just a guess.", "votes": 6, "replies": [ { "id": 1661999, "postDate": "2022-01-23T23:19:51.210Z", "content": "Thank you for pointing this out.
\nI had thought of this once, but never delved into it deeply.
\nWe can estimate the cutoff threshold by the following equation [1].
\nΞ± means the number of FPs added to the prediction.
$$
\n\\frac{TP}{\\alpha} = \\frac{1}{5} \\left( \\frac{1}{F_2^\\alpha} - \\frac{1}{F_2} \\right)^{-1} \\tag{5}
\n$$
Using the same formula as adding one FP to every image, I estimated the TP score to be ~1538.
\nSubstituting this into equation (5), the estimated value of Ξ± for the cutoff for each image is ~129. This is a very likely value.
[1] https://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/302156
", "rawMarkdown": "Thank you for pointing this out.\n\nI had thought of this once, but never delved into it deeply.\n\nWe can estimate the cutoff threshold by the following equation [1].\nΞ± means the number of FPs added to the prediction.\n\n$$\n\\frac{TP}{\\alpha} = \\frac{1}{5} \\left( \\frac{1}{F_2^\\alpha} - \\frac{1}{F_2} \\right)^{-1} \\tag{5}\n$$\n\nUsing the same formula as adding one FP to every image, I estimated the TP score to be ~1538.\nSubstituting this into equation (5), the estimated value of Ξ± for the cutoff for each image is ~129. This is a very likely value.\n\n[1] https://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/302156", "votes": 3 }, { "id": 1662011, "postDate": "2022-01-23T23:39:06.630Z", "content": "\n\nOr maybe you have duplicate boxes they drop
\n
I took duplicated boxes into account. The above code generates unique boxes.
", "rawMarkdown": "> Or maybe you have duplicate boxes they drop\n\nI took duplicated boxes into account. The above code generates unique boxes.", "votes": 1 }, { "id": 1662347, "postDate": "2022-01-24T07:46:54.110Z", "content": "so probably maxdets=100 :)
", "rawMarkdown": "so probably maxdets=100 :)", "votes": 1 }, { "id": 1663022, "postDate": "2022-01-24T18:02:59.850Z", "content": "@philippsinger is correct. Capping the number of predictions per image let us use simple pairwise comparisons in the metric without worrying about excessive scoring runtimes.
", "rawMarkdown": "@philippsinger is correct. Capping the number of predictions per image let us use simple pairwise comparisons in the metric without worrying about excessive scoring runtimes. ", "votes": 3 }, { "id": 1663299, "postDate": "2022-01-25T00:47:34.503Z", "content": "What is the criteria for cutting off? If it is by score, what happens if box are of the same score?
\nSay of i submit 1000 prediction box of same score1.0, the evalution selects The first 100 or the best 100?
\nIt is better to set a rule that at most 100 box predicted per image in this case
", "rawMarkdown": "What is the criteria for cutting off? If it is by score, what happens if box are of the same score? \n\nSay of i submit 1000 prediction box of same score1.0, the evalution selects The first 100 or the best 100? \n\n\nIt is better to set a rule that at most 100 box predicted per image in this case", "votes": 1 }, { "id": 1664600, "postDate": "2022-01-26T05:01:12.133Z", "content": "@sohier Hi, how about the question @hengck23 asked above? I want to know the answer too.
", "rawMarkdown": "@sohier Hi, how about the question @hengck23 asked above? I want to know the answer too." } ] }, { "id": 1661440, "postDate": "2022-01-23T13:21:05.573Z", "content": "@addisonhoward @sohier
\nDear Kaggle staffs, please read through the post and consider to disclose the evaluation code, unless we have to spend a lot of time to solve this issue. Considering the competition end is closing, it would be tough to all of us.
Some quick questions:
\n@sohier Thank you for clarification.
", "rawMarkdown": "@sohier Thank you for clarification." } ] }, { "id": 1662294, "postDate": "2022-01-24T07:01:54.533Z", "content": "I agree we found it, some weeks ago but couldn't prove it. Lots of better models got worse score. We made FP reduction manuever but still score didn't increased much or sometimes dropped. Second observation everytime we jump inference resolution val F2 always drops but on LB almost always explodes to higher value.
", "rawMarkdown": "I agree we found it, some weeks ago but couldn't prove it. Lots of better models got worse score. We made FP reduction manuever but still score didn't increased much or sometimes dropped. Second observation everytime we jump inference resolution val F2 always drops but on LB almost always explodes to higher value. ", "votes": 1, "replies": [ { "id": 1662338, "postDate": "2022-01-24T07:40:34.773Z", "content": "\n\nLots of better models got worse score
\n
I think this problem is caused by something else. The issue I encountered is now have rational explanation.
\nPlease read this thread.
\nhttps://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/302632#1661980
kagglers are inference the evaluation way, too. ππ
", "rawMarkdown": "kagglers are inference the evaluation way, too. ππ" }, { "id": 1662566, "postDate": "2022-01-24T11:54:17.990Z", "content": "just a quick note:
\nto submit really FP, try (x,y,1,1)
Yes. I already use this as below:
\nFPs = [[i % 1280, i // 1280, 1, 1] for i in range(fp_count)]\nBut still 129 Detections should significantly drop F2, for throwing FP boxes
", "rawMarkdown": "But still 129 Detections should significantly drop F2, for throwing FP boxes", "replies": [ { "id": 1662388, "postDate": "2022-01-24T08:34:37.817Z", "content": "No. Itβs as expected. Please calculate by yourself using eq (5) shared above.
", "rawMarkdown": "No. Itβs as expected. Please calculate by yourself using eq (5) shared above.", "votes": 1 } ] }, { "id": 1662303, "postDate": "2022-01-24T07:08:22.360Z", "content": "@tatamikenn Maybe I am dumb but how you add 10000 FN to one frame ?:) False Negative is not detected starfish, eventually remove TP which we don't know. I missed something obvious ?
", "rawMarkdown": "@tatamikenn Maybe I am dumb but how you add 10000 FN to one frame ?:) False Negative is not detected starfish, eventually remove TP which we don't know. I missed something obvious ?", "replies": [ { "id": 1662334, "postDate": "2022-01-24T07:35:38.387Z", "content": "Sorry, FN is typo. I've corrected the notation now(FN->FP)
", "rawMarkdown": "Sorry, FN is typo. I've corrected the notation now(FN->FP)", "votes": 1 } ] }, { "id": 1662217, "postDate": "2022-01-24T05:59:31.063Z", "content": "\n\nOne hypothesis I came up with is the F2 score is calculated per image, and then average each of them.
\n
I would say this hypothesis is not true.
\nI conducted another experiment which put zero prediction on frame idx=5,200 (we denote this experiment as (A)).
\nIf the hypothesis is true, then the calculated F2 value of the frame is zero.
\nOn the contrary, if we add 10,000 FP to the frame, the calculated F2 value of the frame is equal to or greater than zero.
\nThus, the observed LB score in the latter case is equal to or greater than the former case (since F2 of a frame is equal to or greater than the other, whereas the other frame's F2 are the same).
\nTherefore, the LB score of experiment (A) should be less than or equal to the case of +10K * FP case.
However, in the experiment (A), I observed zero drop in LB score (0.601 -> 0.601) which is greater than +10K * FP case(0.595).
\nThis contradicts what we discussed above.
the code is here:
\nfor idx, (img, pred_df) in enumerate(tqdm(iter_test)):\n bboxes, confs = predict(model, img, size=IMG_SIZE, augment=AUGMENT, p_keep=P_KEEP)\n\n # idx=3000 is a public test frame\n # - ref: https://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/302057\n fp_add_frame = 5_200\n if idx == fp_add_frame:\n # blank prediction\n bboxes, confs = np.zeros((0, 4), dtype=int), np.zeros((0), dtype=int)\n\n annot = format_prediction(bboxes, confs) \n pred_df['annotations'] = annot\n env.predict(pred_df)\nI think, that's right.
\n\n\nOne hypothesis I came up with is the F2 score is calculated per image, and then average each of them.
\n
We can actually test it. Add 10000 FN boxes to every image, or to every second image. If F2 drops dramatically, then your theory is right. If not, then something strange is going on here.
\nBut personally, I think, averaging F2 over each image, is a good thing to do π.
", "rawMarkdown": "I think, that's right.\n> One hypothesis I came up with is the F2 score is calculated per image, and then average each of them. \n\nWe can actually test it. Add 10000 FN boxes to every image, or to every second image. If F2 drops dramatically, then your theory is right. If not, then something strange is going on here.\n\nBut personally, I think, averaging F2 over each image, is a good thing to do π.", "replies": [ { "id": 1662007, "postDate": "2022-01-23T23:33:48.460Z", "content": "\n\n\nBut personally, I think, averaging F2 over each image, is a good thing to do
\n
How can you define F2 on the background (non-annotated) image?
", "rawMarkdown": "@vadbeg \n\n>But personally, I think, averaging F2 over each image, is a good thing to do\n\nHow can you define F2 on the background (non-annotated) image?" }, { "id": 1662020, "postDate": "2022-01-23T23:58:19.507Z", "content": "\nMaybe it skips if there are no detections on the background and sets F2 to zero for average if there were detections on a background image. But I have zero clue if I am correct and I am not sure how to test for that. Some clarification is needed.
", "rawMarkdown": "@tatamikenn \n\nMaybe it skips if there are no detections on the background and sets F2 to zero for average if there were detections on a background image. But I have zero clue if I am correct and I am not sure how to test for that. Some clarification is needed." }, { "id": 1662023, "postDate": "2022-01-24T00:01:34.427Z", "content": "\n\n\nMaybe it skips if there are no detections on the background and sets F2 to zero for average if there were detections on a background image.
\n
This is possible, but I think this evaluation method is nonsense. It can't give low score for the model which generates plenty of FPs to the background image.
", "rawMarkdown": "@outwrest \n\n> Maybe it skips if there are no detections on the background and sets F2 to zero for average if there were detections on a background image.\n\nThis is possible, but I think this evaluation method is nonsense. It can't give low score for the model which generates plenty of FPs to the background image." }, { "id": 1662024, "postDate": "2022-01-24T00:02:19.060Z", "content": "See also this thread. I think this option is more probable.
\nhttps://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/302632#1661980
", "rawMarkdown": "See also this thread. I think this option is more probable.\n\nhttps://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/302632#1661980" }, { "id": 1662027, "postDate": "2022-01-24T00:06:39.513Z", "content": "@tatamikenn I see the thread and that is a better explanation.
\n\n\nevaluation method is nonsense
\n
I don't see why F2 was chosen for object detection, mAP would've sufficed and proven to be easier to implement and test. I wasted a lot of time implementing and worrying about it.
", "rawMarkdown": "@tatamikenn I see the thread and that is a better explanation. \n\n> evaluation method is nonsense\n\nI don't see why F2 was chosen for object detection, mAP would've sufficed and proven to be easier to implement and test. I wasted a lot of time implementing and worrying about it. " }, { "id": 1662029, "postDate": "2022-01-24T00:12:04.497Z", "content": "\n\nI don't see why F2 was chosen for object detection, mAP would've sufficed and proven to be easier to implement and test.
\n
I partly agree with that but, as you know, mAP is not appropriate for real application since it doesn't considers confidence threshold.
\nYou set confidence threshold as small as possible for local evaluation, right?
I think they want production model, not academic ones.
\nThey considers setting proper confidence threshold is our task to submit better prediction model as a product.
I think they just cut-off too many predicted boxes per image to not overload the system. The drop in your score would be smaller if it would be local metric. The only explanation I can see is that it is cut-off. Or maybe you have duplicate boxes they drop, I did not check your code. All this is just a guess.
", "votes": 6, "replies": [ { "id": 1661999, "author_name": "Bilzard", "author_url": "", "post_date": "2022-01-23T23:19:51.210000", "content": "Thank you for pointing this out.
\nI had thought of this once, but never delved into it deeply.
\nWe can estimate the cutoff threshold by the following equation [1].
\nΞ± means the number of FPs added to the prediction.
$$
\n\\frac{TP}{\\alpha} = \\frac{1}{5} \\left( \\frac{1}{F_2^\\alpha} - \\frac{1}{F_2} \\right)^{-1} \\tag{5}
\n$$
Using the same formula as adding one FP to every image, I estimated the TP score to be ~1538.
\nSubstituting this into equation (5), the estimated value of Ξ± for the cutoff for each image is ~129. This is a very likely value.
[1] https://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/302156
", "votes": 3, "replies": [] }, { "id": 1662011, "author_name": "Bilzard", "author_url": "", "post_date": "2022-01-23T23:39:06.630000", "content": "\n\nOr maybe you have duplicate boxes they drop
\n
I took duplicated boxes into account. The above code generates unique boxes.
", "votes": 1, "replies": [] }, { "id": 1662347, "author_name": "Lukasz Borecki", "author_url": "", "post_date": "2022-01-24T07:46:54.110000", "content": "so probably maxdets=100 :)
", "votes": 1, "replies": [] }, { "id": 1663022, "author_name": "Sohier Dane", "author_url": "", "post_date": "2022-01-24T18:02:59.850000", "content": "@philippsinger is correct. Capping the number of predictions per image let us use simple pairwise comparisons in the metric without worrying about excessive scoring runtimes.
", "votes": 3, "replies": [] }, { "id": 1663299, "author_name": "hengck23", "author_url": "", "post_date": "2022-01-25T00:47:34.503000", "content": "What is the criteria for cutting off? If it is by score, what happens if box are of the same score?
\nSay of i submit 1000 prediction box of same score1.0, the evalution selects The first 100 or the best 100?
\nIt is better to set a rule that at most 100 box predicted per image in this case
", "votes": 1, "replies": [] }, { "id": 1664600, "author_name": "Bilzard", "author_url": "", "post_date": "2022-01-26T05:01:12.133000", "content": "@sohier Hi, how about the question @hengck23 asked above? I want to know the answer too.
", "votes": 0, "replies": [] } ] }, { "id": 1661440, "author_name": "Bilzard", "author_url": "", "post_date": "2022-01-23T13:21:05.573000", "content": "@addisonhoward @sohier
\nDear Kaggle staffs, please read through the post and consider to disclose the evaluation code, unless we have to spend a lot of time to solve this issue. Considering the competition end is closing, it would be tough to all of us.
Some quick questions:
\n@sohier Thank you for clarification.
", "votes": 0, "replies": [] } ] }, { "id": 1662294, "author_name": "Lukasz Borecki", "author_url": "", "post_date": "2022-01-24T07:01:54.533000", "content": "I agree we found it, some weeks ago but couldn't prove it. Lots of better models got worse score. We made FP reduction manuever but still score didn't increased much or sometimes dropped. Second observation everytime we jump inference resolution val F2 always drops but on LB almost always explodes to higher value.
", "votes": 1, "replies": [ { "id": 1662338, "author_name": "Bilzard", "author_url": "", "post_date": "2022-01-24T07:40:34.773000", "content": "\n\nLots of better models got worse score
\n
I think this problem is caused by something else. The issue I encountered is now have rational explanation.
\nPlease read this thread.
\nhttps://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/302632#1661980
kagglers are inference the evaluation way, too. ππ
", "votes": 0, "replies": [] }, { "id": 1662566, "author_name": "hengck23", "author_url": "", "post_date": "2022-01-24T11:54:17.990000", "content": "just a quick note:
\nto submit really FP, try (x,y,1,1)
Yes. I already use this as below:
\nFPs = [[i % 1280, i // 1280, 1, 1] for i in range(fp_count)]\nBut still 129 Detections should significantly drop F2, for throwing FP boxes
", "votes": 0, "replies": [ { "id": 1662388, "author_name": "Bilzard", "author_url": "", "post_date": "2022-01-24T08:34:37.817000", "content": "No. Itβs as expected. Please calculate by yourself using eq (5) shared above.
", "votes": 1, "replies": [] } ] }, { "id": 1662303, "author_name": "Lukasz Borecki", "author_url": "", "post_date": "2022-01-24T07:08:22.360000", "content": "@tatamikenn Maybe I am dumb but how you add 10000 FN to one frame ?:) False Negative is not detected starfish, eventually remove TP which we don't know. I missed something obvious ?
", "votes": 0, "replies": [ { "id": 1662334, "author_name": "Bilzard", "author_url": "", "post_date": "2022-01-24T07:35:38.387000", "content": "Sorry, FN is typo. I've corrected the notation now(FN->FP)
", "votes": 1, "replies": [] } ] }, { "id": 1662217, "author_name": "Bilzard", "author_url": "", "post_date": "2022-01-24T05:59:31.063000", "content": "\n\nOne hypothesis I came up with is the F2 score is calculated per image, and then average each of them.
\n
I would say this hypothesis is not true.
\nI conducted another experiment which put zero prediction on frame idx=5,200 (we denote this experiment as (A)).
\nIf the hypothesis is true, then the calculated F2 value of the frame is zero.
\nOn the contrary, if we add 10,000 FP to the frame, the calculated F2 value of the frame is equal to or greater than zero.
\nThus, the observed LB score in the latter case is equal to or greater than the former case (since F2 of a frame is equal to or greater than the other, whereas the other frame's F2 are the same).
\nTherefore, the LB score of experiment (A) should be less than or equal to the case of +10K * FP case.
However, in the experiment (A), I observed zero drop in LB score (0.601 -> 0.601) which is greater than +10K * FP case(0.595).
\nThis contradicts what we discussed above.
the code is here:
\nfor idx, (img, pred_df) in enumerate(tqdm(iter_test)):\n bboxes, confs = predict(model, img, size=IMG_SIZE, augment=AUGMENT, p_keep=P_KEEP)\n\n # idx=3000 is a public test frame\n # - ref: https://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/302057\n fp_add_frame = 5_200\n if idx == fp_add_frame:\n # blank prediction\n bboxes, confs = np.zeros((0, 4), dtype=int), np.zeros((0), dtype=int)\n\n annot = format_prediction(bboxes, confs) \n pred_df['annotations'] = annot\n env.predict(pred_df)\nI think, that's right.
\n\n\nOne hypothesis I came up with is the F2 score is calculated per image, and then average each of them.
\n
We can actually test it. Add 10000 FN boxes to every image, or to every second image. If F2 drops dramatically, then your theory is right. If not, then something strange is going on here.
\nBut personally, I think, averaging F2 over each image, is a good thing to do π.
", "votes": 0, "replies": [ { "id": 1662007, "author_name": "Bilzard", "author_url": "", "post_date": "2022-01-23T23:33:48.460000", "content": "\n\n\nBut personally, I think, averaging F2 over each image, is a good thing to do
\n
How can you define F2 on the background (non-annotated) image?
", "votes": 0, "replies": [] }, { "id": 1662020, "author_name": "outwrest", "author_url": "", "post_date": "2022-01-23T23:58:19.507000", "content": "\nMaybe it skips if there are no detections on the background and sets F2 to zero for average if there were detections on a background image. But I have zero clue if I am correct and I am not sure how to test for that. Some clarification is needed.
", "votes": 0, "replies": [] }, { "id": 1662023, "author_name": "Bilzard", "author_url": "", "post_date": "2022-01-24T00:01:34.427000", "content": "\n\n\nMaybe it skips if there are no detections on the background and sets F2 to zero for average if there were detections on a background image.
\n
This is possible, but I think this evaluation method is nonsense. It can't give low score for the model which generates plenty of FPs to the background image.
", "votes": 0, "replies": [] }, { "id": 1662024, "author_name": "Bilzard", "author_url": "", "post_date": "2022-01-24T00:02:19.060000", "content": "See also this thread. I think this option is more probable.
\nhttps://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/302632#1661980
", "votes": 0, "replies": [] }, { "id": 1662027, "author_name": "outwrest", "author_url": "", "post_date": "2022-01-24T00:06:39.513000", "content": "@tatamikenn I see the thread and that is a better explanation.
\n\n\nevaluation method is nonsense
\n
I don't see why F2 was chosen for object detection, mAP would've sufficed and proven to be easier to implement and test. I wasted a lot of time implementing and worrying about it.
", "votes": 0, "replies": [] }, { "id": 1662029, "author_name": "Bilzard", "author_url": "", "post_date": "2022-01-24T00:12:04.497000", "content": "\n\nI don't see why F2 was chosen for object detection, mAP would've sufficed and proven to be easier to implement and test.
\n
I partly agree with that but, as you know, mAP is not appropriate for real application since it doesn't considers confidence threshold.
\nYou set confidence threshold as small as possible for local evaluation, right?
I think they want production model, not academic ones.
\nThey considers setting proper confidence threshold is our task to submit better prediction model as a product.