\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2909490%2F7a882bdb04c1090b38b9ac055ba23d23%2FSharedScreenshot.jpg?generation=1608949250550596&alt=media\")
My dice scores on 3 of the public test are over 0.92 but much worse on c68fe75ea and afa5e8098 (~0.8). So I found a way to probe the number of pixels of c68fe75ea's mask.
\nThe result shows that I missed ~30% of ground truth mask:
\n$$
\nTP:FP:FN \\simeq 0.9:0.1:0.4
\n$$
More specifically, my prediction has 1.3x10^7 pixels, and 0.13x10^7 (10%) of them are wrong, which I think is acceptable. However, there are 1.7x10^7 pixels (130% of my prediction) in ground truth mask.
\nMy prediction mask is:
\n
Did I miss 30% mask (0.4x10^7 pixels) in the light color region or there are large number of mistakes in ground truth?
\nUPDATE
\nMy probing method:
\nSay my prediction is X and ground truth is Y,
\nI split X into two half (I split them into left half and right half):
\n$$
\nX = X_1 \\cup X_2,\\quad X_1\\cap X_2 = \\varnothing
\n$$
Then I submit 3 times, using X, X1, X2, then the dice scores are:
\n$$
\n\\begin{cases}
\ndice_1 = \\frac{X\\cap Y}{X+Y}\\\\
\ndice_2=\\frac{X_1\\cap Y}{X_1+Y}\\\\
\ndice_3=\\frac{X\\cap Y - X_1\\cap Y}{X_2 + Y}
\n\\end{cases}
\n$$
Since
\n$$
\nX,X_1,X_2,dice_1,dice_2,dice_3$$
\nare known, and
\n$$
\nY, X\\cap Y, X_1\\cap Y$$
\nare unknown, we can solve the equations.
My dice scores on 3 of the public test are over 0.92 but much worse on c68fe75ea and afa5e8098 (~0.8). So I found a way to probe the number of pixels of c68fe75ea's mask.
\nThe result shows that I missed ~30% of ground truth mask:
\n$$
\nTP:FP:FN \\simeq 0.9:0.1:0.4
\n$$
More specifically, my prediction has 1.3x10^7 pixels, and 0.13x10^7 (10%) of them are wrong, which I think is acceptable. However, there are 1.7x10^7 pixels (130% of my prediction) in ground truth mask.
\nMy prediction mask is:
\n
Did I miss 30% mask (0.4x10^7 pixels) in the light color region or there are large number of mistakes in ground truth?
\nUPDATE
\nMy probing method:
\nSay my prediction is X and ground truth is Y,
\nI split X into two half (I split them into left half and right half):
\n$$
\nX = X_1 \\cup X_2,\\quad X_1\\cap X_2 = \\varnothing
\n$$
Then I submit 3 times, using X, X1, X2, then the dice scores are:
\n$$
\n\\begin{cases}
\ndice_1 = \\frac{X\\cap Y}{X+Y}\\\\
\ndice_2=\\frac{X_1\\cap Y}{X_1+Y}\\\\
\ndice_3=\\frac{X\\cap Y - X_1\\cap Y}{X_2 + Y}
\n\\end{cases}
\n$$
Since
\n$$
\nX,X_1,X_2,dice_1,dice_2,dice_3$$
\nare known, and
\n$$
\nY, X\\cap Y, X_1\\cap Y$$
\nare unknown, we can solve the equations.
I think the test data and training data may contain wrong marks, but before the organizer does not change these marks, we have to adapt to these wrong marks. This is called the mark has been marked wrong, so we need to adapt to this error. To adapt to test data and training data.
", "rawMarkdown": "I think the test data and training data may contain wrong marks, but before the organizer does not change these marks, we have to adapt to these wrong marks. This is called the mark has been marked wrong, so we need to adapt to this error. To adapt to test data and training data.", "votes": null }, { "id": "1127081", "postDate": "12/26/2020 08:19:44", "content": "a smarter way to probe:
\nstep.1 instead of instance prediction, reduce this to a smaller circle or box that is fully inscribed by the mask. then the intersection in iou= 100%. you can back compute the number of the correct detected instances (now you have zero FP pixel and your error is now only missing pixel). It is important to make sure that all submitted pixel are only true positive in this step.
step.2 random select non-predicted region in step.1 and add to the submission. you can back-calculate the number of true positive pixel in this new region
you may think that your low score is due to missing predicted instances … but it could be the other way round … you are predicting too many false positives? (e.g. certain part of the image are not annotated?)
", "rawMarkdown": "a smarter way to probe:\n- step.1 instead of instance prediction, reduce this to a smaller circle or box that is fully inscribed by the mask. then the intersection in iou= 100%. you can back compute the number of the correct detected instances (now you have zero FP pixel and your error is now only missing pixel). It is important to make sure that all submitted pixel are only true positive in this step.\n\n- step.2 random select non-predicted region in step.1 and add to the submission. you can back-calculate the number of true positive pixel in this new region\n\n\nyou may think that your low score is due to missing predicted instances ... but it could be the other way round ... you are predicting too many false positives? (e.g. certain part of the image are not annotated?)", "votes": null }, { "id": "1127100", "postDate": "12/26/2020 08:31:56", "content": "Thanks for your method!
\nBut according to my method, I can accurately calculate TP, FP and FN at the same time.
\nAnd the result is, my FP/TP ratio is only 0.11 but my FN/TP ratio is 0.44. So I think large FN is the major reason for my low score.
", "rawMarkdown": "Thanks for your method! \n\nBut according to my method, I can accurately calculate TP, FP and FN at the same time.\n\n And the result is, my FP/TP ratio is only 0.11 but my FN/TP ratio is 0.44. So I think large FN is the major reason for my low score.", "votes": null }, { "id": "1127102", "postDate": "12/26/2020 08:38:40", "content": "there is also a possibility that kaggle has false ground truth to prevent people from probing.
\nother possibility is that the ground truth mask is shifted. hence you are missing and having FP at the same time.
small circle or box that is fully inscribed by the mask will not be affected by shifted ground truth. it also let you estimate your IOU sore per instance (instance of per image)
", "rawMarkdown": "there is also a possibility that kaggle has false ground truth to prevent people from probing.\nother possibility is that the ground truth mask is shifted. hence you are missing and having FP at the same time.\n\nsmall circle or box that is fully inscribed by the mask will not be affected by shifted ground truth. it also let you estimate your IOU sore per instance (instance of per image)", "votes": null }, { "id": "1127121", "postDate": "12/26/2020 09:02:39", "content": "Got it! I think I can try it later
\nbtw, I updated my probing method
", "rawMarkdown": "Got it! I think I can try it later\n\nbtw, I updated my probing method", "votes": null }, { "id": "1127899", "postDate": "12/27/2020 02:20:58", "content": "I don't quite understand why dice1, dice2 and dice3 are known? Does dice1, dice2 and dice3 refer to the verification score or the final submission score?
", "rawMarkdown": "I don't quite understand why dice1, dice2 and dice3 are known? Does dice1, dice2 and dice3 refer to the verification score or the final submission score?", "votes": null }, { "id": "1127923", "postDate": "12/27/2020 03:35:14", "content": "submission.csvMay I ask a small question, your afa5e8098 score(~0.8) is really high than mind(~0.74). Does it just come from model predictions same as other three pictures(~0.92)?
", "rawMarkdown": "May I ask a small question, your afa5e8098 score(~0.8) is really high than mind(~0.74). Does it just come from model predictions same as other three pictures(~0.92)?", "votes": null }, { "id": "1128025", "postDate": "12/27/2020 05:40:25", "content": "Nope, I shift my prediction of afa5e8098 manually.
", "rawMarkdown": "Nope, I shift my prediction of afa5e8098 manually.", "votes": null }, { "id": "1128031", "postDate": "12/27/2020 05:55:41", "content": "Thanks for your sharing!
", "rawMarkdown": "Thanks for your sharing!", "votes": null }, { "id": "1137540", "postDate": "01/04/2021 03:03:05", "content": "Thanks for sharing this..
", "rawMarkdown": "Thanks for sharing this..", "votes": null } ], "comments": [ { "id": 1126944, "author_name": "zzzhangzz", "author_url": "", "post_date": "12/26/2020 05:37:34", "content": "I think the test data and training data may contain wrong marks, but before the organizer does not change these marks, we have to adapt to these wrong marks. This is called the mark has been marked wrong, so we need to adapt to this error. To adapt to test data and training data.
", "votes": null, "replies": [] }, { "id": 1127081, "author_name": "hengck23", "author_url": "", "post_date": "12/26/2020 08:19:44", "content": "a smarter way to probe:
\nstep.1 instead of instance prediction, reduce this to a smaller circle or box that is fully inscribed by the mask. then the intersection in iou= 100%. you can back compute the number of the correct detected instances (now you have zero FP pixel and your error is now only missing pixel). It is important to make sure that all submitted pixel are only true positive in this step.
step.2 random select non-predicted region in step.1 and add to the submission. you can back-calculate the number of true positive pixel in this new region
you may think that your low score is due to missing predicted instances … but it could be the other way round … you are predicting too many false positives? (e.g. certain part of the image are not annotated?)
", "votes": null, "replies": [ { "id": 1127100, "author_name": "carnozhao", "author_url": "", "post_date": "12/26/2020 08:31:56", "content": "Thanks for your method!
\nBut according to my method, I can accurately calculate TP, FP and FN at the same time.
\nAnd the result is, my FP/TP ratio is only 0.11 but my FN/TP ratio is 0.44. So I think large FN is the major reason for my low score.
", "votes": null, "replies": [] }, { "id": 1127102, "author_name": "hengck23", "author_url": "", "post_date": "12/26/2020 08:38:40", "content": "there is also a possibility that kaggle has false ground truth to prevent people from probing.
\nother possibility is that the ground truth mask is shifted. hence you are missing and having FP at the same time.
small circle or box that is fully inscribed by the mask will not be affected by shifted ground truth. it also let you estimate your IOU sore per instance (instance of per image)
", "votes": null, "replies": [] }, { "id": 1127121, "author_name": "carnozhao", "author_url": "", "post_date": "12/26/2020 09:02:39", "content": "Got it! I think I can try it later
\nbtw, I updated my probing method
", "votes": null, "replies": [] }, { "id": 1127899, "author_name": "zhangeng", "author_url": "", "post_date": "12/27/2020 02:20:58", "content": "I don't quite understand why dice1, dice2 and dice3 are known? Does dice1, dice2 and dice3 refer to the verification score or the final submission score?
", "votes": null, "replies": [] }, { "id": 1127923, "author_name": "carnozhao", "author_url": "", "post_date": "12/27/2020 03:35:14", "content": "submission.csvMay I ask a small question, your afa5e8098 score(~0.8) is really high than mind(~0.74). Does it just come from model predictions same as other three pictures(~0.92)?
", "votes": null, "replies": [ { "id": 1128025, "author_name": "carnozhao", "author_url": "", "post_date": "12/27/2020 05:40:25", "content": "Nope, I shift my prediction of afa5e8098 manually.
", "votes": null, "replies": [] }, { "id": 1128031, "author_name": "hesene", "author_url": "", "post_date": "12/27/2020 05:55:41", "content": "Thanks for your sharing!
", "votes": null, "replies": [] } ] }, { "id": 1137540, "author_name": "viswanathravindran", "author_url": "", "post_date": "01/04/2021 03:03:05", "content": "Thanks for sharing this..
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1126827": "My dice scores on 3 of the public test are over 0.92 but much worse on **c68fe75ea and afa5e8098** (~0.8). So I found a way to probe the number of pixels of c68fe75ea's mask. \n\nThe result shows that I missed ~30% of ground truth mask:\n\n$$\nTP:FP:FN \\simeq 0.9:0.1:0.4\n$$\n\nMore specifically, my prediction has 1.3x10^7 pixels, and 0.13x10^7 (10%) of them are wrong, which I think is acceptable. However, there are 1.7x10^7 pixels (130% of my prediction) in ground truth mask.\n\nMy prediction mask is:\n\n\nDid I miss 30% mask (0.4x10^7 pixels) in the light color region or there are large number of mistakes in ground truth?\n\n\n**UPDATE**\n\nMy probing method:\n\nSay my prediction is X and ground truth is Y,\n\nI split X into two half (I split them into left half and right half):\n\n$$\nX = X_1 \\cup X_2,\\quad X_1\\cap X_2 = \\varnothing\n$$\n\nThen I submit 3 times, using X, X1, X2, then the dice scores are:\n\n$$\n\\begin{cases}\ndice_1 = \\frac{X\\cap Y}{X+Y}\\\\\\\ndice_2=\\frac{X_1\\cap Y}{X_1+Y}\\\\\\\ndice_3=\\frac{X\\cap Y - X_1\\cap Y}{X_2 + Y}\n\\end{cases}\n$$\n\nSince \n$$\nX,X_1,X_2,dice_1,dice_2,dice_3$$\nare known, and \n$$\nY, X\\cap Y, X_1\\cap Y$$\nare unknown, we can solve the equations.", "1126944": "I think the test data and training data may contain wrong marks, but before the organizer does not change these marks, we have to adapt to these wrong marks. This is called the mark has been marked wrong, so we need to adapt to this error. To adapt to test data and training data.", "1127081": "a smarter way to probe:\n- step.1 instead of instance prediction, reduce this to a smaller circle or box that is fully inscribed by the mask. then the intersection in iou= 100%. you can back compute the number of the correct detected instances (now you have zero FP pixel and your error is now only missing pixel). It is important to make sure that all submitted pixel are only true positive in this step.\n\n- step.2 random select non-predicted region in step.1 and add to the submission. you can back-calculate the number of true positive pixel in this new region\n\n\nyou may think that your low score is due to missing predicted instances ... but it could be the other way round ... you are predicting too many false positives? (e.g. certain part of the image are not annotated?)", "1127100": "Thanks for your method! \n\nBut according to my method, I can accurately calculate TP, FP and FN at the same time.\n\n And the result is, my FP/TP ratio is only 0.11 but my FN/TP ratio is 0.44. So I think large FN is the major reason for my low score.", "1127102": "there is also a possibility that kaggle has false ground truth to prevent people from probing.\nother possibility is that the ground truth mask is shifted. hence you are missing and having FP at the same time.\n\nsmall circle or box that is fully inscribed by the mask will not be affected by shifted ground truth. it also let you estimate your IOU sore per instance (instance of per image)", "1127121": "Got it! I think I can try it later\n\nbtw, I updated my probing method", "1127899": "I don't quite understand why dice1, dice2 and dice3 are known? Does dice1, dice2 and dice3 refer to the verification score or the final submission score?", "1127923": "1. Set others mask to \"0 1\" except c68fe75ea in `submission.csv`\n2. Submit your prediction mask of c68fe75ea to public LB, you will get a c68fe75ea-only dice score, e.g. the submission returns 0.170, then the dice score of c68fe75ea is 0.170 * 5 = 0.850 (dice1)\n3. submit your left half of c68fe75ea's mask to LB (dice2)\n4. Submit right half of c68fe75ea's mask to LB (dice3)", "1128006": "May I ask a small question, your afa5e8098 score(~0.8) is really high than mind(~0.74). Does it just come from model predictions same as other three pictures(~0.92)?", "1128025": "Nope, I shift my prediction of afa5e8098 manually.", "1128031": "Thanks for your sharing!", "1137540": "Thanks for sharing this.." }, "source": "meta" }