{ "id": 302156, "title": "Magic Two: We Can Estimate Number of GT Labels on the Public Test Data", "url": "/competitions/tensorflow-great-barrier-reef/discussion/302156", "author_name": "Bilzard", "post_date": "2022-01-21T05:50:16.010000", "votes": 7, "comment_count": 8, "views": 0, "content": "

In Short

\n

Extending the idea of the previous post[1], we can also estimate the number of GT labels on the public test data as well as TP, FP, FN.

\n

What we need is:

\n
    \n
  1. Estimate of FN/TP, PN/TP of your model by [1]
    2. \n
  2. Add certain number of FN to your prediction (watch out you should add it to the public test frame, not the private test frame) and observe F_2
    3. \n
  3. Estimate TP, FP, FN, GT using equation (5)
    4. \n
\n

Detail

\n

Given added FN of amount alpha, our new F_2 metrics becomes:

\n

$$
\nF_2^\\alpha = \\frac{5TP}{5TP+4FN+FP + \\alpha}
\n$$

\n

Clearing out denominator, and solve for TP, we get

\n

$$
\nTP = \\alpha\\left[ 5\\left( \\frac{1}{F_2^\\alpha} - 1 \\right) - 4\\frac{FN}{TP} - \\frac{FP}{TP} \\right]^{-1} \\tag{3}
\n$$

\n

Since we already have estimate of FN/TP and FP/TP, we can estimate TP by above equation.

\n

And since GT = TP + FN, we can also have an estimate of GT on the public test data.

\n
\n

Note: From Eq (1) and (2) in [1],
\n$$
\n4 \\frac{FN}{TP} + \\frac{FP}{TP} = 5 \\left( \\frac{1}{F_2} - 1 \\right) \\tag{4}
\n$$

\n

substituting (4) to (3), we have simpler form of TP estimation:

\n

$$
\n\\frac{TP}{\\alpha} = \\frac{1}{5} \\left( \\frac{1}{F_2^\\alpha} - \\frac{1}{F_2} \\right)^{-1} \\tag{5}
\n$$

\n

Since eq (5) has no FN/TP, FP/TP term, we have noise-free form of eq(3).

\n

\"Screen-Shot-2022-01-21-at-14-34-32\"

\n

Reference

\n

[1] Here is the magic: Estimating recall and precision from your public LB score

\n
\n

update note

\n", "messages": [ { "id": 1658547, "postDate": "2022-01-21T05:50:16.010Z", "content": "

In Short

\n

Extending the idea of the previous post[1], we can also estimate the number of GT labels on the public test data as well as TP, FP, FN.

\n

What we need is:

\n
    \n
  1. Estimate of FN/TP, PN/TP of your model by [1]
    2. \n
  2. Add certain number of FN to your prediction (watch out you should add it to the public test frame, not the private test frame) and observe F_2
    3. \n
  3. Estimate TP, FP, FN, GT using equation (5)
    4. \n
\n

Detail

\n

Given added FN of amount alpha, our new F_2 metrics becomes:

\n

$$
\nF_2^\\alpha = \\frac{5TP}{5TP+4FN+FP + \\alpha}
\n$$

\n

Clearing out denominator, and solve for TP, we get

\n

$$
\nTP = \\alpha\\left[ 5\\left( \\frac{1}{F_2^\\alpha} - 1 \\right) - 4\\frac{FN}{TP} - \\frac{FP}{TP} \\right]^{-1} \\tag{3}
\n$$

\n

Since we already have estimate of FN/TP and FP/TP, we can estimate TP by above equation.

\n

And since GT = TP + FN, we can also have an estimate of GT on the public test data.

\n
\n

Note: From Eq (1) and (2) in [1],
\n$$
\n4 \\frac{FN}{TP} + \\frac{FP}{TP} = 5 \\left( \\frac{1}{F_2} - 1 \\right) \\tag{4}
\n$$

\n

substituting (4) to (3), we have simpler form of TP estimation:

\n

$$
\n\\frac{TP}{\\alpha} = \\frac{1}{5} \\left( \\frac{1}{F_2^\\alpha} - \\frac{1}{F_2} \\right)^{-1} \\tag{5}
\n$$

\n

Since eq (5) has no FN/TP, FP/TP term, we have noise-free form of eq(3).

\n

\"Screen-Shot-2022-01-21-at-14-34-32\"

\n

Reference

\n

[1] Here is the magic: Estimating recall and precision from your public LB score

\n
\n

update note

\n", "rawMarkdown": "# In Short\n\nExtending the idea of the previous post[1], we can also estimate the number of GT labels on the public test data as well as TP, FP, FN.\n\nWhat we need is:\n\n1. Estimate of FN/TP, PN/TP of your model by [1]\n2. Add certain number of FN to your prediction (watch out you should add it to the public test frame, not the private test frame) and observe F_2\n3. Estimate TP, FP, FN, GT using equation (5)\n\n# Detail\n\nGiven added FN of amount alpha, our new F_2 metrics becomes:\n\n$$\nF_2^\\alpha = \\frac{5TP}{5TP+4FN+FP + \\alpha}\n$$\n\nClearing out denominator, and solve for TP, we get\n\n$$\nTP = \\alpha\\left[ 5\\left( \\frac{1}{F_2^\\alpha} - 1 \\right) - 4\\frac{FN}{TP} - \\frac{FP}{TP} \\right]^{-1} \\tag{3}\n$$\n\nSince we already have estimate of FN/TP and FP/TP, we can estimate TP by above equation.\n\nAnd since GT = TP + FN, we can also have an estimate of GT on the public test data.\n\n---\nNote: From Eq (1) and (2) in [1], \n$$\n4 \\frac{FN}{TP} + \\frac{FP}{TP} = 5 \\left( \\frac{1}{F_2} - 1 \\right) \\tag{4}\n$$\n\nsubstituting (4) to (3), we have simpler form of TP estimation:\n\n$$\n\\frac{TP}{\\alpha} = \\frac{1}{5} \\left( \\frac{1}{F_2^\\alpha} - \\frac{1}{F_2} \\right)^{-1} \\tag{5}\n$$\n\nSince eq (5) has no FN/TP, FP/TP term, we have noise-free form of eq(3).\n\n\"Screen-Shot-2022-01-21-at-14-34-32\"\n\n# Reference\n\n[1] [Here is the magic: Estimating recall and precision from your public LB score](https://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/302130)\n\n---\n# update note\n\n* 2022/1/21 14:59 JST: fix formula mistake\n* 2022/1/22 22:35 JST: add note of simpler form", "votes": 7 }, { "id": 1660209, "postDate": "2022-01-22T13:45:44.060Z", "content": "

Tips: How to estimate the number of GT labels per frame?

\n

It requires a lot of LB probing to distinguish public test frame from private test frame.
\nIf we can simply add one FP prediction to every test frame, the equation (5) become like this:

\n

$$
\n\\frac{TP}{I_{public}} = \\frac{1}{5} \\left( \\frac{1}{F_2^\\alpha} - \\frac{1}{F_2} \\right)^{-1} \\tag{6}
\n$$

\n

where I_public is the number of frames in the public test set.

\n

multiplying FP/TP to the eq(6),

\n

$$
\n\\frac{FP}{I_{public}} = \\frac{FP}{TP} \\frac{TP}{I_{public}}
\n$$

\n

since GT = FP + TP, we obtain

\n

$$
\n\\frac{GT}{I_{public}} = \\left( 1 + \\frac{FP}{TP} \\right) \\frac{TP}{I_{public}} \\tag{7}
\n$$

\n

We can estimate number of GT labels per frame using eq(7).

", "rawMarkdown": "## Tips: How to estimate the number of GT labels per frame?\n\nIt requires a lot of LB probing to distinguish public test frame from private test frame.\nIf we can simply add one FP prediction to every test frame, the equation (5) become like this:\n\n$$\n\\frac{TP}{I_{public}} = \\frac{1}{5} \\left( \\frac{1}{F_2^\\alpha} - \\frac{1}{F_2} \\right)^{-1} \\tag{6}\n$$\n\nwhere I_public is the number of frames in the public test set.\n\nmultiplying FP/TP to the eq(6),\n\n$$\n\\frac{FP}{I_{public}} = \\frac{FP}{TP} \\frac{TP}{I_{public}}\n$$\n\nsince GT = FP + TP, we obtain\n\n$$\n\\frac{GT}{I_{public}} = \\left( 1 + \\frac{FP}{TP} \\right) \\frac{TP}{I_{public}} \\tag{7}\n$$\n\nWe can estimate number of GT labels per frame using eq(7)." }, { "id": 1658978, "postDate": "2022-01-21T13:00:36.957Z", "content": "

Here is the simulation result:

\n

Assumptions in this simulation are:

\n\n

In this assumption, the estimated error is highly low when alpha is sufficiently large number.
\nIn the real situation, since we have certain amount of variance of estimated FN/TP and FP/TP, the estimated error will become large.

\n

\"Screen-Shot-2022-01-21-at-22-20-13\"

\n
\n

update: The estimation becomes numerically unstable if alpha is too small. In the case of the sample used in the simulation, alpha < TP * 0.25.

", "rawMarkdown": "Here is the simulation result:\n\nAssumptions in this simulation are:\n* FN/TP and FP/TP are perfectly estimated (i.e. error is zero)\n* F_2 is rounded at decimal point 3\n\nIn this assumption, the estimated error is highly low when alpha is sufficiently large number.\nIn the real situation, since we have certain amount of variance of estimated FN/TP and FP/TP, the estimated error will become large.\n\n\"Screen-Shot-2022-01-21-at-22-20-13\"\n\n---\nupdate: The estimation becomes numerically unstable if alpha is too small. In the case of the sample used in the simulation, alpha < TP * 0.25.", "replies": [ { "id": 1659537, "postDate": "2022-01-21T22:27:55.373Z", "content": "

Here is another simulation result where noise considered.

\n

Assumptions:

\n\n

If the observed F_2 is sufficiently small, first term of the denominator become large, and it become dominant compared to FN/TN, FP/TN terms. So we can get accurate result if alpha is sufficiently large.
\nIn this simulation case, alpha > 2000 is the safe choice.

\n

$$
\nTP = \\alpha\\left[ 5\\left( \\frac{1}{F_2^\\alpha} - 1 \\right) - 4\\frac{FN}{TP} - \\frac{FP}{TP} \\right]^{-1}
\n$$

\n

\"Screen-Shot-2022-01-22-at-8-14-53\"

", "rawMarkdown": "Here is another simulation result where noise considered.\n\nAssumptions:\n* add Gaussian noise to FN/TP, FP/TP: std=0.022, 0.061 respectively\n\nIf the observed F_2 is sufficiently small, first term of the denominator become large, and it become dominant compared to FN/TN, FP/TN terms. So we can get accurate result if alpha is sufficiently large.\nIn this simulation case, alpha > 2000 is the safe choice.\n\n$$\nTP = \\alpha\\left[ 5\\left( \\frac{1}{F_2^\\alpha} - 1 \\right) - 4\\frac{FN}{TP} - \\frac{FP}{TP} \\right]^{-1}\n$$\n\n\"Screen-Shot-2022-01-22-at-8-14-53\"" }, { "id": 1660232, "postDate": "2022-01-22T14:09:45.067Z", "content": "

This simulation result is no more necessary since we have equation (5).
\nThis simulation is based on the assumption that the noise in FN/TP and FP/TP are independent, but in reality, the two variables are correlated by equation (4).
\nTherefore, we need not to care the noise of FN/TP and TP/TP estimations.

", "rawMarkdown": "This simulation result is no more necessary since we have equation (5).\nThis simulation is based on the assumption that the noise in FN/TP and FP/TP are independent, but in reality, the two variables are correlated by equation (4).\nTherefore, we need not to care the noise of FN/TP and TP/TP estimations." }, { "id": 1662067, "postDate": "2022-01-24T01:57:50.310Z", "content": "

New simulation result is here.
\nI estimated TP using eq (5). This time, I considered no noise.

\n

\"Screen-Shot-2022-01-24-at-11-01-01\"

", "rawMarkdown": "New simulation result is here.\nI estimated TP using eq (5). This time, I considered no noise.\n\n\"Screen-Shot-2022-01-24-at-11-01-01\"" } ] }, { "id": 1658548, "postDate": "2022-01-21T05:50:52.967Z", "content": "

I didn't verified the detail, so maybe some mistake contained.

", "rawMarkdown": "I didn't verified the detail, so maybe some mistake contained." }, { "id": 1660186, "postDate": "2022-01-22T13:25:19.930Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1658554, "postDate": "2022-01-21T06:00:06.713Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1660209, "author_name": "Bilzard", "author_url": "", "post_date": "2022-01-22T13:45:44.060000", "content": "

Tips: How to estimate the number of GT labels per frame?

\n

It requires a lot of LB probing to distinguish public test frame from private test frame.
\nIf we can simply add one FP prediction to every test frame, the equation (5) become like this:

\n

$$
\n\\frac{TP}{I_{public}} = \\frac{1}{5} \\left( \\frac{1}{F_2^\\alpha} - \\frac{1}{F_2} \\right)^{-1} \\tag{6}
\n$$

\n

where I_public is the number of frames in the public test set.

\n

multiplying FP/TP to the eq(6),

\n

$$
\n\\frac{FP}{I_{public}} = \\frac{FP}{TP} \\frac{TP}{I_{public}}
\n$$

\n

since GT = FP + TP, we obtain

\n

$$
\n\\frac{GT}{I_{public}} = \\left( 1 + \\frac{FP}{TP} \\right) \\frac{TP}{I_{public}} \\tag{7}
\n$$

\n

We can estimate number of GT labels per frame using eq(7).

", "votes": 0, "replies": [] }, { "id": 1658978, "author_name": "Bilzard", "author_url": "", "post_date": "2022-01-21T13:00:36.957000", "content": "

Here is the simulation result:

\n

Assumptions in this simulation are:

\n\n

In this assumption, the estimated error is highly low when alpha is sufficiently large number.
\nIn the real situation, since we have certain amount of variance of estimated FN/TP and FP/TP, the estimated error will become large.

\n

\"Screen-Shot-2022-01-21-at-22-20-13\"

\n
\n

update: The estimation becomes numerically unstable if alpha is too small. In the case of the sample used in the simulation, alpha < TP * 0.25.

", "votes": 0, "replies": [ { "id": 1659537, "author_name": "Bilzard", "author_url": "", "post_date": "2022-01-21T22:27:55.373000", "content": "

Here is another simulation result where noise considered.

\n

Assumptions:

\n\n

If the observed F_2 is sufficiently small, first term of the denominator become large, and it become dominant compared to FN/TN, FP/TN terms. So we can get accurate result if alpha is sufficiently large.
\nIn this simulation case, alpha > 2000 is the safe choice.

\n

$$
\nTP = \\alpha\\left[ 5\\left( \\frac{1}{F_2^\\alpha} - 1 \\right) - 4\\frac{FN}{TP} - \\frac{FP}{TP} \\right]^{-1}
\n$$

\n

\"Screen-Shot-2022-01-22-at-8-14-53\"

", "votes": 0, "replies": [] }, { "id": 1660232, "author_name": "Bilzard", "author_url": "", "post_date": "2022-01-22T14:09:45.067000", "content": "

This simulation result is no more necessary since we have equation (5).
\nThis simulation is based on the assumption that the noise in FN/TP and FP/TP are independent, but in reality, the two variables are correlated by equation (4).
\nTherefore, we need not to care the noise of FN/TP and TP/TP estimations.

", "votes": 0, "replies": [] }, { "id": 1662067, "author_name": "Bilzard", "author_url": "", "post_date": "2022-01-24T01:57:50.310000", "content": "

New simulation result is here.
\nI estimated TP using eq (5). This time, I considered no noise.

\n

\"Screen-Shot-2022-01-24-at-11-01-01\"

", "votes": 0, "replies": [] } ] }, { "id": 1658548, "author_name": "Bilzard", "author_url": "", "post_date": "2022-01-21T05:50:52.967000", "content": "

I didn't verified the detail, so maybe some mistake contained.

", "votes": 0, "replies": [] }, { "id": 1660186, "author_name": "", "author_url": "", "post_date": "2022-01-22T13:25:19.930000", "content": "", "votes": 0, "replies": [] }, { "id": 1658554, "author_name": "", "author_url": "", "post_date": "2022-01-21T06:00:06.713000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1658547": "# In Short\n\nExtending the idea of the previous post[1], we can also estimate the number of GT labels on the public test data as well as TP, FP, FN.\n\nWhat we need is:\n\n1. Estimate of FN/TP, PN/TP of your model by [1]\n2. Add certain number of FN to your prediction (watch out you should add it to the public test frame, not the private test frame) and observe F_2\n3. Estimate TP, FP, FN, GT using equation (5)\n\n# Detail\n\nGiven added FN of amount alpha, our new F_2 metrics becomes:\n\n$$\nF_2^\\alpha = \\frac{5TP}{5TP+4FN+FP + \\alpha}\n$$\n\nClearing out denominator, and solve for TP, we get\n\n$$\nTP = \\alpha\\left[ 5\\left( \\frac{1}{F_2^\\alpha} - 1 \\right) - 4\\frac{FN}{TP} - \\frac{FP}{TP} \\right]^{-1} \\tag{3}\n$$\n\nSince we already have estimate of FN/TP and FP/TP, we can estimate TP by above equation.\n\nAnd since GT = TP + FN, we can also have an estimate of GT on the public test data.\n\n---\nNote: From Eq (1) and (2) in [1], \n$$\n4 \\frac{FN}{TP} + \\frac{FP}{TP} = 5 \\left( \\frac{1}{F_2} - 1 \\right) \\tag{4}\n$$\n\nsubstituting (4) to (3), we have simpler form of TP estimation:\n\n$$\n\\frac{TP}{\\alpha} = \\frac{1}{5} \\left( \\frac{1}{F_2^\\alpha} - \\frac{1}{F_2} \\right)^{-1} \\tag{5}\n$$\n\nSince eq (5) has no FN/TP, FP/TP term, we have noise-free form of eq(3).\n\n\"Screen-Shot-2022-01-21-at-14-34-32\"\n\n# Reference\n\n[1] [Here is the magic: Estimating recall and precision from your public LB score](https://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/302130)\n\n---\n# update note\n\n* 2022/1/21 14:59 JST: fix formula mistake\n* 2022/1/22 22:35 JST: add note of simpler form", "1660209": "## Tips: How to estimate the number of GT labels per frame?\n\nIt requires a lot of LB probing to distinguish public test frame from private test frame.\nIf we can simply add one FP prediction to every test frame, the equation (5) become like this:\n\n$$\n\\frac{TP}{I_{public}} = \\frac{1}{5} \\left( \\frac{1}{F_2^\\alpha} - \\frac{1}{F_2} \\right)^{-1} \\tag{6}\n$$\n\nwhere I_public is the number of frames in the public test set.\n\nmultiplying FP/TP to the eq(6),\n\n$$\n\\frac{FP}{I_{public}} = \\frac{FP}{TP} \\frac{TP}{I_{public}}\n$$\n\nsince GT = FP + TP, we obtain\n\n$$\n\\frac{GT}{I_{public}} = \\left( 1 + \\frac{FP}{TP} \\right) \\frac{TP}{I_{public}} \\tag{7}\n$$\n\nWe can estimate number of GT labels per frame using eq(7).", "1658978": "Here is the simulation result:\n\nAssumptions in this simulation are:\n* FN/TP and FP/TP are perfectly estimated (i.e. error is zero)\n* F_2 is rounded at decimal point 3\n\nIn this assumption, the estimated error is highly low when alpha is sufficiently large number.\nIn the real situation, since we have certain amount of variance of estimated FN/TP and FP/TP, the estimated error will become large.\n\n\"Screen-Shot-2022-01-21-at-22-20-13\"\n\n---\nupdate: The estimation becomes numerically unstable if alpha is too small. In the case of the sample used in the simulation, alpha < TP * 0.25.", "1658548": "I didn't verified the detail, so maybe some mistake contained.", "1660186": "", "1658554": "" } }