{ "id": 302577, "title": "Magic Three: Rough Estimate of The Number of Tiny COTS in the Public LB", "url": "/competitions/tensorflow-great-barrier-reef/discussion/302577", "author_name": "Bilzard", "post_date": "2022-01-23T07:50:06.623000", "votes": 10, "comment_count": 4, "views": 0, "content": "

Here I come up with the idea to get a rough estimate of number of tiny cots in the public LB.
\nThis is an extension of magic one[1] and two[2].

\n

In Short

\n

What you need is:

\n
    \n
  1. estimate TN, FN, FP of your model by magic two[2]
    2. \n
  2. drop prediction of your model with probability 1 - rho if the predicted box size is less than threshold (beta) and observe LB score (F2_rho)
    3. \n
  3. repeat 2 for different dropout ratio
    4. \n
  4. estimate #TP<beta / #TP by equation (6)
    5. \n
  5. Use this estimate as a rough estimate of #GT<beta / #GT
    6. \n
\n

Note: If you want to make accurate estimate, you should make your prediction model of recall close to 1.0.

\n

Detail

\n

If you randomly drop predicted bbox of size < beta with probability 1 - rho, the new TP', FN' and FP' are become like this:

\n

$$
\nTP^\\prime = TP - (1 - \\rho) TP_{<\\beta} \\tag{1}
\n$$

\n

$$
\nFN^\\prime = FN + (1 - \\rho) TP_{<\\beta} \\tag{2}
\n$$

\n

$$
\nFP^\\prime = FP - (1 - \\rho) FP_{<\\beta} \\tag{3}
\n$$

\n

Then the new F2 score (F2_rho) is

\n

$$
\nF_2^\\rho = \\frac{5TP^\\prime}{5TP^\\prime + 4FN^\\prime + FP^\\prime} \\tag{4}
\n$$

\n

substituting eq (1)-(3) into eq(4), we obtain

\n

$$
\n\\left( \\frac{5}{F_2^\\rho} - 1 \\right) \\frac{TP_{<\\beta}}{TP} - \\frac{FP_{<\\beta}}{TP} = \\frac{5}{1 - \\rho} \\left( \\frac{1}{F_2^\\rho} - \\frac{1}{F_2} \\right) \\tag{5}
\n$$

\n

If we observe F2_rho for different dropout ratio 1 - rho, we obtain equation form like (5) for N count.
\nThe unknown variable number is two (TP_beta/TP and FP_beta/TP) and the equation number is N > 2, which is the over-determined system.

\n

We can solve approximate solution with least-square fitting.

\n

$$
\nmin||Ax - b||
\n$$

\n

$$
\nx = (A^\\top A)^{-1} A^\\top b \\tag{6}
\n$$

\n

where

\n

$$
\nA = \\begin{bmatrix}
\n5/F_2^{\\rho_1} - 1 & -1 \\\\
\n5/F_2^{\\rho_2} - 1 & -1 \\\\
\n… \\\\
\n5/F_2^{\\rho_N} - 1 & -1
\n\\end{bmatrix}
\n$$

\n

$$
\nb = \\begin{bmatrix}
\n\\frac{5}{1 - \\rho_1} (1/F_2^{\\rho_1} - 1/F_2) \\\\
\n\\frac{5}{1 - \\rho_2} (1/F_2^{\\rho_2} - 1/F_2) \\\\
\n… \\\\
\n\\frac{5}{1 - \\rho_N} (1/F_2^{\\rho_N} - 1/F_2)
\n\\end{bmatrix}
\n$$

\n

$$
\nx = \\begin{bmatrix}
\n\\frac{TP_{<\\beta}}{TP} \\\\
\n\\frac{FP_{<\\beta}}{TP}
\n\\end{bmatrix}
\n$$

\n

Since we have no clue about the non-detected label, we cannot estimate FN<beta directly.
\nSo we need to assume that TP<beta : TP = GT<beta : GT. This is not generally true, since detector performance is generally correlated with target size (e.g., good at/not good at detecting small COTS).

\n

So we use TP<beta / TP as an approximate estimate of GT<beta / GT.

\n

\"Screen-Shot-2022-01-23-at-16-16-47\"

\n

Reference

\n

[1] https://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/302130
\n[2] https://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/302156

\n

Update Note

\n", "messages": [ { "id": 1661069, "postDate": "2022-01-23T07:50:06.623Z", "content": "

Here I come up with the idea to get a rough estimate of number of tiny cots in the public LB.
\nThis is an extension of magic one[1] and two[2].

\n

In Short

\n

What you need is:

\n
    \n
  1. estimate TN, FN, FP of your model by magic two[2]
    2. \n
  2. drop prediction of your model with probability 1 - rho if the predicted box size is less than threshold (beta) and observe LB score (F2_rho)
    3. \n
  3. repeat 2 for different dropout ratio
    4. \n
  4. estimate #TP<beta / #TP by equation (6)
    5. \n
  5. Use this estimate as a rough estimate of #GT<beta / #GT
    6. \n
\n

Note: If you want to make accurate estimate, you should make your prediction model of recall close to 1.0.

\n

Detail

\n

If you randomly drop predicted bbox of size < beta with probability 1 - rho, the new TP', FN' and FP' are become like this:

\n

$$
\nTP^\\prime = TP - (1 - \\rho) TP_{<\\beta} \\tag{1}
\n$$

\n

$$
\nFN^\\prime = FN + (1 - \\rho) TP_{<\\beta} \\tag{2}
\n$$

\n

$$
\nFP^\\prime = FP - (1 - \\rho) FP_{<\\beta} \\tag{3}
\n$$

\n

Then the new F2 score (F2_rho) is

\n

$$
\nF_2^\\rho = \\frac{5TP^\\prime}{5TP^\\prime + 4FN^\\prime + FP^\\prime} \\tag{4}
\n$$

\n

substituting eq (1)-(3) into eq(4), we obtain

\n

$$
\n\\left( \\frac{5}{F_2^\\rho} - 1 \\right) \\frac{TP_{<\\beta}}{TP} - \\frac{FP_{<\\beta}}{TP} = \\frac{5}{1 - \\rho} \\left( \\frac{1}{F_2^\\rho} - \\frac{1}{F_2} \\right) \\tag{5}
\n$$

\n

If we observe F2_rho for different dropout ratio 1 - rho, we obtain equation form like (5) for N count.
\nThe unknown variable number is two (TP_beta/TP and FP_beta/TP) and the equation number is N > 2, which is the over-determined system.

\n

We can solve approximate solution with least-square fitting.

\n

$$
\nmin||Ax - b||
\n$$

\n

$$
\nx = (A^\\top A)^{-1} A^\\top b \\tag{6}
\n$$

\n

where

\n

$$
\nA = \\begin{bmatrix}
\n5/F_2^{\\rho_1} - 1 & -1 \\\\
\n5/F_2^{\\rho_2} - 1 & -1 \\\\
\n… \\\\
\n5/F_2^{\\rho_N} - 1 & -1
\n\\end{bmatrix}
\n$$

\n

$$
\nb = \\begin{bmatrix}
\n\\frac{5}{1 - \\rho_1} (1/F_2^{\\rho_1} - 1/F_2) \\\\
\n\\frac{5}{1 - \\rho_2} (1/F_2^{\\rho_2} - 1/F_2) \\\\
\n… \\\\
\n\\frac{5}{1 - \\rho_N} (1/F_2^{\\rho_N} - 1/F_2)
\n\\end{bmatrix}
\n$$

\n

$$
\nx = \\begin{bmatrix}
\n\\frac{TP_{<\\beta}}{TP} \\\\
\n\\frac{FP_{<\\beta}}{TP}
\n\\end{bmatrix}
\n$$

\n

Since we have no clue about the non-detected label, we cannot estimate FN<beta directly.
\nSo we need to assume that TP<beta : TP = GT<beta : GT. This is not generally true, since detector performance is generally correlated with target size (e.g., good at/not good at detecting small COTS).

\n

So we use TP<beta / TP as an approximate estimate of GT<beta / GT.

\n

\"Screen-Shot-2022-01-23-at-16-16-47\"

\n

Reference

\n

[1] https://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/302130
\n[2] https://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/302156

\n

Update Note

\n", "rawMarkdown": "Here I come up with the idea to get a rough estimate of number of tiny cots in the public LB.\nThis is an extension of magic one[1] and two[2].\n\n# In Short\n\nWhat you need is:\n\n1. estimate TN, FN, FP of your model by magic two[2]\n2. drop prediction of your model with probability 1 - rho if the predicted box size is less than threshold (beta) and observe LB score (F2_rho)\n3. repeat 2 for different dropout ratio\n4. estimate #TP 2, which is the over-determined system.\n\nWe can solve approximate solution with least-square fitting.\n\n$$\nmin||Ax - b||\n$$\n\n$$\nx = (A^\\top A)^{-1} A^\\top b \\tag{6}\n$$\n\nwhere\n\n$$\nA = \\begin{bmatrix}\n5/F_2^{\\rho_1} - 1 & -1 \\\\\\\\\n5/F_2^{\\rho_2} - 1 & -1 \\\\\\\\\n... \\\\\\\\\n5/F_2^{\\rho_N} - 1 & -1\n\\end{bmatrix}\n$$\n\n$$\nb = \\begin{bmatrix}\n\\frac{5}{1 - \\rho_1} (1/F_2^{\\rho_1} - 1/F_2) \\\\\\\\\n\\frac{5}{1 - \\rho_2} (1/F_2^{\\rho_2} - 1/F_2) \\\\\\\\\n... \\\\\\\\\n\\frac{5}{1 - \\rho_N} (1/F_2^{\\rho_N} - 1/F_2)\n\\end{bmatrix}\n$$\n\n$$\nx = \\begin{bmatrix}\n\\frac{TP_{<\\beta}}{TP} \\\\\\\\\n\\frac{FP_{<\\beta}}{TP}\n\\end{bmatrix}\n$$\n\nSince we have no clue about the non-detected label, we cannot estimate FN\"Screen-Shot-2022-01-23-at-16-16-47\"\n\n# Reference\n\n[1] https://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/302130\n[2] https://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/302156\n\n# Update Note\n\n* 2022/1/23 use TPNumerical Simulation\n

Assumption:

\n
    \n
  • F_2 score is rounded at decimal point 3
    • \n
  • TP, FN, FP are perfectly estimated (error is zero)
    • \n
\n

Result

\n

Since observed F_2 is rounded at decimal point 3, the estimated beta (GT>beta / GT) has some amount of variance.
\nIf you have much observation (i.e. submission), you can get more accurate estimation.

\n
\n

2022/1/23 17:50 JST: update

\n

We can't estimate FN<beta directly, so we have to set assumption that TP<beta : TP = GT<beta : GT. It is not generally true because generally the detectors performance is correlated to the target size (e.g. good at / poor at detecting tiny COTS).

\n

Therefore, we estimate TP<beta / TP and we use this estimate for approximate estimate of GT<beta / GT.

\n
========================================\nrhos = array([0.1, 0.5, 0.9])\ntp_beta_tp = 0.371\ntp_beta_tp_est = 0.381\nnp.abs(tp_beta_tp_est - tp_beta_tp) = 0.010\n========================================\nrhos = array([0.1, 0.3, 0.5, 0.7, 0.9])\ntp_beta_tp = 0.371\ntp_beta_tp_est = 0.376\nnp.abs(tp_beta_tp_est - tp_beta_tp) = 0.005\n========================================\nrhos = array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9])\ntp_beta_tp = 0.371\ntp_beta_tp_est = 0.370\nnp.abs(tp_beta_tp_est - tp_beta_tp) = 0.001\n
\n

Code

\n
import numpy as np\nfrom numpy.linalg import inv\n\n\ndef f2(tp, fn, fp):\n    return 5 * tp / (5 * tp + 4 * fn + fp)\n\n\ndef estimate(tp, fn, fp, tp_beta, fp_beta, rhos, dp=3):\n    \"\"\"\n    return:\n        - TP(< beta) / TP\n        - FP(< beta) / FP\n    \"\"\"\n    F2 = np.round(f2(tp, fn, fp), dp)\n\n    A = np.zeros((len(rhos), 2))\n    b = np.zeros((len(rhos), 1))\n    for i, rho in enumerate(rhos):\n        tp_prime = tp - (1 - rho) * tp_beta\n        fn_prime = fn + (1 - rho) * tp_beta\n        fp_prime = fp - (1 - rho) * fp_beta\n        F2_prime = round(f2(tp_prime, fn_prime, fp_prime), dp)\n        assert (tp + fn) - (tp_prime + fn_prime) <= 1\n        A[i, :] = [5 / F2_prime - 1, -1]\n        b[i, :] = 5 / (1 - rho) * (1 / F2_prime - 1 / F2)\n\n    x = inv(A.T @ A) @ A.T @ b\n    tp_beta_tp_est, fp_beta_tp_est = x[:, 0]\n\n    return tp_beta_tp_est, fp_beta_tp_est\n\n\ndef main():\n    \"\"\"\n    simulate estimation of beta (GT(size<beta) / GT)\n    \"\"\"\n    tp, fn, fp, tp_beta, fp_beta = 421, 324, 313, 156, 172\n    for num_obs in (3, 5, 9):\n        rhos = np.linspace(0.1, 0.9, num_obs)\n        tp_beta_tp = tp_beta / tp\n        tp_beta_tp_est, _ = estimate(tp, fn, fp, tp_beta, fp_beta, rhos)\n        print(\"=\" * 40)\n        print(f\"{rhos = }\")\n        print(f\"{tp_beta_tp = :.3f}\")\n        print(f\"{tp_beta_tp_est = :.3f}\")\n        print(f\"{np.abs(tp_beta_tp_est - tp_beta_tp) = :.3f}\")\n\n\nif __name__ == \"__main__\":\n    main()\n
", "rawMarkdown": "## Numerical Simulation\n\nAssumption:\n* F_2 score is rounded at decimal point 3\n* TP, FN, FP are perfectly estimated (error is zero)\n\n## Result\n\nSince observed F_2 is rounded at decimal point 3, the estimated beta (GT>beta / GT) has some amount of variance.\nIf you have much observation (i.e. submission), you can get more accurate estimation.\n\n----\n2022/1/23 17:50 JST: update\n\nWe can't estimate FN@que62932 Thank you for reading the post.
\nNo, I don't. In fact, I haven't even experimented with this yet.
\nSince I found we can't directly estimate GT_beta/GT (see the last paragraph of detail section), I put this task's priority low for now.

", "rawMarkdown": "@que62932 Thank you for reading the post.\nNo, I don't. In fact, I haven't even experimented with this yet.\nSince I found we can't directly estimate GT_beta/GT (see the last paragraph of detail section), I put this task's priority low for now." }, { "id": 1661491, "postDate": "2022-01-23T13:49:03.420Z", "content": "

If you want to do it by yourself, the code snippet shared on this discussion might be useful:
\nhttps://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/302130

", "rawMarkdown": "If you want to do it by yourself, the code snippet shared on this discussion might be useful:\nhttps://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/302130" } ] } ], "comments": [ { "id": 1661072, "author_name": "Bilzard", "author_url": "", "post_date": "2022-01-23T07:54:39.987000", "content": "

Numerical Simulation

\n

Assumption:

\n
    \n
  • F_2 score is rounded at decimal point 3
    • \n
  • TP, FN, FP are perfectly estimated (error is zero)
    • \n
\n

Result

\n

Since observed F_2 is rounded at decimal point 3, the estimated beta (GT>beta / GT) has some amount of variance.
\nIf you have much observation (i.e. submission), you can get more accurate estimation.

\n
\n

2022/1/23 17:50 JST: update

\n

We can't estimate FN<beta directly, so we have to set assumption that TP<beta : TP = GT<beta : GT. It is not generally true because generally the detectors performance is correlated to the target size (e.g. good at / poor at detecting tiny COTS).

\n

Therefore, we estimate TP<beta / TP and we use this estimate for approximate estimate of GT<beta / GT.

\n
========================================\nrhos = array([0.1, 0.5, 0.9])\ntp_beta_tp = 0.371\ntp_beta_tp_est = 0.381\nnp.abs(tp_beta_tp_est - tp_beta_tp) = 0.010\n========================================\nrhos = array([0.1, 0.3, 0.5, 0.7, 0.9])\ntp_beta_tp = 0.371\ntp_beta_tp_est = 0.376\nnp.abs(tp_beta_tp_est - tp_beta_tp) = 0.005\n========================================\nrhos = array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9])\ntp_beta_tp = 0.371\ntp_beta_tp_est = 0.370\nnp.abs(tp_beta_tp_est - tp_beta_tp) = 0.001\n
\n

Code

\n
import numpy as np\nfrom numpy.linalg import inv\n\n\ndef f2(tp, fn, fp):\n    return 5 * tp / (5 * tp + 4 * fn + fp)\n\n\ndef estimate(tp, fn, fp, tp_beta, fp_beta, rhos, dp=3):\n    \"\"\"\n    return:\n        - TP(< beta) / TP\n        - FP(< beta) / FP\n    \"\"\"\n    F2 = np.round(f2(tp, fn, fp), dp)\n\n    A = np.zeros((len(rhos), 2))\n    b = np.zeros((len(rhos), 1))\n    for i, rho in enumerate(rhos):\n        tp_prime = tp - (1 - rho) * tp_beta\n        fn_prime = fn + (1 - rho) * tp_beta\n        fp_prime = fp - (1 - rho) * fp_beta\n        F2_prime = round(f2(tp_prime, fn_prime, fp_prime), dp)\n        assert (tp + fn) - (tp_prime + fn_prime) <= 1\n        A[i, :] = [5 / F2_prime - 1, -1]\n        b[i, :] = 5 / (1 - rho) * (1 / F2_prime - 1 / F2)\n\n    x = inv(A.T @ A) @ A.T @ b\n    tp_beta_tp_est, fp_beta_tp_est = x[:, 0]\n\n    return tp_beta_tp_est, fp_beta_tp_est\n\n\ndef main():\n    \"\"\"\n    simulate estimation of beta (GT(size<beta) / GT)\n    \"\"\"\n    tp, fn, fp, tp_beta, fp_beta = 421, 324, 313, 156, 172\n    for num_obs in (3, 5, 9):\n        rhos = np.linspace(0.1, 0.9, num_obs)\n        tp_beta_tp = tp_beta / tp\n        tp_beta_tp_est, _ = estimate(tp, fn, fp, tp_beta, fp_beta, rhos)\n        print(\"=\" * 40)\n        print(f\"{rhos = }\")\n        print(f\"{tp_beta_tp = :.3f}\")\n        print(f\"{tp_beta_tp_est = :.3f}\")\n        print(f\"{np.abs(tp_beta_tp_est - tp_beta_tp) = :.3f}\")\n\n\nif __name__ == \"__main__\":\n    main()\n
", "votes": 0, "replies": [] }, { "id": 1661469, "author_name": "", "author_url": "", "post_date": "2022-01-23T13:37:15.723000", "content": "", "votes": 0, "replies": [ { "id": 1661483, "author_name": "Bilzard", "author_url": "", "post_date": "2022-01-23T13:43:01.540000", "content": "

@que62932 Thank you for reading the post.
\nNo, I don't. In fact, I haven't even experimented with this yet.
\nSince I found we can't directly estimate GT_beta/GT (see the last paragraph of detail section), I put this task's priority low for now.

", "votes": 0, "replies": [] }, { "id": 1661491, "author_name": "Bilzard", "author_url": "", "post_date": "2022-01-23T13:49:03.420000", "content": "

If you want to do it by yourself, the code snippet shared on this discussion might be useful:
\nhttps://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/302130

", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1661069": "Here I come up with the idea to get a rough estimate of number of tiny cots in the public LB.\nThis is an extension of magic one[1] and two[2].\n\n# In Short\n\nWhat you need is:\n\n1. estimate TN, FN, FP of your model by magic two[2]\n2. drop prediction of your model with probability 1 - rho if the predicted box size is less than threshold (beta) and observe LB score (F2_rho)\n3. repeat 2 for different dropout ratio\n4. estimate #TP 2, which is the over-determined system.\n\nWe can solve approximate solution with least-square fitting.\n\n$$\nmin||Ax - b||\n$$\n\n$$\nx = (A^\\top A)^{-1} A^\\top b \\tag{6}\n$$\n\nwhere\n\n$$\nA = \\begin{bmatrix}\n5/F_2^{\\rho_1} - 1 & -1 \\\\\\\\\n5/F_2^{\\rho_2} - 1 & -1 \\\\\\\\\n... \\\\\\\\\n5/F_2^{\\rho_N} - 1 & -1\n\\end{bmatrix}\n$$\n\n$$\nb = \\begin{bmatrix}\n\\frac{5}{1 - \\rho_1} (1/F_2^{\\rho_1} - 1/F_2) \\\\\\\\\n\\frac{5}{1 - \\rho_2} (1/F_2^{\\rho_2} - 1/F_2) \\\\\\\\\n... \\\\\\\\\n\\frac{5}{1 - \\rho_N} (1/F_2^{\\rho_N} - 1/F_2)\n\\end{bmatrix}\n$$\n\n$$\nx = \\begin{bmatrix}\n\\frac{TP_{<\\beta}}{TP} \\\\\\\\\n\\frac{FP_{<\\beta}}{TP}\n\\end{bmatrix}\n$$\n\nSince we have no clue about the non-detected label, we cannot estimate FN\"Screen-Shot-2022-01-23-at-16-16-47\"\n\n# Reference\n\n[1] https://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/302130\n[2] https://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/302156\n\n# Update Note\n\n* 2022/1/23 use TPbeta / GT) has some amount of variance.\nIf you have much observation (i.e. submission), you can get more accurate estimation.\n\n----\n2022/1/23 17:50 JST: update\n\nWe can't estimate FN