![\"Screen-Shot-2022-05-02-at-15-20-00\"](\"https://i.ibb.co/1MGSXxk/Screen-Shot-2022-05-02-at-15-20-00.png\")
I will share on how to estimate the contribution of each bird species to the LB score.
\nHere I make the following two assumptions.
\nscore(y, 1-y_pred) = 1 - score(y, y_pred).(We can be sure about 2 by submitting all predictions inverted for any given model.)
\nUnder the above assumptions, the scores for individual bird species are s_1, . , s_21, the LB score is calculated as follows.
\n$$
\nm_0 = \\frac{1}{21} (s_0 + … + s_i + … + s_{20}) \\tag{1}
\n$$
Now, if we invert the predictions for only one bird species (let's say i), the score would be
\n$$
\nm_1 = \\frac{1}{21} \\left(s_0 + … + (1 - s_i) + … + s_{20} \\right) \\tag{2}
\n$$
Solving equations (1) and (2) for s_i yields
\n$$
\ns_i = \\frac{21}{2} (m_0 - m_1) + 0.5 \\tag{3}
\n$$
That is, we can estimate the score of bird i by substituting the difference between the score m_0 for a normal submission and the predicted inverted score m_1 for a certain bird species into equation (3).
\nI tried the above experiment on the most sampled bird species, \"skylar\", and found no difference in scores between m_0 and m_1. I assume this is due to either the score of \"skylar\" alone being less than 0.6, or the \"skylar\" not being used in the publicLB score calculation.
\nIn Experiment 1, we were unable to detect score differences due to prediction reversal due to the small number of bird species, but increasing the number of bird species makes it easier to detect score differences due to prediction reversal.
\nTherefore, I decided to invert predictions for the top 10 bird species with the largest sample size and estimate the mean of the scores for these bird species.
\nLetting s_top10 be the average of the scores of top 10 species and s_low11 be the average of the scores of the remaining 11 bird species, the LB score is calculated as follows.
\n$$
\n21 m_0 = 10 s_\\text{top10} + 11 s_\\text{low11} \\tag{4}
\n$$
The inverted predicted scores for the top 10 bird species can also be calculated as follows
\n$$
\n21 m_1 = 10 (1 - s_\\text{top10}) + 11 s_\\text{low11} \\tag{5}
\n$$
Solving equations (4) and (5) for s_top10 yields
\n$$
\ns_\\text{top10} = \\frac{21}{20} (m_0 - m_1) + 0.5 \\tag{6}
\n$$
I calculated m_0 and m_1 using the public Notebook model[1] and found them to be 0.71 and 0.49, respectively. Therefore, s_top10 can be calculated to be 0.731.
\nFor the bottom 11 species, we can follow in the similar process, we estimated s_low11 to be 0.700.
\nThese results indicate that the top 10 bird species with the largest sample size contribute more to the LB score.
\nAlthough the low precision of the LB score does not allow us to predict exact values, these methods give us a rough idea of the degree to which your model's predictions are biased toward a particular species for public test data.
\nFinally, I will share the source code I used for my experiments.
\nHappy Kaggle.
\ndef invert_target(input_df, birds_to_invert):\n assert type(birds_to_invert) == list, type(birds)\n tmp_df = input_df.copy()\n prefixes, dates, birds, end_secs = zip(*tmp_df.row_id.str.split(\"_\"))\n tmp_df[\"prefix\"] = prefixes\n tmp_df[\"date\"] = dates\n tmp_df[\"bird\"] = birds\n tmp_df[\"end_sec\"] = end_secs\n\n idxs = tmp_df.query(\"bird in @birds_to_invert\").index\n tmp_df.loc[idxs, \"target\"] = tmp_df.loc[idxs, \"target\"].apply(lambda x: not(x))\n\n return tmp_df[[\"row_id\", \"target\"]]\nsample_submission = ... # your model's prediction\n\ntrain = pd.read_csv(\"../input/birdclef-2022/train_metadata.csv\")\nscored = train.query(\"primary_label in @scored_birds\")\nscored_count = scored[\"primary_label\"].value_counts()\ntop10 = scored_count[:10].index.tolist()\nlow11 = scored_count[10:].index.tolist()\ntmp_df = invert_target(sample_submission, low11)\nsample_submission[\"target\"] = tmp_df[\"target\"]\nI will share on how to estimate the contribution of each bird species to the LB score.
\nHere I make the following two assumptions.
\nscore(y, 1-y_pred) = 1 - score(y, y_pred).(We can be sure about 2 by submitting all predictions inverted for any given model.)
\nUnder the above assumptions, the scores for individual bird species are s_1, . , s_21, the LB score is calculated as follows.
\n$$
\nm_0 = \\frac{1}{21} (s_0 + … + s_i + … + s_{20}) \\tag{1}
\n$$
Now, if we invert the predictions for only one bird species (let's say i), the score would be
\n$$
\nm_1 = \\frac{1}{21} \\left(s_0 + … + (1 - s_i) + … + s_{20} \\right) \\tag{2}
\n$$
Solving equations (1) and (2) for s_i yields
\n$$
\ns_i = \\frac{21}{2} (m_0 - m_1) + 0.5 \\tag{3}
\n$$
That is, we can estimate the score of bird i by substituting the difference between the score m_0 for a normal submission and the predicted inverted score m_1 for a certain bird species into equation (3).
\nI tried the above experiment on the most sampled bird species, \"skylar\", and found no difference in scores between m_0 and m_1. I assume this is due to either the score of \"skylar\" alone being less than 0.6, or the \"skylar\" not being used in the publicLB score calculation.
\nIn Experiment 1, we were unable to detect score differences due to prediction reversal due to the small number of bird species, but increasing the number of bird species makes it easier to detect score differences due to prediction reversal.
\nTherefore, I decided to invert predictions for the top 10 bird species with the largest sample size and estimate the mean of the scores for these bird species.
\nLetting s_top10 be the average of the scores of top 10 species and s_low11 be the average of the scores of the remaining 11 bird species, the LB score is calculated as follows.
\n$$
\n21 m_0 = 10 s_\\text{top10} + 11 s_\\text{low11} \\tag{4}
\n$$
The inverted predicted scores for the top 10 bird species can also be calculated as follows
\n$$
\n21 m_1 = 10 (1 - s_\\text{top10}) + 11 s_\\text{low11} \\tag{5}
\n$$
Solving equations (4) and (5) for s_top10 yields
\n$$
\ns_\\text{top10} = \\frac{21}{20} (m_0 - m_1) + 0.5 \\tag{6}
\n$$
I calculated m_0 and m_1 using the public Notebook model[1] and found them to be 0.71 and 0.49, respectively. Therefore, s_top10 can be calculated to be 0.731.
\nFor the bottom 11 species, we can follow in the similar process, we estimated s_low11 to be 0.700.
\nThese results indicate that the top 10 bird species with the largest sample size contribute more to the LB score.
\nAlthough the low precision of the LB score does not allow us to predict exact values, these methods give us a rough idea of the degree to which your model's predictions are biased toward a particular species for public test data.
\nFinally, I will share the source code I used for my experiments.
\nHappy Kaggle.
\ndef invert_target(input_df, birds_to_invert):\n assert type(birds_to_invert) == list, type(birds)\n tmp_df = input_df.copy()\n prefixes, dates, birds, end_secs = zip(*tmp_df.row_id.str.split(\"_\"))\n tmp_df[\"prefix\"] = prefixes\n tmp_df[\"date\"] = dates\n tmp_df[\"bird\"] = birds\n tmp_df[\"end_sec\"] = end_secs\n\n idxs = tmp_df.query(\"bird in @birds_to_invert\").index\n tmp_df.loc[idxs, \"target\"] = tmp_df.loc[idxs, \"target\"].apply(lambda x: not(x))\n\n return tmp_df[[\"row_id\", \"target\"]]\nsample_submission = ... # your model's prediction\n\ntrain = pd.read_csv(\"../input/birdclef-2022/train_metadata.csv\")\nscored = train.query(\"primary_label in @scored_birds\")\nscored_count = scored[\"primary_label\"].value_counts()\ntop10 = scored_count[:10].index.tolist()\nlow11 = scored_count[10:].index.tolist()\ntmp_df = invert_target(sample_submission, low11)\nsample_submission[\"target\"] = tmp_df[\"target\"]\nNote: I confirmed assumption 2 seems to be correct, by obtaining below result:
\nscore(y, y_pred) + score(y, 1 - y_pred) = 0.71 + 0.28 = 0.99\nStrictly speaking, the sum of the scores is 0.99 instead of 1. This is probably because the LB scores are truncated to the third decimal place. In that case, the expected value of the sum of scores would be 0.99.
", "rawMarkdown": "Note: I confirmed assumption 2 seems to be correct, by obtaining below result:\n\n```\nscore(y, y_pred) + score(y, 1 - y_pred) = 0.71 + 0.28 = 0.99\n```\n\nStrictly speaking, the sum of the scores is 0.99 instead of 1. This is probably because the LB scores are truncated to the third decimal place. In that case, the expected value of the sum of scores would be 0.99.", "votes": null }, { "id": "1774284", "postDate": "05/02/2022 03:49:35", "content": "In Experiment 2, assuming that the sum of the scores with and without inverting predictions is 0.99, the estimated scores are as follows:
\ns_top10 = 0.726\ns_low11 = 0.695\nI have discovered one interesting fact.
\nIn the public notebook model (show reference [1] in the original post), the top 5 most frequent bird species in the training data contribute an estimated average of only 0.558 to the score. The top 6 to 10 each contribute the most to the score, with an estimated mean score of 0.894.
\nThis result indicates that a large frequency of occurrence in the training data does not necessarily mean a large contribution to the public LB score. The reason is unknown at this time, but it is possible that the distribution of species in the public test data is quite variable from species to species or that some species are excluded from the public test data calculations.
top5, mid_top5, mid_low5, low6 = \n(['skylar', 'houfin', 'jabwar', 'warwhe1', 'yefcan'],\n ['apapan', 'iiwi', 'omao', 'hawama', 'hawcre'],\n ['barpet', 'akiapo', 'elepai', 'aniani', 'hawgoo'],\n ['ercfra', 'hawpet1', 'puaioh', 'hawhaw', 'crehon', 'maupar'])\n\noriginal score: 0.71\nscore with inverted prediction of top5: 0.68\nscore with inverted prediction of mid_top5: 0.52\nscore with inverted prediction of mid_low5: 0.56\nscore with inverted prediction of top5: 0.65\n\nestimated mean score of top5: 0.558\nestimated mean score of mid_top5: 0.894\nestimated mean score of mid_low5: 0.810\nestimated mean score of low6: 0.600\nI use below code to estimate above scores.
\ndef est_mean_score(m0, m1, n, s=0.99):\n \"\"\"\n m0: original score\n m1: score with predictions inverted in some species\n n: number of species for which predictions were inverted\n s: expected value of the sum of the original score and the predicted inverted score\n \"\"\"\n return 21 / (2 * n) * (m0 - m1) + s / 2\nIt is surprising that the species in the top 11-15 sample size had the largest contribution to the score, even though they were only given less than 20 samples for training.
\n
(source: https://www.kaggle.com/code/tatamikenn/birdclef22-eda-on-the-scored-species )
", "rawMarkdown": "It is surprising that the species in the top 11-15 sample size had the largest contribution to the score, even though they were only given less than 20 samples for training.\n\n\n\n(source: https://www.kaggle.com/code/tatamikenn/birdclef22-eda-on-the-scored-species )",
"votes": null
},
{
"id": "1775366",
"postDate": "05/03/2022 00:32:20",
"content": "\n\nThis result indicates that a large frequency of occurrence in the training data does not necessarily mean a large contribution to the public LB score.
\n
This may be due to the difference in the distribution of labels between the training dataset and the test soundscape.
\nEven if the frequency of occurrence of a label in the training dataset is high, it does not necessarily mean that the frequency in the test soundscape is high. What may be happening here is that the top five most frequent labels in the training dataset have a much lower frequency of occurrence in the test soundscape.
\nFor the labels that appear more frequently in training, the model is likely to output more positive predictions than for the other species. In contrast, in the test soundscape, the model generates more FPs when the frequency of occurrence is lower than expected. This would result in a smaller TNR and thus a smaller contribution to the score for these species.
As evidence to support the above reasoning, I cite the following comment. According to this, 10 of the 21 species were initially targeted by the host, and the other 11 were accidentally recorded on the collection of target calls. Therefore, it is possible that samples of non-target calls in the test soundscape are less frequent than those of the target calls.
\n\n\nIf I recall correctly, the other 11 were recorded incidentally during attempts to get audio for the 10 that are of primary interest.
\n
\n https://www.kaggle.com/competitions/birdclef-2022/discussion/307752#1730837
Also, to begin with, most of the sound sources of non-Hawaiian endemic species downloaded from xeno-canto were recorded outside of Hawaii, and it is natural to imagine that the frequency of labels in the training data set is considerably higher than the frequency of labels actually recorded in Hawaii .
", "rawMarkdown": "As evidence to support the above reasoning, I cite the following comment. According to this, 10 of the 21 species were initially targeted by the host, and the other 11 were accidentally recorded on the collection of target calls. Therefore, it is possible that samples of non-target calls in the test soundscape are less frequent than those of the target calls.\n\n> If I recall correctly, the other 11 were recorded incidentally during attempts to get audio for the 10 that are of primary interest.\nhttps://www.kaggle.com/competitions/birdclef-2022/discussion/307752#1730837\n\nAlso, to begin with, most of the sound sources of non-Hawaiian endemic species downloaded from xeno-canto were recorded outside of Hawaii, and it is natural to imagine that the frequency of labels in the training data set is considerably higher than the frequency of labels actually recorded in Hawaii .", "votes": null }, { "id": "1775373", "postDate": "05/03/2022 00:47:50", "content": "The implications derived from the above are as follows. The frequency of species labels in the training data could be adjusted to be closer to that in the test soundscape, thereby increasing the score contribution of species with low score contribution at this point in time.
", "rawMarkdown": "The implications derived from the above are as follows. The frequency of species labels in the training data could be adjusted to be closer to that in the test soundscape, thereby increasing the score contribution of species with low score contribution at this point in time.", "votes": null }, { "id": "1776347", "postDate": "05/03/2022 23:49:51", "content": "You've been publishing very useful information, thank you! 😄
\nDo you think the most likely scenario is that the most frequent species regain their contribution to the scores in the private set?
", "rawMarkdown": "You've been publishing very useful information, thank you! 😄\n\nDo you think the most likely scenario is that the most frequent species regain their contribution to the scores in the private set?", "votes": null }, { "id": "1776362", "postDate": "05/04/2022 00:11:06", "content": "@flrotm While we cannot say for sure about the distribution of the private test set, the conclusion of this thread is that we need to be aware of the difference in species distribution between the training data set and the public (private) data set.
", "rawMarkdown": "flrotm While we cannot say for sure about the distribution of the private test set, the conclusion of this thread is that we need to be aware of the difference in species distribution between the training data set and the public (private) data set.", "votes": null }, { "id": "1778251", "postDate": "05/05/2022 07:33:51", "content": "In general, let U be a subset of the species, and the average score s_U of U can be calculated as follows.
\n$$
\ns_U = \\frac{21}{2|U|}(m_0 - m_1) + 0.99 \\tag{7}
\n$$
In this case, since m_0 and m_1 have significant digits to the third decimal place, the theoretical upper bound of error for s_U can be calculated as follows.
\n$$
\n\\text{error}(s_U) = \\frac{21}{2|U|}0.01 \\tag{8}
\n$$
Since|S|=5 for top5, mid_top5 and mid_low_5, the error of the score is as follows.
\n$$
\n\\text{error}(s_{\\text{top5}}) = 0.021 \\tag{9}
\n$$
Also, since|S|=6 for low6, we obtain the following result.
\n$$
\n\\text{error}(s_{\\text{low6}}) = 0.0175 \\tag{10}
\n$$
Furthermore, for |S|=1, we get the following result.
\n$$
\n\\text{error}(s_i) = 0.105 \\tag{11}
\n$$
Note: I confirmed assumption 2 seems to be correct, by obtaining below result:
\nscore(y, y_pred) + score(y, 1 - y_pred) = 0.71 + 0.28 = 0.99\nStrictly speaking, the sum of the scores is 0.99 instead of 1. This is probably because the LB scores are truncated to the third decimal place. In that case, the expected value of the sum of scores would be 0.99.
", "votes": null, "replies": [ { "id": 1774284, "author_name": "tatamikenn", "author_url": "", "post_date": "05/02/2022 03:49:35", "content": "In Experiment 2, assuming that the sum of the scores with and without inverting predictions is 0.99, the estimated scores are as follows:
\ns_top10 = 0.726\ns_low11 = 0.695\nI have discovered one interesting fact.
\nIn the public notebook model (show reference [1] in the original post), the top 5 most frequent bird species in the training data contribute an estimated average of only 0.558 to the score. The top 6 to 10 each contribute the most to the score, with an estimated mean score of 0.894.
\nThis result indicates that a large frequency of occurrence in the training data does not necessarily mean a large contribution to the public LB score. The reason is unknown at this time, but it is possible that the distribution of species in the public test data is quite variable from species to species or that some species are excluded from the public test data calculations.
top5, mid_top5, mid_low5, low6 = \n(['skylar', 'houfin', 'jabwar', 'warwhe1', 'yefcan'],\n ['apapan', 'iiwi', 'omao', 'hawama', 'hawcre'],\n ['barpet', 'akiapo', 'elepai', 'aniani', 'hawgoo'],\n ['ercfra', 'hawpet1', 'puaioh', 'hawhaw', 'crehon', 'maupar'])\n\noriginal score: 0.71\nscore with inverted prediction of top5: 0.68\nscore with inverted prediction of mid_top5: 0.52\nscore with inverted prediction of mid_low5: 0.56\nscore with inverted prediction of top5: 0.65\n\nestimated mean score of top5: 0.558\nestimated mean score of mid_top5: 0.894\nestimated mean score of mid_low5: 0.810\nestimated mean score of low6: 0.600\nI use below code to estimate above scores.
\ndef est_mean_score(m0, m1, n, s=0.99):\n \"\"\"\n m0: original score\n m1: score with predictions inverted in some species\n n: number of species for which predictions were inverted\n s: expected value of the sum of the original score and the predicted inverted score\n \"\"\"\n return 21 / (2 * n) * (m0 - m1) + s / 2\nIt is surprising that the species in the top 11-15 sample size had the largest contribution to the score, even though they were only given less than 20 samples for training.
\n
(source: https://www.kaggle.com/code/tatamikenn/birdclef22-eda-on-the-scored-species )
", "votes": null, "replies": [] }, { "id": 1775366, "author_name": "tatamikenn", "author_url": "", "post_date": "05/03/2022 00:32:20", "content": "\n\nThis result indicates that a large frequency of occurrence in the training data does not necessarily mean a large contribution to the public LB score.
\n
This may be due to the difference in the distribution of labels between the training dataset and the test soundscape.
\nEven if the frequency of occurrence of a label in the training dataset is high, it does not necessarily mean that the frequency in the test soundscape is high. What may be happening here is that the top five most frequent labels in the training dataset have a much lower frequency of occurrence in the test soundscape.
\nFor the labels that appear more frequently in training, the model is likely to output more positive predictions than for the other species. In contrast, in the test soundscape, the model generates more FPs when the frequency of occurrence is lower than expected. This would result in a smaller TNR and thus a smaller contribution to the score for these species.
As evidence to support the above reasoning, I cite the following comment. According to this, 10 of the 21 species were initially targeted by the host, and the other 11 were accidentally recorded on the collection of target calls. Therefore, it is possible that samples of non-target calls in the test soundscape are less frequent than those of the target calls.
\n\n\nIf I recall correctly, the other 11 were recorded incidentally during attempts to get audio for the 10 that are of primary interest.
\n
\n https://www.kaggle.com/competitions/birdclef-2022/discussion/307752#1730837
Also, to begin with, most of the sound sources of non-Hawaiian endemic species downloaded from xeno-canto were recorded outside of Hawaii, and it is natural to imagine that the frequency of labels in the training data set is considerably higher than the frequency of labels actually recorded in Hawaii .
", "votes": null, "replies": [] }, { "id": 1775373, "author_name": "tatamikenn", "author_url": "", "post_date": "05/03/2022 00:47:50", "content": "The implications derived from the above are as follows. The frequency of species labels in the training data could be adjusted to be closer to that in the test soundscape, thereby increasing the score contribution of species with low score contribution at this point in time.
", "votes": null, "replies": [] }, { "id": 1776347, "author_name": "flrotm", "author_url": "", "post_date": "05/03/2022 23:49:51", "content": "You've been publishing very useful information, thank you! 😄
\nDo you think the most likely scenario is that the most frequent species regain their contribution to the scores in the private set?
", "votes": null, "replies": [] }, { "id": 1776362, "author_name": "tatamikenn", "author_url": "", "post_date": "05/04/2022 00:11:06", "content": "@flrotm While we cannot say for sure about the distribution of the private test set, the conclusion of this thread is that we need to be aware of the difference in species distribution between the training data set and the public (private) data set.
", "votes": null, "replies": [] } ] }, { "id": 1778251, "author_name": "tatamikenn", "author_url": "", "post_date": "05/05/2022 07:33:51", "content": "In general, let U be a subset of the species, and the average score s_U of U can be calculated as follows.
\n$$
\ns_U = \\frac{21}{2|U|}(m_0 - m_1) + 0.99 \\tag{7}
\n$$
In this case, since m_0 and m_1 have significant digits to the third decimal place, the theoretical upper bound of error for s_U can be calculated as follows.
\n$$
\n\\text{error}(s_U) = \\frac{21}{2|U|}0.01 \\tag{8}
\n$$
Since|S|=5 for top5, mid_top5 and mid_low_5, the error of the score is as follows.
\n$$
\n\\text{error}(s_{\\text{top5}}) = 0.021 \\tag{9}
\n$$
Also, since|S|=6 for low6, we obtain the following result.
\n$$
\n\\text{error}(s_{\\text{low6}}) = 0.0175 \\tag{10}
\n$$
Furthermore, for |S|=1, we get the following result.
\n$$
\n\\text{error}(s_i) = 0.105 \\tag{11}
\n$$
\n\n(source: https://www.kaggle.com/code/tatamikenn/birdclef22-eda-on-the-scored-species )",
"1775366": "> This result indicates that a large frequency of occurrence in the training data does not necessarily mean a large contribution to the public LB score.\n\nThis may be due to the difference in the distribution of labels between the training dataset and the test soundscape.\nEven if the frequency of occurrence of a label in the training dataset is high, it does not necessarily mean that the frequency in the test soundscape is high. What may be happening here is that the top five most frequent labels in the training dataset have a much lower frequency of occurrence in the test soundscape.\nFor the labels that appear more frequently in training, the model is likely to output more positive predictions than for the other species. In contrast, in the test soundscape, the model generates more FPs when the frequency of occurrence is lower than expected. This would result in a smaller TNR and thus a smaller contribution to the score for these species.",
"1775368": "As evidence to support the above reasoning, I cite the following comment. According to this, 10 of the 21 species were initially targeted by the host, and the other 11 were accidentally recorded on the collection of target calls. Therefore, it is possible that samples of non-target calls in the test soundscape are less frequent than those of the target calls.\n\n> If I recall correctly, the other 11 were recorded incidentally during attempts to get audio for the 10 that are of primary interest.\nhttps://www.kaggle.com/competitions/birdclef-2022/discussion/307752#1730837\n\nAlso, to begin with, most of the sound sources of non-Hawaiian endemic species downloaded from xeno-canto were recorded outside of Hawaii, and it is natural to imagine that the frequency of labels in the training data set is considerably higher than the frequency of labels actually recorded in Hawaii .",
"1775373": "The implications derived from the above are as follows. The frequency of species labels in the training data could be adjusted to be closer to that in the test soundscape, thereby increasing the score contribution of species with low score contribution at this point in time.",
"1776347": "You've been publishing very useful information, thank you! 😄\n\nDo you think the most likely scenario is that the most frequent species regain their contribution to the scores in the private set?",
"1776362": "flrotm While we cannot say for sure about the distribution of the private test set, the conclusion of this thread is that we need to be aware of the difference in species distribution between the training data set and the public (private) data set.",
"1778251": "# Theoretical error in scores\n\nIn general, let U be a subset of the species, and the average score s_U of U can be calculated as follows.\n\n$$\ns_U = \\frac{21}{2\\|U\\|}(m_0 - m_1) + 0.99 \\tag{7}\n$$\n\nIn this case, since m_0 and m_1 have significant digits to the third decimal place, the theoretical upper bound of error for s_U can be calculated as follows.\n\n$$\n\\text{error}(s_U) = \\frac{21}{2\\|U\\|}0.01 \\tag{8}\n$$\n\nSince|S|=5 for top5, mid_top5 and mid_low_5, the error of the score is as follows.\n\n$$\n\\text{error}(s_{\\text{top5}}) = 0.021 \\tag{9}\n$$\n\nAlso, since|S|=6 for low6, we obtain the following result.\n\n$$\n\\text{error}(s_{\\text{low6}}) = 0.0175 \\tag{10}\n$$\n\nFurthermore, for |S|=1, we get the following result.\n\n$$\n\\text{error}(s_i) = 0.105 \\tag{11}\n$$"
},
"source": "meta"
}