Submission with a constant value of 1 scores 0.414 on the public leaderboard and this confuses me a little.
\nLet's say test set has S samples = P positives + N negatives. Then our constant submission has
\ntp = P, fp = N, fn = 0. Then
\n$$
\nF_1 = \\frac{2tp}{2tp+fp+fn} = \\frac{2P}{2P+N+0} = \\frac{2P}{P+S} = \\frac{2\\frac{P}{S}}{\\frac{P}{S}+1}=0.414,
\n$$
\nhence
\n$$
\n\\frac{P}{S}\\approx0.261
\n$$
\nwhich is much lower than the average proportion of positive samples in the training set. Where am I wrong?
UPD And the second strange thing for me. Submission with a constant value of 0 scores 0.226 on the LB. Shouldn't the F1 metric equal zero when tp=0?
", "messages": [ { "id": 2133525, "postDate": "2023-02-07T13:15:45.747Z", "content": "Submission with a constant value of 1 scores 0.414 on the public leaderboard and this confuses me a little.
\nLet's say test set has S samples = P positives + N negatives. Then our constant submission has
\ntp = P, fp = N, fn = 0. Then
\n$$
\nF_1 = \\frac{2tp}{2tp+fp+fn} = \\frac{2P}{2P+N+0} = \\frac{2P}{P+S} = \\frac{2\\frac{P}{S}}{\\frac{P}{S}+1}=0.414,
\n$$
\nhence
\n$$
\n\\frac{P}{S}\\approx0.261
\n$$
\nwhich is much lower than the average proportion of positive samples in the training set. Where am I wrong?
UPD And the second strange thing for me. Submission with a constant value of 0 scores 0.226 on the LB. Shouldn't the F1 metric equal zero when tp=0?
", "rawMarkdown": "Submission with a constant value of **1** scores **0.414** on the public leaderboard and this confuses me a little. \n\nLet's say test set has **S** samples = **P** positives + **N** negatives. Then our constant submission has \ntp = P, fp = N, fn = 0. Then\n$$\nF_1 = \\frac{2tp}{2tp+fp+fn} = \\frac{2P}{2P+N+0} = \\frac{2P}{P+S} = \\frac{2\\frac{P}{S}}{\\frac{P}{S}+1}=0.414,\n$$\nhence\n$$\n\\frac{P}{S}\\approx0.261\n$$\nwhich is much lower than the average proportion of positive samples in the training set. Where am I wrong?\n\n**UPD** And the second strange thing for me. Submission with a constant value of **0** scores **0.226** on the LB. Shouldn't the F1 metric equal zero when tp=0?", "votes": 27 }, { "id": 2133695, "postDate": "2023-02-07T15:02:12.713Z", "content": "The competition metric is sklearn.metrics.f1_score(true, pred, average='macro'). We compute the f1_score for class 0 then compute the f1_score for class 1 then average the two equally. Blog explains this here
Oh, thanks for clarifying @cdeotte I didn't find that in Competition description.
", "rawMarkdown": "Oh, thanks for clarifying @cdeotte I didn't find that in Competition description.", "votes": 3, "replies": [ { "id": 2133714, "postDate": "2023-02-07T15:10:27.953Z", "content": "I was confused too. I tried different f1_score metrics until I got validation score of all zeros to equal 0.226 and all ones to equal 0.414 to match LB results.
since f1_score macro is average of positive f1_score and negative f1_scores.
\n@seshurajup LB truncates scores, so
\n$$
\n0.414\\leq\\frac{P}{2P+N}<0.415
\n$$
\nand
\n$$
\n0.226\\leq\\frac{N}{2N+P}<0.227
\n$$
\nhence
\n$$
\n2.406976<\\frac{P}{N}<2.441176
\n$$
\nand
\n$$
\n2.405286<\\frac{P}{N}<2.424778
\n$$
Finally,
\n$$
\n2.406976<\\frac{P}{N}<2.424778
\n$$
@allvor Thanks for your explanation. Now it makes sense more clear.
", "rawMarkdown": "@allvor Thanks for your explanation. Now it makes sense more clear." } ] }, { "id": 2134322, "postDate": "2023-02-07T22:26:34.480Z", "content": "Great find Chris!
", "rawMarkdown": "Great find Chris!", "votes": 2 }, { "id": 2134551, "postDate": "2023-02-08T04:53:15.227Z", "content": "great blog, thank for sharing!
", "rawMarkdown": "great blog, thank for sharing!", "votes": 2 }, { "id": 2146758, "postDate": "2023-02-16T05:42:35.743Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 2140067, "postDate": "2023-02-11T12:34:24.667Z", "content": "here is simple code to calculate this metric; it is a lot faster than sklearn.metrics.f1_score.
\ndef f1l(labels, preds): # my calc of macro F1 - way faster
\n c11 = (preds + labels == 2).mean()
\n c00 = (preds + labels == 0).mean()
\n e = 1 - (1 - c00 - c11) / (1 - (c00 - c11)**2)
\n return e
Here is above code implemented as CatboostClassifier custom metric
\n (): \n c11 = (preds + labels == ).mean()\n c00 = (preds + labels == ).mean()\n e = - ( - c00 - c11) / ( - (c00 - c11)**)\n e\n\n ():\n threshold = \n ():\n error\n\n ():\n \n\n () -> [, ]:\n approx = approxes[]\n approx = [ / ( + np.exp(-x)) x approx]\n pred_labels = [ x > threshold x approx]\n f1 = f1_avg(pred_labels,target)\n f1, \nI am still struggling with numba warnings. I tried annotating it with @njit but it keeps having some numba errors. If anybody finds a solution I will highly appreciate it.
", "rawMarkdown": "I am still struggling with numba warnings. I tried annotating it with @njit but it keeps having some numba errors. If anybody finds a solution I will highly appreciate it." } ] } ] }, { "id": 2133687, "postDate": "2023-02-07T14:59:24.740Z", "content": "Looks like a mood-enhancing bonus: score+=0.2
", "rawMarkdown": "Looks like a mood-enhancing bonus: score+=0.2 ", "votes": 1 } ], "comments": [ { "id": 2133695, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-02-07T15:02:12.713000", "content": "The competition metric is sklearn.metrics.f1_score(true, pred, average='macro'). We compute the f1_score for class 0 then compute the f1_score for class 1 then average the two equally. Blog explains this here
Oh, thanks for clarifying @cdeotte I didn't find that in Competition description.
", "votes": 3, "replies": [ { "id": 2133714, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-02-07T15:10:27.953000", "content": "I was confused too. I tried different f1_score metrics until I got validation score of all zeros to equal 0.226 and all ones to equal 0.414 to match LB results.
since f1_score macro is average of positive f1_score and negative f1_scores.
\n@seshurajup LB truncates scores, so
\n$$
\n0.414\\leq\\frac{P}{2P+N}<0.415
\n$$
\nand
\n$$
\n0.226\\leq\\frac{N}{2N+P}<0.227
\n$$
\nhence
\n$$
\n2.406976<\\frac{P}{N}<2.441176
\n$$
\nand
\n$$
\n2.405286<\\frac{P}{N}<2.424778
\n$$
Finally,
\n$$
\n2.406976<\\frac{P}{N}<2.424778
\n$$
@allvor Thanks for your explanation. Now it makes sense more clear.
", "votes": 0, "replies": [] } ] }, { "id": 2134322, "author_name": "Mayukh Bhattacharyya", "author_url": "", "post_date": "2023-02-07T22:26:34.480000", "content": "Great find Chris!
", "votes": 2, "replies": [] }, { "id": 2134551, "author_name": "Tim Yee", "author_url": "", "post_date": "2023-02-08T04:53:15.227000", "content": "great blog, thank for sharing!
", "votes": 2, "replies": [] }, { "id": 2146758, "author_name": "", "author_url": "", "post_date": "2023-02-16T05:42:35.743000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 2140067, "author_name": "Youri Matiounine", "author_url": "", "post_date": "2023-02-11T12:34:24.667000", "content": "here is simple code to calculate this metric; it is a lot faster than sklearn.metrics.f1_score.
\ndef f1l(labels, preds): # my calc of macro F1 - way faster
\n c11 = (preds + labels == 2).mean()
\n c00 = (preds + labels == 0).mean()
\n e = 1 - (1 - c00 - c11) / (1 - (c00 - c11)**2)
\n return e
Here is above code implemented as CatboostClassifier custom metric
\n (): \n c11 = (preds + labels == ).mean()\n c00 = (preds + labels == ).mean()\n e = - ( - c00 - c11) / ( - (c00 - c11)**)\n e\n\n ():\n threshold = \n ():\n error\n\n ():\n \n\n () -> [, ]:\n approx = approxes[]\n approx = [ / ( + np.exp(-x)) x approx]\n pred_labels = [ x > threshold x approx]\n f1 = f1_avg(pred_labels,target)\n f1, \nI am still struggling with numba warnings. I tried annotating it with @njit but it keeps having some numba errors. If anybody finds a solution I will highly appreciate it.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2133687, "author_name": "Wojtek Rosa", "author_url": "", "post_date": "2023-02-07T14:59:24.740000", "content": "Looks like a mood-enhancing bonus: score+=0.2
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "2133525": "Submission with a constant value of **1** scores **0.414** on the public leaderboard and this confuses me a little. \n\nLet's say test set has **S** samples = **P** positives + **N** negatives. Then our constant submission has \ntp = P, fp = N, fn = 0. Then\n$$\nF_1 = \\frac{2tp}{2tp+fp+fn} = \\frac{2P}{2P+N+0} = \\frac{2P}{P+S} = \\frac{2\\frac{P}{S}}{\\frac{P}{S}+1}=0.414,\n$$\nhence\n$$\n\\frac{P}{S}\\approx0.261\n$$\nwhich is much lower than the average proportion of positive samples in the training set. Where am I wrong?\n\n**UPD** And the second strange thing for me. Submission with a constant value of **0** scores **0.226** on the LB. Shouldn't the F1 metric equal zero when tp=0?", "2133695": "The competition metric is `sklearn.metrics.f1_score(true, pred, average='macro')`. We compute the `f1_score` for class `0` then compute the `f1_score` for class `1` then average the two equally. Blog explains this [here][1]\n\n[1]: https://towardsdatascience.com/micro-macro-weighted-averages-of-f1-score-clearly-explained-b603420b292f#989c", "2140067": "here is simple code to calculate this metric; it is a lot faster than sklearn.metrics.f1_score.\ndef f1l(labels, preds): # my calc of macro F1 - way faster\n c11 = (preds + labels == 2).mean()\n c00 = (preds + labels == 0).mean()\n e = 1 - (1 - c00 - c11) / (1 - (c00 - c11)**2)\n return e", "2133687": "Looks like a mood-enhancing bonus: score+=0.2 " } }