![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1920073%2F91ebe60d08b67b1f0953362d338ed46b%2FScreenshot%202020-06-21%20at%2009.30.23.png?generation=1592721081921068&alt=media\")
Hi everyone!
\n\nI have created kernel with explanation and implementation of competition metrics:
\n\n\n\nHope it helps you!
\n\nHi everyone!
\n\nI have created kernel with explanation and implementation of competition metrics:
\n\n\n\nHope it helps you!
\n\nexample:\n| actual | pred | TP | FN | FP | row's micro F1 |\n| --- | --- | --- | ---| ---| ---| \n| \"a b\"| \"a b c\" | 2 | 0 | 1 | 4/5 | \n| \"m n o\"| \"m p a\" | 1 | 2 | 2 | 1/3 | \n| Total | |3| 2| 3| |
\n\n(1) AVG of row wise F1 = 17/30\n(2) F1 using total TP, FN, FP = 6/11
\n\nI am thinking comp metric is (1) but I can be wrong.
", "rawMarkdown": "example:\n| actual | pred | TP | FN | FP | row's micro F1 |\n| --- | --- | --- | ---| ---| ---| \n| \"a b\"| \"a b c\" | 2 | 0 | 1 | 4/5 | \n| \"m n o\"| \"m p a\" | 1 | 2 | 2 | 1/3 | \n| Total | |3| 2| 3| |\n\n(1) AVG of row wise F1 = 17/30\n(2) F1 using total TP, FN, FP = 6/11\n\nI am thinking comp metric is (1) but I can be wrong.", "votes": null }, { "id": "895161", "postDate": "06/21/2020 06:38:38", "content": "@dhananjay3 thank you for example!
\n\nIn sklearn documentation about f1 score:
\n\n
\n\n\nSubmissions will be evaluated based on their row-wise micro averaged F1 score.
\n
So I suppose that correct variant is (2) (in your example). Globally counting the total TP, FN and FP. But I can be wrong also.
\n\nWhat are you thinking about competition metric? @stefankahl @sohier @tomdenton
", "rawMarkdown": "dhananjay3 thank you for example!\n\n[In sklearn documentation about f1 score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html):\n\n\n\n> Submissions will be evaluated based on their row-wise **micro averaged F1 score**.\n\nSo I suppose that correct variant is (2) (in your example). Globally counting the total TP, FN and FP. But I can be wrong also. \n\nWhat are you thinking about competition metric? @stefankahl @sohier @tomdenton", "votes": null }, { "id": "895177", "postDate": "06/21/2020 06:52:18", "content": "if you observe closely while we are calculating row wise F1 we are counting total TP, FP, FN in this row. and that's what I believe row wise micro F1 will be. for comparison row wise Macro F1 will be average of all F1 scores for all 264 classes for this row.
", "rawMarkdown": "if you observe closely while we are calculating row wise F1 we are counting total TP, FP, FN in this row. and that's what I believe row wise micro F1 will be. for comparison row wise Macro F1 will be average of all F1 scores for all 264 classes for this row.", "votes": null }, { "id": "895184", "postDate": "06/21/2020 06:56:31", "content": "I did a little experiment to check this exact thing. given an all 'nocall' submission scores 0.54. and if we modify few rows to predict all 264 classes that row's F1 will be nearly zero but number of this type of rows is low then (1) score should not be affected much. but in case (2) total FN and FP will be greatly increased and score (2) should be highly affected.
\n\nEdit: looks like I forgot to import random and out of submissions for today. so I will update results tomorrow.
", "rawMarkdown": "I did a little experiment to check this exact thing. given an all 'nocall' submission scores 0.54. and if we modify few rows to predict all 264 classes that row's F1 will be nearly zero but number of this type of rows is low then (1) score should not be affected much. but in case (2) total FN and FP will be greatly increased and score (2) should be highly affected.\n\nEdit: looks like I forgot to import random and out of submissions for today. so I will update results tomorrow.", "votes": null }, { "id": "895265", "postDate": "06/21/2020 08:03:48", "content": "@carriesmi good idea, thank you! I have tried to make submission with this update:
\n\n\nfor i in range(min([10, submission.shape[0]])):\n submission.iloc[i]['birds'] = ' '.join(all_birds)\n
It gives 0.54. I suppose (1) is correct finally. I will make some corrections in calculating metrics.
\n\nThank you @carriesmi @dhananjay3 for getting right answer together! :)
\n\nI think you should make corrections in description metrics, current version is not clearly understanding - many competitors (not only novices) will ask these questions again and again, thanks in advance @stefankahl @sohier @tomdenton
", "rawMarkdown": "carriesmi good idea, thank you! I have tried to make submission with this update:\n\n```\nfor i in range(min([10, submission.shape[0]])):\n submission.iloc[i]['birds'] = ' '.join(all_birds)\n```\n\nIt gives 0.54. I suppose (1) is correct finally. I will make some corrections in calculating metrics.\n\nThank you @carriesmi @dhananjay3 for getting right answer together! :)\n\nI think you should make corrections in description metrics, current version is not clearly understanding - many competitors (not only novices) will ask these questions again and again, thanks in advance @stefankahl @sohier @tomdenton", "votes": null }, { "id": "895978", "postDate": "06/21/2020 18:20:00", "content": "Do I understand right that 0.54 for nocall means exactly that 54% of all observations have nocall, i.e. only 46% observations have birds in them?
", "rawMarkdown": "Do I understand right that 0.54 for nocall means exactly that 54% of all observations have nocall, i.e. only 46% observations have birds in them?", "votes": null }, { "id": "979157", "postDate": "08/20/2020 16:58:47", "content": "In the public test set, it seems possible. But I think for private set it would be completely different.
", "rawMarkdown": "In the public test set, it seems possible. But I think for private set it would be completely different.", "votes": null } ], "comments": [ { "id": 895144, "author_name": "dhananjay3", "author_url": "", "post_date": "06/21/2020 06:12:32", "content": "example:\n| actual | pred | TP | FN | FP | row's micro F1 |\n| --- | --- | --- | ---| ---| ---| \n| \"a b\"| \"a b c\" | 2 | 0 | 1 | 4/5 | \n| \"m n o\"| \"m p a\" | 1 | 2 | 2 | 1/3 | \n| Total | |3| 2| 3| |
\n\n(1) AVG of row wise F1 = 17/30\n(2) F1 using total TP, FN, FP = 6/11
\n\nI am thinking comp metric is (1) but I can be wrong.
", "votes": null, "replies": [ { "id": 895161, "author_name": "shonenkov", "author_url": "", "post_date": "06/21/2020 06:38:38", "content": "@dhananjay3 thank you for example!
\n\nIn sklearn documentation about f1 score:
\n\n
\n\n\nSubmissions will be evaluated based on their row-wise micro averaged F1 score.
\n
So I suppose that correct variant is (2) (in your example). Globally counting the total TP, FN and FP. But I can be wrong also.
\n\nWhat are you thinking about competition metric? @stefankahl @sohier @tomdenton
", "votes": null, "replies": [] }, { "id": 895177, "author_name": "dhananjay3", "author_url": "", "post_date": "06/21/2020 06:52:18", "content": "if you observe closely while we are calculating row wise F1 we are counting total TP, FP, FN in this row. and that's what I believe row wise micro F1 will be. for comparison row wise Macro F1 will be average of all F1 scores for all 264 classes for this row.
", "votes": null, "replies": [] }, { "id": 895184, "author_name": "carriesmi", "author_url": "", "post_date": "06/21/2020 06:56:31", "content": "I did a little experiment to check this exact thing. given an all 'nocall' submission scores 0.54. and if we modify few rows to predict all 264 classes that row's F1 will be nearly zero but number of this type of rows is low then (1) score should not be affected much. but in case (2) total FN and FP will be greatly increased and score (2) should be highly affected.
\n\nEdit: looks like I forgot to import random and out of submissions for today. so I will update results tomorrow.
", "votes": null, "replies": [] }, { "id": 895265, "author_name": "shonenkov", "author_url": "", "post_date": "06/21/2020 08:03:48", "content": "@carriesmi good idea, thank you! I have tried to make submission with this update:
\n\n\nfor i in range(min([10, submission.shape[0]])):\n submission.iloc[i]['birds'] = ' '.join(all_birds)\n
It gives 0.54. I suppose (1) is correct finally. I will make some corrections in calculating metrics.
\n\nThank you @carriesmi @dhananjay3 for getting right answer together! :)
\n\nI think you should make corrections in description metrics, current version is not clearly understanding - many competitors (not only novices) will ask these questions again and again, thanks in advance @stefankahl @sohier @tomdenton
", "votes": null, "replies": [] } ] }, { "id": 895978, "author_name": "snovik1975", "author_url": "", "post_date": "06/21/2020 18:20:00", "content": "Do I understand right that 0.54 for nocall means exactly that 54% of all observations have nocall, i.e. only 46% observations have birds in them?
", "votes": null, "replies": [ { "id": 979157, "author_name": "rsinda", "author_url": "", "post_date": "08/20/2020 16:58:47", "content": "In the public test set, it seems possible. But I think for private set it would be completely different.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "894798": "Hi everyone!\n\nI have created kernel with explanation and implementation of competition metrics:\n\n[[Competition Metrics]](https://www.kaggle.com/shonenkov/competition-metrics)\n\nHope it helps you!\n\n### If you find any imprecision in calculating metrics, please, let me know!", "895144": "example:\n| actual | pred | TP | FN | FP | row's micro F1 |\n| --- | --- | --- | ---| ---| ---| \n| \"a b\"| \"a b c\" | 2 | 0 | 1 | 4/5 | \n| \"m n o\"| \"m p a\" | 1 | 2 | 2 | 1/3 | \n| Total | |3| 2| 3| |\n\n(1) AVG of row wise F1 = 17/30\n(2) F1 using total TP, FN, FP = 6/11\n\nI am thinking comp metric is (1) but I can be wrong.", "895161": "dhananjay3 thank you for example!\n\n[In sklearn documentation about f1 score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html):\n\n\n\n> Submissions will be evaluated based on their row-wise **micro averaged F1 score**.\n\nSo I suppose that correct variant is (2) (in your example). Globally counting the total TP, FN and FP. But I can be wrong also. \n\nWhat are you thinking about competition metric? @stefankahl @sohier @tomdenton", "895177": "if you observe closely while we are calculating row wise F1 we are counting total TP, FP, FN in this row. and that's what I believe row wise micro F1 will be. for comparison row wise Macro F1 will be average of all F1 scores for all 264 classes for this row.", "895184": "I did a little experiment to check this exact thing. given an all 'nocall' submission scores 0.54. and if we modify few rows to predict all 264 classes that row's F1 will be nearly zero but number of this type of rows is low then (1) score should not be affected much. but in case (2) total FN and FP will be greatly increased and score (2) should be highly affected.\n\nEdit: looks like I forgot to import random and out of submissions for today. so I will update results tomorrow.", "895265": "carriesmi good idea, thank you! I have tried to make submission with this update:\n\n```\nfor i in range(min([10, submission.shape[0]])):\n submission.iloc[i]['birds'] = ' '.join(all_birds)\n```\n\nIt gives 0.54. I suppose (1) is correct finally. I will make some corrections in calculating metrics.\n\nThank you @carriesmi @dhananjay3 for getting right answer together! :)\n\nI think you should make corrections in description metrics, current version is not clearly understanding - many competitors (not only novices) will ask these questions again and again, thanks in advance @stefankahl @sohier @tomdenton", "895978": "Do I understand right that 0.54 for nocall means exactly that 54% of all observations have nocall, i.e. only 46% observations have birds in them?", "979157": "In the public test set, it seems possible. But I think for private set it would be completely different." }, "source": "meta" }