\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1577135%2Fda6a49def3c4dad79a1cc3ded664559e%2FScreenshot%202022-11-02%20230403.png?generation=1667405054127695&alt=media\")
Thanks to @simamumu, it was revealed that the simple baseline (last 20 aids) can get 0.945 on Public leaderboard.
\nIt seems very high to me.
Is there any problem on the competition data? or, is it time to change the evaluation metric?
", "messages": [ { "id": "2014369", "postDate": "11/02/2022 14:21:44", "content": "Thanks to @simamumu, it was revealed that the simple baseline (last 20 aids) can get 0.945 on Public leaderboard.
\nIt seems very high to me.
Is there any problem on the competition data? or, is it time to change the evaluation metric?
", "rawMarkdown": "Thanks to @simamumu, it was revealed that [the simple baseline (last 20 aids)](https://www.kaggle.com/code/simamumu/test-data-last-20-aid-get-lb0-947) can get 0.945 on Public leaderboard. \nIt seems very high to me.\n\nIs there any problem on the competition data? or, is it time to change the evaluation metric?", "votes": null }, { "id": "2014384", "postDate": "11/02/2022 14:27:53", "content": "I believe the points made in this post from a previous competition apply here:
\nhttps://www.kaggle.com/competitions/carvana-image-masking-challenge/discussion/37137
", "rawMarkdown": "I believe the points made in this post from a previous competition apply here:\n\nhttps://www.kaggle.com/competitions/carvana-image-masking-challenge/discussion/37137", "votes": null }, { "id": "2014404", "postDate": "11/02/2022 14:35:24", "content": "Recall@20 seems to be too easy. It can be changed to 5 or 10.
", "rawMarkdown": "Recall@20 seems to be too easy. It can be changed to 5 or 10.", "votes": null }, { "id": "2014436", "postDate": "11/02/2022 14:55:10", "content": "We only have 3 digits and current #1 is 0.012 away from the perfect score.
\nThis leaves 13 unique values (0.988, 0.989, … 1.000) to work on, which seems kinda weird to me
I agree. I think the real questions are - would it be worthwhile spending tons of time working on models that will only improve at the 4th or 5th decimal point and what is the real world value?
", "rawMarkdown": "I agree. I think the real questions are - would it be worthwhile spending tons of time working on models that will only improve at the 4th or 5th decimal point and what is the real world value?", "votes": null }, { "id": "2014457", "postDate": "11/02/2022 15:08:23", "content": "kagglers want hard challengeđŸ˜…
", "rawMarkdown": "kagglers want hard challengeđŸ˜…", "votes": null }, { "id": "2014488", "postDate": "11/02/2022 15:47:20", "content": "Its also weird to use a metric that does not care about the order of the predictions. If a true poisitve its on the 19th position instead of 1st it would achieve a 1.0 on the recall metric. Are they sure this is the way they want to measure the models?
", "rawMarkdown": "Its also weird to use a metric that does not care about the order of the predictions. If a true poisitve its on the 19th position instead of 1st it would achieve a 1.0 on the recall metric. Are they sure this is the way they want to measure the models?", "votes": null }, { "id": "2014513", "postDate": "11/02/2022 16:04:36", "content": "@theoviel you sure 'bout that?
\n
Good old kaggle.
", "rawMarkdown": "Good old kaggle.", "votes": null }, { "id": "2014523", "postDate": "11/02/2022 16:09:24", "content": "I posted a new topic about the strange score:
\nhttps://www.kaggle.com/competitions/otto-recommender-system/discussion/363658
what about MAP@20 ?
", "rawMarkdown": "what about MAP@20 ?", "votes": null }, { "id": "2015296", "postDate": "11/03/2022 07:42:55", "content": "Thanks for your sharing
", "rawMarkdown": "Thanks for your sharing", "votes": null } ], "comments": [ { "id": 2014384, "author_name": "inversion", "author_url": "", "post_date": "11/02/2022 14:27:53", "content": "I believe the points made in this post from a previous competition apply here:
\nhttps://www.kaggle.com/competitions/carvana-image-masking-challenge/discussion/37137
", "votes": null, "replies": [ { "id": 2014436, "author_name": "theoviel", "author_url": "", "post_date": "11/02/2022 14:55:10", "content": "We only have 3 digits and current #1 is 0.012 away from the perfect score.
\nThis leaves 13 unique values (0.988, 0.989, … 1.000) to work on, which seems kinda weird to me
Its also weird to use a metric that does not care about the order of the predictions. If a true poisitve its on the 19th position instead of 1st it would achieve a 1.0 on the recall metric. Are they sure this is the way they want to measure the models?
", "votes": null, "replies": [] }, { "id": 2014513, "author_name": "suicaokhoailang", "author_url": "", "post_date": "11/02/2022 16:04:36", "content": "@theoviel you sure 'bout that?
\n
Good old kaggle.
", "votes": null, "replies": [] }, { "id": 2014523, "author_name": "ttahara", "author_url": "", "post_date": "11/02/2022 16:09:24", "content": "I posted a new topic about the strange score:
\nhttps://www.kaggle.com/competitions/otto-recommender-system/discussion/363658
Recall@20 seems to be too easy. It can be changed to 5 or 10.
", "votes": null, "replies": [ { "id": 2014454, "author_name": "xxxxyyyy80008", "author_url": "", "post_date": "11/02/2022 15:06:05", "content": "I agree. I think the real questions are - would it be worthwhile spending tons of time working on models that will only improve at the 4th or 5th decimal point and what is the real world value?
", "votes": null, "replies": [] } ] }, { "id": 2014457, "author_name": "senkin13", "author_url": "", "post_date": "11/02/2022 15:08:23", "content": "kagglers want hard challengeđŸ˜…
", "votes": null, "replies": [] }, { "id": 2014656, "author_name": "titericz", "author_url": "", "post_date": "11/02/2022 18:13:31", "content": "what about MAP@20 ?
", "votes": null, "replies": [] }, { "id": 2015296, "author_name": "stonehang", "author_url": "", "post_date": "11/03/2022 07:42:55", "content": "Thanks for your sharing
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2014369": "Thanks to @simamumu, it was revealed that [the simple baseline (last 20 aids)](https://www.kaggle.com/code/simamumu/test-data-last-20-aid-get-lb0-947) can get 0.945 on Public leaderboard. \nIt seems very high to me.\n\nIs there any problem on the competition data? or, is it time to change the evaluation metric?", "2014384": "I believe the points made in this post from a previous competition apply here:\n\nhttps://www.kaggle.com/competitions/carvana-image-masking-challenge/discussion/37137", "2014404": "Recall@20 seems to be too easy. It can be changed to 5 or 10.", "2014436": "We only have 3 digits and current #1 is 0.012 away from the perfect score.\nThis leaves 13 unique values (0.988, 0.989, ... 1.000) to work on, which seems kinda weird to me", "2014454": "I agree. I think the real questions are - would it be worthwhile spending tons of time working on models that will only improve at the 4th or 5th decimal point and what is the real world value?", "2014457": "kagglers want hard challengeđŸ˜…", "2014488": "Its also weird to use a metric that does not care about the order of the predictions. If a true poisitve its on the 19th position instead of 1st it would achieve a 1.0 on the recall metric. Are they sure this is the way they want to measure the models?", "2014513": "theoviel you sure 'bout that?\n ", "2014515": "Good old kaggle.", "2014523": "I posted a new topic about the strange score:\nhttps://www.kaggle.com/competitions/otto-recommender-system/discussion/363658", "2014656": "what about MAP@20 ?", "2015296": "Thanks for your sharing" }, "source": "meta" }