I know these teams have all used \"metric hacks\", but I don't know why everyone needs to score on the public rankings. The final ranking is based on private rankings, so why do 20% of the teams need to score on the public rankings? I haven't submitted it for a month now and I haven't used \"metric hacks\" yet.
", "messages": [ { "id": 2824622, "postDate": "2024-05-19T23:33:03.323Z", "content": "I know these teams have all used \"metric hacks\", but I don't know why everyone needs to score on the public rankings. The final ranking is based on private rankings, so why do 20% of the teams need to score on the public rankings? I haven't submitted it for a month now and I haven't used \"metric hacks\" yet.
", "rawMarkdown": "I know these teams have all used \"metric hacks\", but I don't know why everyone needs to score on the public rankings. The final ranking is based on private rankings, so why do 20% of the teams need to score on the public rankings? I haven't submitted it for a month now and I haven't used \"metric hacks\" yet.", "votes": 6 }, { "id": 2836928, "postDate": "2024-05-26T07:03:14.433Z", "content": "Because people are hedging their bets since this competition has become somewhat of a lottery. I Believe most teams will choose a regular model and possibly a 2nd one with a metric hack as their 2 final submissions.
", "rawMarkdown": "Because people are hedging their bets since this competition has become somewhat of a lottery. I Believe most teams will choose a regular model and possibly a 2nd one with a metric hack as their 2 final submissions.", "votes": 1 }, { "id": 2826490, "postDate": "2024-05-21T01:23:33.433Z", "content": "This depends on your strategy. I have experienced several competitions(ICR & HuBMAP) where public LB has no meaning, but participants do not know if the same will happen in this competition.
\nHowever, one thing that both competitions have in common is the addition of unnecessary post-processing.
\n\nHowever, one thing that both competitions have in common is the addition of unnecessary post-processing.
\n
@mitsuyasuhoshino With post-processing, you mean metric hacking or meta-featuring (including model predictions as training feature)?
", "rawMarkdown": "> However, one thing that both competitions have in common is the addition of unnecessary post-processing.\n\n@mitsuyasuhoshino With post-processing, you mean metric hacking or meta-featuring (including model predictions as training feature)?", "votes": 1, "replies": [ { "id": 2827587, "postDate": "2024-05-21T15:08:19.047Z", "content": "@andreasbis Rather than metric hacking, they are overfitting to public LB.
", "rawMarkdown": "@andreasbis Rather than metric hacking, they are overfitting to public LB.", "votes": 1 } ] } ] }, { "id": 2825170, "postDate": "2024-05-20T07:28:56.333Z", "content": "Partly agree that finding parameters that overfit to public dataset is nonsense,
\nbut how can we know if some model (including hacks) performs well on private rankings though we don't have access to private dataset?
", "rawMarkdown": "Partly agree that finding parameters that overfit to public dataset is nonsense,\n\nbut how can we know if some model (including hacks) performs well on private rankings though we don't have access to private dataset?", "votes": 2 }, { "id": 2829819, "postDate": "2024-05-22T20:16:10.667Z", "content": "What about the stability prize in these circumstances?
", "rawMarkdown": "What about the stability prize in these circumstances?", "replies": [ { "id": 2836432, "postDate": "2024-05-25T21:23:26.803Z", "content": "isnt it will be evaluated on private test?
", "rawMarkdown": "isnt it will be evaluated on private test?" } ] }, { "id": 2824666, "postDate": "2024-05-20T01:03:31.420Z", "content": "We are working hard to guess-fit the leaderboard :)
", "rawMarkdown": "We are working hard to guess-fit the leaderboard :)" }, { "id": 2830527, "postDate": "2024-05-23T08:20:17.547Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 2836928, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2024-05-26T07:03:14.433000", "content": "Because people are hedging their bets since this competition has become somewhat of a lottery. I Believe most teams will choose a regular model and possibly a 2nd one with a metric hack as their 2 final submissions.
", "votes": 1, "replies": [] }, { "id": 2826490, "author_name": "mh", "author_url": "", "post_date": "2024-05-21T01:23:33.433000", "content": "This depends on your strategy. I have experienced several competitions(ICR & HuBMAP) where public LB has no meaning, but participants do not know if the same will happen in this competition.
\nHowever, one thing that both competitions have in common is the addition of unnecessary post-processing.
\n\nHowever, one thing that both competitions have in common is the addition of unnecessary post-processing.
\n
@mitsuyasuhoshino With post-processing, you mean metric hacking or meta-featuring (including model predictions as training feature)?
", "votes": 1, "replies": [ { "id": 2827587, "author_name": "mh", "author_url": "", "post_date": "2024-05-21T15:08:19.047000", "content": "@andreasbis Rather than metric hacking, they are overfitting to public LB.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2825170, "author_name": "ano", "author_url": "", "post_date": "2024-05-20T07:28:56.333000", "content": "Partly agree that finding parameters that overfit to public dataset is nonsense,
\nbut how can we know if some model (including hacks) performs well on private rankings though we don't have access to private dataset?
", "votes": 2, "replies": [] }, { "id": 2829819, "author_name": "gromml", "author_url": "", "post_date": "2024-05-22T20:16:10.667000", "content": "What about the stability prize in these circumstances?
", "votes": 0, "replies": [ { "id": 2836432, "author_name": "Andrey Chankin", "author_url": "", "post_date": "2024-05-25T21:23:26.803000", "content": "isnt it will be evaluated on private test?
", "votes": 0, "replies": [] } ] }, { "id": 2824666, "author_name": "xiao-xiao", "author_url": "", "post_date": "2024-05-20T01:03:31.420000", "content": "We are working hard to guess-fit the leaderboard :)
", "votes": 0, "replies": [] }, { "id": 2830527, "author_name": "", "author_url": "", "post_date": "2024-05-23T08:20:17.547000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2824622": "I know these teams have all used \"metric hacks\", but I don't know why everyone needs to score on the public rankings. The final ranking is based on private rankings, so why do 20% of the teams need to score on the public rankings? I haven't submitted it for a month now and I haven't used \"metric hacks\" yet.", "2836928": "Because people are hedging their bets since this competition has become somewhat of a lottery. I Believe most teams will choose a regular model and possibly a 2nd one with a metric hack as their 2 final submissions.", "2826490": "This depends on your strategy. I have experienced several competitions([ICR](https://www.kaggle.com/competitions/icr-identify-age-related-conditions) & [HuBMAP](https://www.kaggle.com/competitions/hubmap-hacking-the-human-vasculature)) where public LB has no meaning, but participants do not know if the same will happen in this competition.\nHowever, one thing that both competitions have in common is the addition of **unnecessary** post-processing.", "2825170": "Partly agree that finding parameters that overfit to public dataset is nonsense,\n\nbut how can we know if some model (including hacks) performs well on private rankings though we don't have access to private dataset?", "2829819": "What about the stability prize in these circumstances?", "2824666": "We are working hard to guess-fit the leaderboard :)", "2830527": "" } }