It seems to me there are not enough digits in the Leaderboard scores. There are a lot of scores which are the same. That is really odd as the first competitor who makes a submission with a score will have a better place than the ones who submitted after him/her.
", "messages": [ { "id": "2072128", "postDate": "12/21/2022 18:26:57", "content": "It seems to me there are not enough digits in the Leaderboard scores. There are a lot of scores which are the same. That is really odd as the first competitor who makes a submission with a score will have a better place than the ones who submitted after him/her.
", "rawMarkdown": "It seems to me there are not enough digits in the Leaderboard scores. There are a lot of scores which are the same. That is really odd as the first competitor who makes a submission with a score will have a better place than the ones who submitted after him/her.", "votes": null }, { "id": "2072210", "postDate": "12/21/2022 21:14:42", "content": "Dont worry.
\n1) Many of those repeated scores are because they indeed submitted the same thing, so, makes sense to rank lower.
\n2) Final score is based on private, which might not necessarily match that \"repeated public score\"
\n3) I want to believe Kaggle ranks at a higher resolution, but does not show.
Let us hope you're right. For nb. 1: Are we to understand that they are forks of high score notebooks? This practice is a problem for high level competitions. And for nb. 3 : It should show, I might say, because otherwise we get situations like this.
", "rawMarkdown": "Let us hope you're right. For nb. 1: Are we to understand that they are forks of high score notebooks? This practice is a problem for high level competitions. And for nb. 3 : It should show, I might say, because otherwise we get situations like this.", "votes": null }, { "id": "2072335", "postDate": "12/22/2022 01:19:59", "content": "For 1) Yes, most likely true. And for 3) You should not rely on the Leaderboard score, if you are, that is a problem, trust your CV instead.
", "rawMarkdown": "For 1) Yes, most likely true. And for 3) You should not rely on the Leaderboard score, if you are, that is a problem, trust your CV instead.", "votes": null }, { "id": "2072440", "postDate": "12/22/2022 05:36:43", "content": "You are right too. CV is always to be trusted. Nevertheless the leaderboard is still weird. Well, we shall see at the end.
", "rawMarkdown": "You are right too. CV is always to be trusted. Nevertheless the leaderboard is still weird. Well, we shall see at the end.", "votes": null }, { "id": "2077862", "postDate": "12/27/2022 23:17:08", "content": "Don't worry about the scores on the leaderboard being the same for multiple entries. This is normal, and Kaggle takes into account the private leaderboard scores when determining ranks. The public scores are just a representation of everyone's progress, and the final scores will be based on the private leaderboard. Kaggle generally uses a higher resolution in its rankings, even if it doesn't always show it on the leaderboard.
\nThe Devastator.
", "rawMarkdown": "Don't worry about the scores on the leaderboard being the same for multiple entries. This is normal, and Kaggle takes into account the private leaderboard scores when determining ranks. The public scores are just a representation of everyone's progress, and the final scores will be based on the private leaderboard. Kaggle generally uses a higher resolution in its rankings, even if it doesn't always show it on the leaderboard. \n\nThe Devastator.", "votes": null }, { "id": "2079041", "postDate": "12/28/2022 23:00:01", "content": "Hallo, you are right, but I did not see that very often in other competitions. Let us hope scores will be different in the end.
", "rawMarkdown": "Hallo, you are right, but I did not see that very often in other competitions. Let us hope scores will be different in the end.", "votes": null }, { "id": "2079170", "postDate": "12/29/2022 04:52:05", "content": "From the Data page -
\n\"This is a code competition. When you submit, your model will be rerun on a set of 61 unseen plays located in a holdout test set. The publicly provided test videos are simply a set of mock plays (copied from the training set) which are not used in scoring.\"
\nSo what is on the public LB really does not matter, except to ensure your submissions are working. At some point if people overfit to the plays copied from the training set or figure out which ones these are, then they may have a great public LB score but not necessarily a great private LB score. This is common in code competitions and why it is best to trust your CV and focus on the private LB. If you can, try to find some unseen plays, e.g. from previous competitions, see how your model performs on those.
\nGood Luck!
", "rawMarkdown": "From the Data page -\n\n\"This is a code competition. When you submit, your model will be rerun on a set of 61 unseen plays located in a holdout test set. The publicly provided test videos are simply a set of mock plays (copied from the training set) which are not used in scoring.\"\n\nSo what is on the public LB really does not matter, except to ensure your submissions are working. At some point if people overfit to the plays copied from the training set or figure out which ones these are, then they may have a great public LB score but not necessarily a great private LB score. This is common in code competitions and why it is best to trust your CV and focus on the private LB. If you can, try to find some unseen plays, e.g. from previous competitions, see how your model performs on those.\n\nGood Luck!", "votes": null }, { "id": "2084878", "postDate": "01/03/2023 20:15:56", "content": "\n\n3) I want to believe Kaggle ranks at a higher resolution, but does not show.
\n
Yes, we rank based on full precision.
", "rawMarkdown": "> 3) I want to believe Kaggle ranks at a higher resolution, but does not show.\n\nYes, we rank based on full precision.", "votes": null }, { "id": "2084908", "postDate": "01/03/2023 20:42:06", "content": "OK thank you for your answer. That is a good explanation.
", "rawMarkdown": "OK thank you for your answer. That is a good explanation.", "votes": null } ], "comments": [ { "id": 2072210, "author_name": "carloshuertas", "author_url": "", "post_date": "12/21/2022 21:14:42", "content": "Dont worry.
\n1) Many of those repeated scores are because they indeed submitted the same thing, so, makes sense to rank lower.
\n2) Final score is based on private, which might not necessarily match that \"repeated public score\"
\n3) I want to believe Kaggle ranks at a higher resolution, but does not show.
Let us hope you're right. For nb. 1: Are we to understand that they are forks of high score notebooks? This practice is a problem for high level competitions. And for nb. 3 : It should show, I might say, because otherwise we get situations like this.
", "votes": null, "replies": [ { "id": 2072335, "author_name": "carloshuertas", "author_url": "", "post_date": "12/22/2022 01:19:59", "content": "For 1) Yes, most likely true. And for 3) You should not rely on the Leaderboard score, if you are, that is a problem, trust your CV instead.
", "votes": null, "replies": [ { "id": 2072440, "author_name": "catadanna", "author_url": "", "post_date": "12/22/2022 05:36:43", "content": "You are right too. CV is always to be trusted. Nevertheless the leaderboard is still weird. Well, we shall see at the end.
", "votes": null, "replies": [] } ] } ] }, { "id": 2084878, "author_name": "wcukierski", "author_url": "", "post_date": "01/03/2023 20:15:56", "content": "\n\n3) I want to believe Kaggle ranks at a higher resolution, but does not show.
\n
Yes, we rank based on full precision.
", "votes": null, "replies": [ { "id": 2084908, "author_name": "catadanna", "author_url": "", "post_date": "01/03/2023 20:42:06", "content": "OK thank you for your answer. That is a good explanation.
", "votes": null, "replies": [] } ] } ] }, { "id": 2077862, "author_name": "thedevastator", "author_url": "", "post_date": "12/27/2022 23:17:08", "content": "Don't worry about the scores on the leaderboard being the same for multiple entries. This is normal, and Kaggle takes into account the private leaderboard scores when determining ranks. The public scores are just a representation of everyone's progress, and the final scores will be based on the private leaderboard. Kaggle generally uses a higher resolution in its rankings, even if it doesn't always show it on the leaderboard.
\nThe Devastator.
", "votes": null, "replies": [ { "id": 2079041, "author_name": "catadanna", "author_url": "", "post_date": "12/28/2022 23:00:01", "content": "Hallo, you are right, but I did not see that very often in other competitions. Let us hope scores will be different in the end.
", "votes": null, "replies": [] } ] }, { "id": 2079170, "author_name": "something4kag", "author_url": "", "post_date": "12/29/2022 04:52:05", "content": "From the Data page -
\n\"This is a code competition. When you submit, your model will be rerun on a set of 61 unseen plays located in a holdout test set. The publicly provided test videos are simply a set of mock plays (copied from the training set) which are not used in scoring.\"
\nSo what is on the public LB really does not matter, except to ensure your submissions are working. At some point if people overfit to the plays copied from the training set or figure out which ones these are, then they may have a great public LB score but not necessarily a great private LB score. This is common in code competitions and why it is best to trust your CV and focus on the private LB. If you can, try to find some unseen plays, e.g. from previous competitions, see how your model performs on those.
\nGood Luck!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2072128": "It seems to me there are not enough digits in the Leaderboard scores. There are a lot of scores which are the same. That is really odd as the first competitor who makes a submission with a score will have a better place than the ones who submitted after him/her.", "2072210": "Dont worry.\n1) Many of those repeated scores are because they indeed submitted the same thing, so, makes sense to rank lower.\n2) Final score is based on private, which might not necessarily match that \"repeated public score\"\n3) I want to believe Kaggle ranks at a higher resolution, but does not show.", "2072310": "Let us hope you're right. For nb. 1: Are we to understand that they are forks of high score notebooks? This practice is a problem for high level competitions. And for nb. 3 : It should show, I might say, because otherwise we get situations like this.", "2072335": "For 1) Yes, most likely true. And for 3) You should not rely on the Leaderboard score, if you are, that is a problem, trust your CV instead.", "2072440": "You are right too. CV is always to be trusted. Nevertheless the leaderboard is still weird. Well, we shall see at the end.", "2077862": "Don't worry about the scores on the leaderboard being the same for multiple entries. This is normal, and Kaggle takes into account the private leaderboard scores when determining ranks. The public scores are just a representation of everyone's progress, and the final scores will be based on the private leaderboard. Kaggle generally uses a higher resolution in its rankings, even if it doesn't always show it on the leaderboard. \n\nThe Devastator.", "2079041": "Hallo, you are right, but I did not see that very often in other competitions. Let us hope scores will be different in the end.", "2079170": "From the Data page -\n\n\"This is a code competition. When you submit, your model will be rerun on a set of 61 unseen plays located in a holdout test set. The publicly provided test videos are simply a set of mock plays (copied from the training set) which are not used in scoring.\"\n\nSo what is on the public LB really does not matter, except to ensure your submissions are working. At some point if people overfit to the plays copied from the training set or figure out which ones these are, then they may have a great public LB score but not necessarily a great private LB score. This is common in code competitions and why it is best to trust your CV and focus on the private LB. If you can, try to find some unseen plays, e.g. from previous competitions, see how your model performs on those.\n\nGood Luck!", "2084878": "> 3) I want to believe Kaggle ranks at a higher resolution, but does not show.\n\nYes, we rank based on full precision.", "2084908": "OK thank you for your answer. That is a good explanation." }, "source": "meta" }