To make the LB look better and the competition more interesting, would adding 2 more digits (so 5 digits in total) be a good idea 😊?
", "messages": [ { "id": "1939228", "postDate": "09/14/2022 15:29:52", "content": "To make the LB look better and the competition more interesting, would adding 2 more digits (so 5 digits in total) be a good idea 😊?
", "rawMarkdown": "To make the LB look better and the competition more interesting, would adding 2 more digits (so 5 digits in total) be a good idea 😊?", "votes": null }, { "id": "1947333", "postDate": "09/20/2022 12:18:45", "content": "Hi @khahuras I don't have much experience with changing the leaderboard to add precision, but I'll check with our Kaggle counterparts and investigate.
", "rawMarkdown": "Hi @khahuras I don't have much experience with changing the leaderboard to add precision, but I'll check with our Kaggle counterparts and investigate.", "votes": null }, { "id": "1947513", "postDate": "09/20/2022 14:00:07", "content": "Hi @khahuras, We limit the number of digits shown on the public LB while the competition is active to help prevent probing / overfitting. We'll reveal more digits (and the private LB) after the competition closes.
", "rawMarkdown": "Hi @khahuras, We limit the number of digits shown on the public LB while the competition is active to help prevent probing / overfitting. We'll reveal more digits (and the private LB) after the competition closes.", "votes": null }, { "id": "1949817", "postDate": "09/21/2022 23:15:44", "content": "I appreciate the arguments regarding how much precision should be shown on the public leaderboard. Currently, however, there are 87 entries tied at 0.811, and the maximum score is 0.814. Clearly there are some theoretical limits to the maximum accuracy that can be obtained and evan a competitive entry will be washed out in the field of 87 at 0.811.
\n@ryanholbrook - are the current public scores ranked by greater that 3 decimal places?
", "rawMarkdown": "I appreciate the arguments regarding how much precision should be shown on the public leaderboard. Currently, however, there are 87 entries tied at 0.811, and the maximum score is 0.814. Clearly there are some theoretical limits to the maximum accuracy that can be obtained and evan a competitive entry will be washed out in the field of 87 at 0.811.\n\n@ryanholbrook - are the current public scores ranked by greater that 3 decimal places?", "votes": null }, { "id": "1950685", "postDate": "09/22/2022 13:18:01", "content": "Yes, submissions are ranked by the full precision score. Improving your ranking would be evidence that the public score improved.
", "rawMarkdown": "Yes, submissions are ranked by the full precision score. Improving your ranking would be evidence that the public score improved.", "votes": null }, { "id": "1951239", "postDate": "09/22/2022 23:23:16", "content": "Thank you for the confirmation.
\nGiven the size of the dataset and the scoring methodology, small changes in predictions will certainly lead to very small magnitude correlations. I anticipate a big shake up once the Private Scores are released. It will certainly be interesting to see the final Leaderboard.
", "rawMarkdown": "Thank you for the confirmation. \n\nGiven the size of the dataset and the scoring methodology, small changes in predictions will certainly lead to very small magnitude correlations. I anticipate a big shake up once the Private Scores are released. It will certainly be interesting to see the final Leaderboard.", "votes": null }, { "id": "1952273", "postDate": "09/23/2022 14:37:25", "content": "Just wanted to add that in most competitions training set CV will generally be a more reliable indicator of generalization performance than the public LB alone -- perhaps especially so here since the public / private data splits differ (by donor vs by day).
\nThere's a nice discussion of model selection techniques with both CV / public LB in this thread from the Feedback Prize competition.
", "rawMarkdown": "Just wanted to add that in most competitions training set CV will generally be a more reliable indicator of generalization performance than the public LB alone -- perhaps especially so here since the public / private data splits differ (by donor vs by day).\n\nThere's a nice discussion of model selection techniques with both CV / public LB in [this thread](https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/352800) from the Feedback Prize competition.", "votes": null }, { "id": "1993490", "postDate": "10/18/2022 12:20:47", "content": "@ryanholbrook & @danielburkhardt any update on this?
", "rawMarkdown": "ryanholbrook & @danielburkhardt any update on this?", "votes": null }, { "id": "1993497", "postDate": "10/18/2022 12:27:57", "content": "Hi @callmeb, as noted in Ryan's comment above, we're following the Kaggle data science team recommendation not to increase the number of digits on the LB
\n\n", "rawMarkdown": "Hi @callmeb, as noted in Ryan's comment above, we're following the Kaggle data science team recommendation not to increase the number of digits on the LB\n\n> We limit the number of digits shown on the public LB while the competition is active to help prevent probing / overfitting. We'll reveal more digits (and the private LB) after the competition closes.", "votes": null }, { "id": "1993506", "postDate": "10/18/2022 12:36:20", "content": "We limit the number of digits shown on the public LB while the competition is active to help prevent probing / overfitting. We'll reveal more digits (and the private LB) after the competition closes.
\n
Yes, I was hoping for a change of mind in the last few days. In many compositions this rule is loosened towards the end of the competition :)
", "rawMarkdown": "Yes, I was hoping for a change of mind in the last few days. In many compositions this rule is loosened towards the end of the competition :)", "votes": null }, { "id": "2031680", "postDate": "11/16/2022 08:00:42", "content": "@danielburkhardt @ryanholbrook now the comp has ended. Can we extend both Lb to 5 or 6 digits so people can make some model/submission analysis and learn? Thanks
", "rawMarkdown": "danielburkhardt @ryanholbrook now the comp has ended. Can we extend both Lb to 5 or 6 digits so people can make some model/submission analysis and learn? Thanks", "votes": null }, { "id": "2032201", "postDate": "11/16/2022 13:33:09", "content": "Sure thing!
", "rawMarkdown": "Sure thing!", "votes": null } ], "comments": [ { "id": 1947333, "author_name": "danielburkhardt", "author_url": "", "post_date": "09/20/2022 12:18:45", "content": "Hi @khahuras I don't have much experience with changing the leaderboard to add precision, but I'll check with our Kaggle counterparts and investigate.
", "votes": null, "replies": [] }, { "id": 1947513, "author_name": "ryanholbrook", "author_url": "", "post_date": "09/20/2022 14:00:07", "content": "Hi @khahuras, We limit the number of digits shown on the public LB while the competition is active to help prevent probing / overfitting. We'll reveal more digits (and the private LB) after the competition closes.
", "votes": null, "replies": [] }, { "id": 1949817, "author_name": "kirkdco", "author_url": "", "post_date": "09/21/2022 23:15:44", "content": "I appreciate the arguments regarding how much precision should be shown on the public leaderboard. Currently, however, there are 87 entries tied at 0.811, and the maximum score is 0.814. Clearly there are some theoretical limits to the maximum accuracy that can be obtained and evan a competitive entry will be washed out in the field of 87 at 0.811.
\n@ryanholbrook - are the current public scores ranked by greater that 3 decimal places?
", "votes": null, "replies": [ { "id": 1950685, "author_name": "ryanholbrook", "author_url": "", "post_date": "09/22/2022 13:18:01", "content": "Yes, submissions are ranked by the full precision score. Improving your ranking would be evidence that the public score improved.
", "votes": null, "replies": [] }, { "id": 1951239, "author_name": "kirkdco", "author_url": "", "post_date": "09/22/2022 23:23:16", "content": "Thank you for the confirmation.
\nGiven the size of the dataset and the scoring methodology, small changes in predictions will certainly lead to very small magnitude correlations. I anticipate a big shake up once the Private Scores are released. It will certainly be interesting to see the final Leaderboard.
", "votes": null, "replies": [] } ] }, { "id": 1952273, "author_name": "ryanholbrook", "author_url": "", "post_date": "09/23/2022 14:37:25", "content": "Just wanted to add that in most competitions training set CV will generally be a more reliable indicator of generalization performance than the public LB alone -- perhaps especially so here since the public / private data splits differ (by donor vs by day).
\nThere's a nice discussion of model selection techniques with both CV / public LB in this thread from the Feedback Prize competition.
", "votes": null, "replies": [] }, { "id": 1993490, "author_name": "callmeb", "author_url": "", "post_date": "10/18/2022 12:20:47", "content": "@ryanholbrook & @danielburkhardt any update on this?
", "votes": null, "replies": [ { "id": 1993497, "author_name": "danielburkhardt", "author_url": "", "post_date": "10/18/2022 12:27:57", "content": "Hi @callmeb, as noted in Ryan's comment above, we're following the Kaggle data science team recommendation not to increase the number of digits on the LB
\n\n", "votes": null, "replies": [] }, { "id": 1993506, "author_name": "callmeb", "author_url": "", "post_date": "10/18/2022 12:36:20", "content": "We limit the number of digits shown on the public LB while the competition is active to help prevent probing / overfitting. We'll reveal more digits (and the private LB) after the competition closes.
\n
Yes, I was hoping for a change of mind in the last few days. In many compositions this rule is loosened towards the end of the competition :)
", "votes": null, "replies": [] } ] }, { "id": 2031680, "author_name": "khahuras", "author_url": "", "post_date": "11/16/2022 08:00:42", "content": "@danielburkhardt @ryanholbrook now the comp has ended. Can we extend both Lb to 5 or 6 digits so people can make some model/submission analysis and learn? Thanks
", "votes": null, "replies": [ { "id": 2032201, "author_name": "ryanholbrook", "author_url": "", "post_date": "11/16/2022 13:33:09", "content": "Sure thing!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1939228": "To make the LB look better and the competition more interesting, would adding 2 more digits (so 5 digits in total) be a good idea 😊?", "1947333": "Hi @khahuras I don't have much experience with changing the leaderboard to add precision, but I'll check with our Kaggle counterparts and investigate.", "1947513": "Hi @khahuras, We limit the number of digits shown on the public LB while the competition is active to help prevent probing / overfitting. We'll reveal more digits (and the private LB) after the competition closes.", "1949817": "I appreciate the arguments regarding how much precision should be shown on the public leaderboard. Currently, however, there are 87 entries tied at 0.811, and the maximum score is 0.814. Clearly there are some theoretical limits to the maximum accuracy that can be obtained and evan a competitive entry will be washed out in the field of 87 at 0.811.\n\n@ryanholbrook - are the current public scores ranked by greater that 3 decimal places?", "1950685": "Yes, submissions are ranked by the full precision score. Improving your ranking would be evidence that the public score improved.", "1951239": "Thank you for the confirmation. \n\nGiven the size of the dataset and the scoring methodology, small changes in predictions will certainly lead to very small magnitude correlations. I anticipate a big shake up once the Private Scores are released. It will certainly be interesting to see the final Leaderboard.", "1952273": "Just wanted to add that in most competitions training set CV will generally be a more reliable indicator of generalization performance than the public LB alone -- perhaps especially so here since the public / private data splits differ (by donor vs by day).\n\nThere's a nice discussion of model selection techniques with both CV / public LB in [this thread](https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/352800) from the Feedback Prize competition.", "1993490": "ryanholbrook & @danielburkhardt any update on this?", "1993497": "Hi @callmeb, as noted in Ryan's comment above, we're following the Kaggle data science team recommendation not to increase the number of digits on the LB\n\n> We limit the number of digits shown on the public LB while the competition is active to help prevent probing / overfitting. We'll reveal more digits (and the private LB) after the competition closes.", "1993506": "Yes, I was hoping for a change of mind in the last few days. In many compositions this rule is loosened towards the end of the competition :)", "2031680": "danielburkhardt @ryanholbrook now the comp has ended. Can we extend both Lb to 5 or 6 digits so people can make some model/submission analysis and learn? Thanks", "2032201": "Sure thing!" }, "source": "meta" }