Public Score ragnar's notebook (test_lgbm_baseline_5fold_seed_blend.csv) is above of thedevastator's notebook (the fine art of hyperparameter tuning.csv) \n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7011497%2F507136b88040c632c533604cac03dfcd%2Fdoubt_in_best_score.png?generation=1658145240285002&alt=media\")
Hi, I have noticed a very strange behavior of the Leaderboard and the best score representation. I have done a small experiment for this.
\nI have made two submissions:
\n1) Output of this notebook of @ragnar123 pblb score: 0.799
\n2) Output of this notebook of @thedevastator pblb score: 0.799
In My Submission tab when I sort it by Public Score ragnar's notebook (test_lgbm_baseline_5fold_seed_blend.csv) is above of thedevastator's notebook (the fine art of hyperparameter tuning.csv)
\n
But when we sort public notebooks by Best Score order is just opposite.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7011497%2Fca5a6a46a2ae34265791c5b063e3130c%2Fdoubt_in_best_score2%20-%20Copy.png?generation=1658146878026573&alt=media\")
Why is this happening?
", "messages": [ { "id": "1860618", "postDate": "07/18/2022 12:27:37", "content": "Hi, I have noticed a very strange behavior of the Leaderboard and the best score representation. I have done a small experiment for this.
\nI have made two submissions:
\n1) Output of this notebook of @ragnar123 pblb score: 0.799
\n2) Output of this notebook of @thedevastator pblb score: 0.799
In My Submission tab when I sort it by Public Score ragnar's notebook (test_lgbm_baseline_5fold_seed_blend.csv) is above of thedevastator's notebook (the fine art of hyperparameter tuning.csv)
\n
But when we sort public notebooks by Best Score order is just opposite.
\n
Why is this happening?
", "rawMarkdown": "Hi, I have noticed a very strange behavior of the Leaderboard and the best score representation. I have done a small experiment for this. \nI have made two submissions:\n1) Output of [this ](https://www.kaggle.com/code/ragnar123/amex-lgbm-dart-cv-0-7977)notebook of @ragnar123 pblb score: 0.799\n2) Output of [this ](https://www.kaggle.com/code/thedevastator/the-fine-art-of-hyperparameter-tuning)notebook of @thedevastator pblb score: 0.799\n\nIn My Submission tab when I sort it by `Public Score` ragnar's notebook (`test_lgbm_baseline_5fold_seed_blend.csv`) is above of thedevastator's notebook (`the fine art of hyperparameter tuning.csv`) \n\n\nBut when we sort public notebooks by `Best Score` order is just opposite.\n\n\nWhy is this happening?", "votes": null }, { "id": "1860676", "postDate": "07/18/2022 13:05:14", "content": "The score associated with the notebook in the public notebooks tab may stem from another version than the one from which you downloaded the submission.
", "rawMarkdown": "The score associated with the notebook in the public notebooks tab may stem from another version than the one from which you downloaded the submission.", "votes": null }, { "id": "1860680", "postDate": "07/18/2022 13:07:48", "content": "I too thought this but both the notebooks have only 1 version.
", "rawMarkdown": "I too thought this but both the notebooks have only 1 version.", "votes": null }, { "id": "1860714", "postDate": "07/18/2022 13:36:31", "content": "public notebook sorting have a bug
", "rawMarkdown": "public notebook sorting have a bug", "votes": null }, { "id": "1860763", "postDate": "07/18/2022 14:38:33", "content": "thanks @mahluo , this seems reasonable explanation, there were similar issue sometimes back. Many of the blending notebooks were shown higher in public notebook sorting than their actual score in pblb.
", "rawMarkdown": "thanks @mahluo , this seems reasonable explanation, there were similar issue sometimes back. Many of the blending notebooks were shown higher in public notebook sorting than their actual score in pblb.", "votes": null }, { "id": "1860855", "postDate": "07/18/2022 16:05:29", "content": "I think it is not a bug, it randomly sorts the exact predictions.
", "rawMarkdown": "I think it is not a bug, it randomly sorts the exact predictions.", "votes": null }, { "id": "1860900", "postDate": "07/18/2022 16:29:46", "content": "Its not a bug, internally kaggles systems store the full output score and then it gets rounded for LB, you are seeing the correct order
", "rawMarkdown": "Its not a bug, internally kaggles systems store the full output score and then it gets rounded for LB, you are seeing the correct order", "votes": null }, { "id": "1866310", "postDate": "07/22/2022 11:48:10", "content": "As the author of one of those notebook, I must say this is really frustrating.
\nI am running this hyperparameter search and submit the results of it daily (as it gets a better CV over time).
\nI see the exact same thing, on my own submission tab it looks like it is improving (and better than the blending notebooks) but on the notebooks tab it is not the case..
The score associated with the notebook in the public notebooks tab may stem from another version than the one from which you downloaded the submission.
", "votes": null, "replies": [ { "id": 1860680, "author_name": "raj401", "author_url": "", "post_date": "07/18/2022 13:07:48", "content": "I too thought this but both the notebooks have only 1 version.
", "votes": null, "replies": [] } ] }, { "id": 1860714, "author_name": "mahluo", "author_url": "", "post_date": "07/18/2022 13:36:31", "content": "public notebook sorting have a bug
", "votes": null, "replies": [ { "id": 1860763, "author_name": "raj401", "author_url": "", "post_date": "07/18/2022 14:38:33", "content": "thanks @mahluo , this seems reasonable explanation, there were similar issue sometimes back. Many of the blending notebooks were shown higher in public notebook sorting than their actual score in pblb.
", "votes": null, "replies": [] }, { "id": 1860855, "author_name": "mohammadrahmati", "author_url": "", "post_date": "07/18/2022 16:05:29", "content": "I think it is not a bug, it randomly sorts the exact predictions.
", "votes": null, "replies": [] } ] }, { "id": 1860900, "author_name": "munumbutt", "author_url": "", "post_date": "07/18/2022 16:29:46", "content": "Its not a bug, internally kaggles systems store the full output score and then it gets rounded for LB, you are seeing the correct order
", "votes": null, "replies": [] }, { "id": 1866310, "author_name": "thedevastator", "author_url": "", "post_date": "07/22/2022 11:48:10", "content": "As the author of one of those notebook, I must say this is really frustrating.
\nI am running this hyperparameter search and submit the results of it daily (as it gets a better CV over time).
\nI see the exact same thing, on my own submission tab it looks like it is improving (and better than the blending notebooks) but on the notebooks tab it is not the case..