\n
![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8531166%2F72c6173ab353c2fd1313061b289f52c2%2FScreenshot%20from%202024-05-15%2020-08-12.png?generation=1715818141305738&alt=media\")
For example, the LB score of a model at epoch 20 is 0.72, and for a slightly modified model, I get 0.73. However, at epoch 22, the original model still scores 0.72, while the modified model scores 0.71. I have no idea what's working and what's not๐ต I feel like I'm rolling dice every day๐
", "messages": [ { "id": 2818341, "postDate": "2024-05-17T13:00:03.093Z", "content": "To prevent LB probing.
\n\n\nIn most competitions, the scores are displayed with all the decimal places after the end
\n
This is the case after the competition end, and it probably will be the case here as well. But in most code competitions LB scores are truncated during the competition.
\nI agree with you that it makes it harder to estimate if a model change results in a significant LB score change or not.
", "rawMarkdown": "To prevent LB probing.\n\n> In most competitions, the scores are displayed with all the decimal places after the end\n\nThis is the case after the competition end, and it probably will be the case here as well. But in most code competitions LB scores are truncated during the competition.\n\nI agree with you that it makes it harder to estimate if a model change results in a significant LB score change or not.\n\n", "votes": 3, "replies": [ { "id": 2821016, "postDate": "2024-05-17T19:51:00.753Z", "content": "It's to prevent LB probing. Thank you for letting me know!๐
", "rawMarkdown": "It's to prevent LB probing. Thank you for letting me know!๐", "votes": 1 } ] }, { "id": 2812683, "postDate": "2024-05-14T10:56:15.570Z", "content": "For example, the LB score of a model at epoch 20 is 0.72, and for a slightly modified model, I get 0.73. However, at epoch 22, the original model still scores 0.72, while the modified model scores 0.71. I have no idea what's working and what's not๐ต I feel like I'm rolling dice every day๐
", "rawMarkdown": "For example, the LB score of a model at epoch 20 is 0.72, and for a slightly modified model, I get 0.73. However, at epoch 22, the original model still scores 0.72, while the modified model scores 0.71. I have no idea what's working and what's not๐ต I feel like I'm rolling dice every day๐ ", "votes": 1 }, { "id": 2815415, "postDate": "2024-05-15T21:25:00.507Z", "content": "In this competition, the score is rounded to two decimal points. However, the real score is held in the background were you can't see it. In most competitions, if two teams have the same score the one that came first is above the other, in this competition the one with the higher hidden score (not rounded) will be above the other. In your case the slight modification in the model changed the score just enough for the site to register your LB as .73. Hopefully, this helps you. Have a great day and good luck in this competition (nice job on the second place so far๐)!
", "rawMarkdown": "In this competition, the score is rounded to two decimal points. However, the real score is held in the background were you can't see it. In most competitions, if two teams have the same score the one that came first is above the other, in this competition the one with the higher hidden score (not rounded) will be above the other. In your case the slight modification in the model changed the score just enough for the site to register your LB as .73. Hopefully, this helps you. Have a great day and good luck in this competition (nice job on the second place so far๐)!", "votes": 2, "replies": [ { "id": 2815526, "postDate": "2024-05-15T23:36:37.397Z", "content": "Thank you for your kind comment. In most competitions, the scores are displayed with all the decimal places after the end. In this competition, there is a significant discrepancy between the CV and LB scores, so being able to see the scores up to the third decimal place would be very helpful. Is showing only up to the second decimal place just a convention, or is there a specific reason for it?
", "rawMarkdown": "Thank you for your kind comment. In most competitions, the scores are displayed with all the decimal places after the end. In this competition, there is a significant discrepancy between the CV and LB scores, so being able to see the scores up to the third decimal place would be very helpful. Is showing only up to the second decimal place just a convention, or is there a specific reason for it?", "replies": [ { "id": 2815546, "postDate": "2024-05-16T00:01:39.857Z", "content": "In this case, I do not know this competition is very weird in the loss function, data, and everything in between. You can ask a competition host.
", "rawMarkdown": "In this case, I do not know this competition is very weird in the loss function, data, and everything in between. You can ask a competition host." }, { "id": 2815548, "postDate": "2024-05-16T00:06:11.563Z", "content": "I believe one thing you could probably do is to take one terrible pick for the final leaderboard score and temporarily choose it for the final test set, kaggle will automatically choose the next best submission for the final leaderboard score. This can show you if one submission is slightly better than another. But I have not tested this myself and you should take this with a grain of salt. I do believe this will work because it's how the leaderboard works but again I am not 100 percent sure.
", "rawMarkdown": "I believe one thing you could probably do is to take one terrible pick for the final leaderboard score and temporarily choose it for the final test set, kaggle will automatically choose the next best submission for the final leaderboard score. This can show you if one submission is slightly better than another. But I have not tested this myself and you should take this with a grain of salt. I do believe this will work because it's how the leaderboard works but again I am not 100 percent sure." }, { "id": 2815550, "postDate": "2024-05-16T00:09:40.053Z", "content": "In this example the leaderboard chooses the middle submission and this is the one that should be running better!
\n
I chose a bad submission with a score below the score of my three best ones and kaggle auto-picked the best one for me!
", "rawMarkdown": "I chose a bad submission with a score below the score of my three best ones and kaggle auto-picked the best one for me!" }, { "id": 2815601, "postDate": "2024-05-16T01:37:11.950Z", "content": "You can check the order of Public LB scores.
\nSubmissions page and change the sorting order to Public Score.
\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8577629%2F84590ac50bc4e159c4d34e08952fdb3b%2Fsort.png?generation=1715822698058635&alt=media\")
--
\nIn a recent competition I participated in, the scores were displayed to two decimal places during the event.
\n(https://www.kaggle.com/competitions/llm-prompt-recovery)
However, even with the same score, the LB rankings changed (e.g., from 0.60 to 0.60, the Public LB ranking increased).
\nThis suggests that scores were managed internally to more than two decimal places.
\nIt may not be the same in this competition, but for your reference.
Great idea I had no clue this was a thing thank you very much!๐
", "rawMarkdown": "Great idea I had no clue this was a thing thank you very much!๐ " }, { "id": 2815612, "postDate": "2024-05-16T02:00:07.820Z", "content": "If scores are the same, the submission made earlier is ranked higher, and your explanation is accurate. However, even if there are 10 submissions with a score of 0.72, it's likely that most of them are clustered around 0.720 to 0.722 or 0.727 to 0.729, so this kind of inference might not be as helpful.
", "rawMarkdown": "If scores are the same, the submission made earlier is ranked higher, and your explanation is accurate. However, even if there are 10 submissions with a score of 0.72, it's likely that most of them are clustered around 0.720 to 0.722 or 0.727 to 0.729, so this kind of inference might not be as helpful.", "votes": 2 }, { "id": 2816703, "postDate": "2024-05-16T13:43:28.840Z", "content": "Yes thats true if they are all very close it will not give much insight to the other submissions, which for you is a problem because most of them are about 0.72๐
\n\n\nmost of them are clustered around 0.720 to 0.722
\n
I don't know how to check the scores but the 2 highest score will be revealed. Good Luck!
", "rawMarkdown": "Yes thats true if they are all very close it will not give much insight to the other submissions, which for you is a problem because most of them are about 0.72๐\n >most of them are clustered around 0.720 to 0.722\n\nI don't know how to check the scores but the 2 highest score will be revealed. Good Luck!" }, { "id": 2816709, "postDate": "2024-05-16T13:45:11.250Z", "content": "I did find this link but it doesnt solve the problem it just says that competitions will show score up to 5 decimal points after competition is over.
", "rawMarkdown": "I did find this [link](https://www.kaggle.com/discussions/product-feedback/63896) but it doesnt solve the problem it just says that competitions will show score up to 5 decimal points after competition is over." }, { "id": 2821007, "postDate": "2024-05-17T19:35:04.463Z", "content": "Wanted to congratulate the first place!
", "rawMarkdown": "Wanted to congratulate the first place!" } ] } ] }, { "id": 2820804, "postDate": "2024-05-17T18:00:48.770Z", "content": "\n\nfor a slightly modified model, I get 0.73.
\n
May I ask what was the modification? Asking for a friendโฆ
", "rawMarkdown": "> for a slightly modified model, I get 0.73. \n\nMay I ask what was the modification? Asking for a friend...", "replies": [ { "id": 2820834, "postDate": "2024-05-17T18:15:50.873Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2821014, "postDate": "2024-05-17T19:49:09.800Z", "content": "Modifications like mixup, upsampling, and changes to the loss function are things that everyone has probably tried. The score I mentioned was just an example, and the score of the single model is low. I wanted to point out that the model is unstable, and I don't think any particular technique made the instability significantly worse. Although my contribution to our team's score is quite low, our team doesn't have any extraordinary methods.
", "rawMarkdown": "Modifications like mixup, upsampling, and changes to the loss function are things that everyone has probably tried. The score I mentioned was just an example, and the score of the single model is low. I wanted to point out that the model is unstable, and I don't think any particular technique made the instability significantly worse. Although my contribution to our team's score is quite low, our team doesn't have any extraordinary methods.", "votes": 2 } ] } ], "comments": [ { "id": 2818341, "author_name": "CPMP", "author_url": "", "post_date": "2024-05-17T13:00:03.093000", "content": "To prevent LB probing.
\n\n\nIn most competitions, the scores are displayed with all the decimal places after the end
\n
This is the case after the competition end, and it probably will be the case here as well. But in most code competitions LB scores are truncated during the competition.
\nI agree with you that it makes it harder to estimate if a model change results in a significant LB score change or not.
", "votes": 3, "replies": [ { "id": 2821016, "author_name": "HB", "author_url": "", "post_date": "2024-05-17T19:51:00.753000", "content": "It's to prevent LB probing. Thank you for letting me know!๐
", "votes": 1, "replies": [] } ] }, { "id": 2815415, "author_name": "MaxG6", "author_url": "", "post_date": "2024-05-15T21:25:00.507000", "content": "In this competition, the score is rounded to two decimal points. However, the real score is held in the background were you can't see it. In most competitions, if two teams have the same score the one that came first is above the other, in this competition the one with the higher hidden score (not rounded) will be above the other. In your case the slight modification in the model changed the score just enough for the site to register your LB as .73. Hopefully, this helps you. Have a great day and good luck in this competition (nice job on the second place so far๐)!
", "votes": 2, "replies": [ { "id": 2815526, "author_name": "HB", "author_url": "", "post_date": "2024-05-15T23:36:37.397000", "content": "Thank you for your kind comment. In most competitions, the scores are displayed with all the decimal places after the end. In this competition, there is a significant discrepancy between the CV and LB scores, so being able to see the scores up to the third decimal place would be very helpful. Is showing only up to the second decimal place just a convention, or is there a specific reason for it?
", "votes": 0, "replies": [ { "id": 2815546, "author_name": "MaxG6", "author_url": "", "post_date": "2024-05-16T00:01:39.857000", "content": "In this case, I do not know this competition is very weird in the loss function, data, and everything in between. You can ask a competition host.
", "votes": 0, "replies": [] }, { "id": 2815548, "author_name": "MaxG6", "author_url": "", "post_date": "2024-05-16T00:06:11.563000", "content": "I believe one thing you could probably do is to take one terrible pick for the final leaderboard score and temporarily choose it for the final test set, kaggle will automatically choose the next best submission for the final leaderboard score. This can show you if one submission is slightly better than another. But I have not tested this myself and you should take this with a grain of salt. I do believe this will work because it's how the leaderboard works but again I am not 100 percent sure.
", "votes": 0, "replies": [] }, { "id": 2815550, "author_name": "MaxG6", "author_url": "", "post_date": "2024-05-16T00:09:40.053000", "content": "In this example the leaderboard chooses the middle submission and this is the one that should be running better!
\n
I chose a bad submission with a score below the score of my three best ones and kaggle auto-picked the best one for me!
", "votes": 0, "replies": [] }, { "id": 2815601, "author_name": "neilus", "author_url": "", "post_date": "2024-05-16T01:37:11.950000", "content": "You can check the order of Public LB scores.
\nSubmissions page and change the sorting order to Public Score.
\n
--
\nIn a recent competition I participated in, the scores were displayed to two decimal places during the event.
\n(https://www.kaggle.com/competitions/llm-prompt-recovery)
However, even with the same score, the LB rankings changed (e.g., from 0.60 to 0.60, the Public LB ranking increased).
\nThis suggests that scores were managed internally to more than two decimal places.
\nIt may not be the same in this competition, but for your reference.
Great idea I had no clue this was a thing thank you very much!๐
", "votes": 0, "replies": [] }, { "id": 2815612, "author_name": "HB", "author_url": "", "post_date": "2024-05-16T02:00:07.820000", "content": "If scores are the same, the submission made earlier is ranked higher, and your explanation is accurate. However, even if there are 10 submissions with a score of 0.72, it's likely that most of them are clustered around 0.720 to 0.722 or 0.727 to 0.729, so this kind of inference might not be as helpful.
", "votes": 2, "replies": [] }, { "id": 2816703, "author_name": "MaxG6", "author_url": "", "post_date": "2024-05-16T13:43:28.840000", "content": "Yes thats true if they are all very close it will not give much insight to the other submissions, which for you is a problem because most of them are about 0.72๐
\n\n\nmost of them are clustered around 0.720 to 0.722
\n
I don't know how to check the scores but the 2 highest score will be revealed. Good Luck!
", "votes": 0, "replies": [] }, { "id": 2816709, "author_name": "MaxG6", "author_url": "", "post_date": "2024-05-16T13:45:11.250000", "content": "I did find this link but it doesnt solve the problem it just says that competitions will show score up to 5 decimal points after competition is over.
", "votes": 0, "replies": [] }, { "id": 2821007, "author_name": "MaxG6", "author_url": "", "post_date": "2024-05-17T19:35:04.463000", "content": "Wanted to congratulate the first place!
", "votes": 0, "replies": [] } ] } ] }, { "id": 2820804, "author_name": "CPMP", "author_url": "", "post_date": "2024-05-17T18:00:48.770000", "content": "\n\nfor a slightly modified model, I get 0.73.
\n
May I ask what was the modification? Asking for a friendโฆ
", "votes": 0, "replies": [ { "id": 2820834, "author_name": "", "author_url": "", "post_date": "2024-05-17T18:15:50.873000", "content": "", "votes": 0, "replies": [] }, { "id": 2821014, "author_name": "HB", "author_url": "", "post_date": "2024-05-17T19:49:09.800000", "content": "Modifications like mixup, upsampling, and changes to the loss function are things that everyone has probably tried. The score I mentioned was just an example, and the score of the single model is low. I wanted to point out that the model is unstable, and I don't think any particular technique made the instability significantly worse. Although my contribution to our team's score is quite low, our team doesn't have any extraordinary methods.
", "votes": 2, "replies": [] } ] } ], "raw_markdown_by_id": { "2818341": "To prevent LB probing.\n\n> In most competitions, the scores are displayed with all the decimal places after the end\n\nThis is the case after the competition end, and it probably will be the case here as well. But in most code competitions LB scores are truncated during the competition.\n\nI agree with you that it makes it harder to estimate if a model change results in a significant LB score change or not.\n\n", "2812683": "For example, the LB score of a model at epoch 20 is 0.72, and for a slightly modified model, I get 0.73. However, at epoch 22, the original model still scores 0.72, while the modified model scores 0.71. I have no idea what's working and what's not๐ต I feel like I'm rolling dice every day๐ ", "2815415": "In this competition, the score is rounded to two decimal points. However, the real score is held in the background were you can't see it. In most competitions, if two teams have the same score the one that came first is above the other, in this competition the one with the higher hidden score (not rounded) will be above the other. In your case the slight modification in the model changed the score just enough for the site to register your LB as .73. Hopefully, this helps you. Have a great day and good luck in this competition (nice job on the second place so far๐)!", "2820804": "> for a slightly modified model, I get 0.73. \n\nMay I ask what was the modification? Asking for a friend..." } }