{ "id": 308620, "title": "How does kaggle score on private set", "url": "/competitions/happy-whale-and-dolphin/discussion/308620", "author_name": "", "post_date": "2022-02-19T13:50:13.898862300Z", "votes": 4, "comment_count": 6, "views": 0, "content": "

Can anyone help me understand how kaggle calculates private score given only public test submission. i.e. without prediction code like in code competitions.

", "messages": [ { "id": "1697265", "postDate": "02/19/2022 13:50:13", "content": "

Can anyone help me understand how kaggle calculates private score given only public test submission. i.e. without prediction code like in code competitions.

", "rawMarkdown": "Can anyone help me understand how kaggle calculates private score given only public test submission. i.e. without prediction code like in code competitions.", "votes": null }, { "id": "1697322", "postDate": "02/19/2022 14:35:46", "content": "

The submission contains predictions both for Public/Private LB. So, Public/Private LB scores are calculated at once, but the only the public is exposed to us.

", "rawMarkdown": "The submission contains predictions both for Public/Private LB. So, Public/Private LB scores are calculated at once, but the only the public is exposed to us.", "votes": null }, { "id": "1697470", "postDate": "02/19/2022 16:37:58", "content": "

thanks for your answer. Do you mean that test images contain private+public set? If so, isn't it a leakage.

", "rawMarkdown": "thanks for your answer. Do you mean that test images contain private+public set? If so, isn't it a leakage.", "votes": null }, { "id": "1697487", "postDate": "02/19/2022 16:49:36", "content": "

A leakage happens when somehow the info about the hidden test set is available and can be learned or used.

\n

To give an example of how things work on Kaggle

\n

Let's say:

\n

You make a set of predictions lets say you predict the following:

\n

A, B, C, D, E

\n

The public and private LBs would have the following answers:

\n

B, B , C, X, Z

\n

So you get 50% accuracy on public LB and face a shakeup and get 33.3% accuracy on private LB.

\n

Hope this answers your question :)

", "rawMarkdown": "A leakage happens when somehow the info about the hidden test set is available and can be learned or used. \n\nTo give an example of how things work on Kaggle\n\nLet's say:\n\nYou make a set of predictions lets say you predict the following:\n\n`A, B, C, D, E`\n\nThe public and private LBs would have the following answers:\n\n`B, B` , `C, X, Z`\n\nSo you get 50% accuracy on public LB and face a shakeup and get 33.3% accuracy on private LB. \n\nHope this answers your question :)", "votes": null }, { "id": "1697516", "postDate": "02/19/2022 17:08:37", "content": "

@init27 yes, you cleared my confusion. By leakage i mean that, we already know how images look like for private lb. In contrast, we would not know about test images for private lb in code competitions.

", "rawMarkdown": "init27 yes, you cleared my confusion. By leakage i mean that, we already know how images look like for private lb. In contrast, we would not know about test images for private lb in code competitions.", "votes": null }, { "id": "1697578", "postDate": "02/19/2022 17:39:11", "content": "

@xanjay We know they are whales and dolphins (hopefully, although @andradaolteanu's thread also showcased some garbage examples :D However, we don't have the labels.

\n

A leak would happen if somehow, on the internet or maybe in the metadata you can find the labels. The images are super hard to hand-label too, so I don't think anyone can do that if that's what you're referring to

", "rawMarkdown": "xanjay We know they are whales and dolphins (hopefully, although @andradaolteanu's [thread](https://www.kaggle.com/c/happy-whale-and-dolphin/discussion/308026) also showcased some garbage examples :D However, we don't have the labels. \n\nA leak would happen if somehow, on the internet or maybe in the metadata you can find the labels. The images are super hard to hand-label too, so I don't think anyone can do that if that's what you're referring to", "votes": null }, { "id": "1699337", "postDate": "02/21/2022 05:46:34", "content": "

are you sure or anywhere for official statement?

\n

because I am considering model for cropped dataset, if the private test data not preprocessed , then have to deal with it for inference.

", "rawMarkdown": "are you sure or anywhere for official statement?\n\nbecause I am considering model for cropped dataset, if the private test data not preprocessed , then have to deal with it for inference.", "votes": null } ], "comments": [ { "id": 1697322, "author_name": "meowmeowmeowmeowmeow", "author_url": "", "post_date": "02/19/2022 14:35:46", "content": "

The submission contains predictions both for Public/Private LB. So, Public/Private LB scores are calculated at once, but the only the public is exposed to us.

", "votes": null, "replies": [ { "id": 1697470, "author_name": "xanjay", "author_url": "", "post_date": "02/19/2022 16:37:58", "content": "

thanks for your answer. Do you mean that test images contain private+public set? If so, isn't it a leakage.

", "votes": null, "replies": [] }, { "id": 1697487, "author_name": "init27", "author_url": "", "post_date": "02/19/2022 16:49:36", "content": "

A leakage happens when somehow the info about the hidden test set is available and can be learned or used.

\n

To give an example of how things work on Kaggle

\n

Let's say:

\n

You make a set of predictions lets say you predict the following:

\n

A, B, C, D, E

\n

The public and private LBs would have the following answers:

\n

B, B , C, X, Z

\n

So you get 50% accuracy on public LB and face a shakeup and get 33.3% accuracy on private LB.

\n

Hope this answers your question :)

", "votes": null, "replies": [] }, { "id": 1697516, "author_name": "xanjay", "author_url": "", "post_date": "02/19/2022 17:08:37", "content": "

@init27 yes, you cleared my confusion. By leakage i mean that, we already know how images look like for private lb. In contrast, we would not know about test images for private lb in code competitions.

", "votes": null, "replies": [] }, { "id": 1697578, "author_name": "init27", "author_url": "", "post_date": "02/19/2022 17:39:11", "content": "

@xanjay We know they are whales and dolphins (hopefully, although @andradaolteanu's thread also showcased some garbage examples :D However, we don't have the labels.

\n

A leak would happen if somehow, on the internet or maybe in the metadata you can find the labels. The images are super hard to hand-label too, so I don't think anyone can do that if that's what you're referring to

", "votes": null, "replies": [] }, { "id": 1699337, "author_name": "dragonzhang", "author_url": "", "post_date": "02/21/2022 05:46:34", "content": "

are you sure or anywhere for official statement?

\n

because I am considering model for cropped dataset, if the private test data not preprocessed , then have to deal with it for inference.

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1697265": "Can anyone help me understand how kaggle calculates private score given only public test submission. i.e. without prediction code like in code competitions.", "1697322": "The submission contains predictions both for Public/Private LB. So, Public/Private LB scores are calculated at once, but the only the public is exposed to us.", "1697470": "thanks for your answer. Do you mean that test images contain private+public set? If so, isn't it a leakage.", "1697487": "A leakage happens when somehow the info about the hidden test set is available and can be learned or used. \n\nTo give an example of how things work on Kaggle\n\nLet's say:\n\nYou make a set of predictions lets say you predict the following:\n\n`A, B, C, D, E`\n\nThe public and private LBs would have the following answers:\n\n`B, B` , `C, X, Z`\n\nSo you get 50% accuracy on public LB and face a shakeup and get 33.3% accuracy on private LB. \n\nHope this answers your question :)", "1697516": "init27 yes, you cleared my confusion. By leakage i mean that, we already know how images look like for private lb. In contrast, we would not know about test images for private lb in code competitions.", "1697578": "xanjay We know they are whales and dolphins (hopefully, although @andradaolteanu's [thread](https://www.kaggle.com/c/happy-whale-and-dolphin/discussion/308026) also showcased some garbage examples :D However, we don't have the labels. \n\nA leak would happen if somehow, on the internet or maybe in the metadata you can find the labels. The images are super hard to hand-label too, so I don't think anyone can do that if that's what you're referring to", "1699337": "are you sure or anywhere for official statement?\n\nbecause I am considering model for cropped dataset, if the private test data not preprocessed , then have to deal with it for inference." }, "source": "meta" }