{ "id": 411778, "title": "How is the public/private leaderboard splitted?", "url": "/competitions/image-matching-challenge-2023/discussion/411778", "author_name": "", "post_date": "2023-05-21T03:53:34.954344700Z", "votes": 5, "comment_count": 6, "views": 0, "content": "

Hi @oldufo @eduardtrulls , thanks for the challenging and interesting competition. I wonder:

\n
    \n

  1. Is it possible that public and private leaderboard have overlapped datasets? For example, if a dataset contains scene1 and scene2, is it possible that scene1 is in public leaderboard and scene2 is in private leaderboard? Or public and private leaderboard have completely different and unoverlapped datasets?

    2. \n

  2. If there are overlapped datasets in public and private leaderboard, how to get the score for both leaderboard?

    3. \n
\n

Best regards

", "messages": [ { "id": "2267532", "postDate": "05/21/2023 03:53:34", "content": "

Hi @oldufo @eduardtrulls , thanks for the challenging and interesting competition. I wonder:

\n
    \n

  1. Is it possible that public and private leaderboard have overlapped datasets? For example, if a dataset contains scene1 and scene2, is it possible that scene1 is in public leaderboard and scene2 is in private leaderboard? Or public and private leaderboard have completely different and unoverlapped datasets?

    2. \n

  2. If there are overlapped datasets in public and private leaderboard, how to get the score for both leaderboard?

    3. \n
\n

Best regards

", "rawMarkdown": "Hi @oldufo @eduardtrulls , thanks for the challenging and interesting competition. I wonder:\n\n1. Is it possible that public and private leaderboard have overlapped datasets? For example, if a dataset contains `scene1` and `scene2`, is it possible that `scene1` is in public leaderboard and `scene2` is in private leaderboard? Or public and private leaderboard have completely different and unoverlapped datasets?\n\n2. If there are overlapped datasets in public and private leaderboard, how to get the score for both leaderboard?\n\nBest regards", "votes": null }, { "id": "2268197", "postDate": "05/21/2023 14:42:54", "content": "

Same as here: https://www.kaggle.com/general/207861

\n

PS: no overlap within leaderboards

", "rawMarkdown": "Same as here: https://www.kaggle.com/general/207861\n\nPS: no overlap within leaderboards", "votes": null }, { "id": "2268887", "postDate": "05/22/2023 05:00:18", "content": "

Hi @fabiobellavia , thank you for your fast reply.
\nTo confirm,

\n
\n

PS: no overlap within leaderboards

\n
\n

this means that if a dataset contains scene1 and scene2, then these two scenes will definitely appear in the public LB and private LB at the same time. And it is impossible that scene1 and scene2 appear in two leaderboards respectively?

", "rawMarkdown": "Hi @fabiobellavia , thank you for your fast reply. \nTo confirm, \n\n>PS: no overlap within leaderboards\n\nthis means that if a dataset contains `scene1` and `scene2`, then these two scenes will **definitely appear in the public LB and private LB at the same time**. And it is impossible that `scene1` and `scene2` appear in two leaderboards respectively?", "votes": null }, { "id": "2268967", "postDate": "05/22/2023 06:20:03", "content": "

If an image pair is used for the public LB then this image pair is not used for the private LB and vice versa. Two different image pairs cannot appear at the same in the public and private LBs otherwise would be shared.

", "rawMarkdown": "If an image pair is used for the public LB then this image pair is not used for the private LB and vice versa. Two different image pairs cannot appear at the same in the public and private LBs otherwise would be shared.", "votes": null }, { "id": "2269279", "postDate": "05/22/2023 10:56:11", "content": "

@fabiobellavia Thanks for the info, it's still not clear to me, it's clear there is no overlap between pairs. But I think the question is, is there an overlap between the datasets or between the scenes?! For wording, we can take the bike and chair scenes as an example: is it possible that half of the bike scene is public and the other half is private? ! Or is it guaranteed that if the bike scene is public then it should be fully public and if the bike scene is private then it should be fully private?! And if 1 is the case, then how do you calculate MAA? If 2 is the case, then what is the point of having a public leaderboard? We can take any online dataset and evaluate our solution, but It will be very difficult to guarantee stability between private and public LBs.

", "rawMarkdown": "fabiobellavia Thanks for the info, it's still not clear to me, it's clear there is no overlap between pairs. But I think the question is, is there an overlap between the datasets or between the scenes?! For wording, we can take the bike and chair scenes as an example: is it possible that half of the bike scene is public and the other half is private? ! Or is it guaranteed that if the bike scene is public then it should be fully public and if the bike scene is private then it should be fully private?! And if 1 is the case, then how do you calculate MAA? If 2 is the case, then what is the point of having a public leaderboard? We can take any online dataset and evaluate our solution, but It will be very difficult to guarantee stability between private and public LBs.", "votes": null }, { "id": "2269297", "postDate": "05/22/2023 11:11:14", "content": "

We will not give further details about the public/private splits until the conclusion of the challenge. Given the nature of the problem it is less obvious how to do this than in many other competitions, and there are pretty big trade-offs. We plan to explain our choices at the worshop.

", "rawMarkdown": "We will not give further details about the public/private splits until the conclusion of the challenge. Given the nature of the problem it is less obvious how to do this than in many other competitions, and there are pretty big trade-offs. We plan to explain our choices at the worshop.", "votes": null }, { "id": "2269305", "postDate": "05/22/2023 11:20:59", "content": "

@eduardtrulls Thanks for the quick reply !!

", "rawMarkdown": "eduardtrulls Thanks for the quick reply !!", "votes": null } ], "comments": [ { "id": 2268197, "author_name": "fabiobellavia", "author_url": "", "post_date": "05/21/2023 14:42:54", "content": "

Same as here: https://www.kaggle.com/general/207861

\n

PS: no overlap within leaderboards

", "votes": null, "replies": [ { "id": 2268887, "author_name": "forcewithme", "author_url": "", "post_date": "05/22/2023 05:00:18", "content": "

Hi @fabiobellavia , thank you for your fast reply.
\nTo confirm,

\n
\n

PS: no overlap within leaderboards

\n
\n

this means that if a dataset contains scene1 and scene2, then these two scenes will definitely appear in the public LB and private LB at the same time. And it is impossible that scene1 and scene2 appear in two leaderboards respectively?

", "votes": null, "replies": [ { "id": 2268967, "author_name": "fabiobellavia", "author_url": "", "post_date": "05/22/2023 06:20:03", "content": "

If an image pair is used for the public LB then this image pair is not used for the private LB and vice versa. Two different image pairs cannot appear at the same in the public and private LBs otherwise would be shared.

", "votes": null, "replies": [ { "id": 2269279, "author_name": "ammarali32", "author_url": "", "post_date": "05/22/2023 10:56:11", "content": "

@fabiobellavia Thanks for the info, it's still not clear to me, it's clear there is no overlap between pairs. But I think the question is, is there an overlap between the datasets or between the scenes?! For wording, we can take the bike and chair scenes as an example: is it possible that half of the bike scene is public and the other half is private? ! Or is it guaranteed that if the bike scene is public then it should be fully public and if the bike scene is private then it should be fully private?! And if 1 is the case, then how do you calculate MAA? If 2 is the case, then what is the point of having a public leaderboard? We can take any online dataset and evaluate our solution, but It will be very difficult to guarantee stability between private and public LBs.

", "votes": null, "replies": [ { "id": 2269297, "author_name": "eduardtrulls", "author_url": "", "post_date": "05/22/2023 11:11:14", "content": "

We will not give further details about the public/private splits until the conclusion of the challenge. Given the nature of the problem it is less obvious how to do this than in many other competitions, and there are pretty big trade-offs. We plan to explain our choices at the worshop.

", "votes": null, "replies": [ { "id": 2269305, "author_name": "ammarali32", "author_url": "", "post_date": "05/22/2023 11:20:59", "content": "

@eduardtrulls Thanks for the quick reply !!

", "votes": null, "replies": [] } ] } ] } ] } ] } ] } ], "raw_markdown_by_id": { "2267532": "Hi @oldufo @eduardtrulls , thanks for the challenging and interesting competition. I wonder:\n\n1. Is it possible that public and private leaderboard have overlapped datasets? For example, if a dataset contains `scene1` and `scene2`, is it possible that `scene1` is in public leaderboard and `scene2` is in private leaderboard? Or public and private leaderboard have completely different and unoverlapped datasets?\n\n2. If there are overlapped datasets in public and private leaderboard, how to get the score for both leaderboard?\n\nBest regards", "2268197": "Same as here: https://www.kaggle.com/general/207861\n\nPS: no overlap within leaderboards", "2268887": "Hi @fabiobellavia , thank you for your fast reply. \nTo confirm, \n\n>PS: no overlap within leaderboards\n\nthis means that if a dataset contains `scene1` and `scene2`, then these two scenes will **definitely appear in the public LB and private LB at the same time**. And it is impossible that `scene1` and `scene2` appear in two leaderboards respectively?", "2268967": "If an image pair is used for the public LB then this image pair is not used for the private LB and vice versa. Two different image pairs cannot appear at the same in the public and private LBs otherwise would be shared.", "2269279": "fabiobellavia Thanks for the info, it's still not clear to me, it's clear there is no overlap between pairs. But I think the question is, is there an overlap between the datasets or between the scenes?! For wording, we can take the bike and chair scenes as an example: is it possible that half of the bike scene is public and the other half is private? ! Or is it guaranteed that if the bike scene is public then it should be fully public and if the bike scene is private then it should be fully private?! And if 1 is the case, then how do you calculate MAA? If 2 is the case, then what is the point of having a public leaderboard? We can take any online dataset and evaluate our solution, but It will be very difficult to guarantee stability between private and public LBs.", "2269297": "We will not give further details about the public/private splits until the conclusion of the challenge. Given the nature of the problem it is less obvious how to do this than in many other competitions, and there are pretty big trade-offs. We plan to explain our choices at the worshop.", "2269305": "eduardtrulls Thanks for the quick reply !!" }, "source": "meta" }