{
  "id": 401514,
  "title": "Question about hidden test set",
  "url": "/competitions/vesuvius-challenge-ink-detection/discussion/401514",
  "author_name": "",
  "post_date": "2023-04-13T16:05:07.011523500Z",
  "votes": 5,
  "comment_count": 6,
  "views": 0,
  "content": "<p>Do we know what is the % of the public test data, we know it is a small subset but could we have an idea? 1%, 5%, 10%, 20%. Maybe this is part of the challenge and we can't know 😅. If it is 1% leaderboard is useless 🙃</p>\n<p>Is there a difference between the hidden test set and the public test, I want to add a logic to do different thing between the submission and the private re-run? For example in some competition you could do it by the length of the submission file but best guess is that we also have 2 scrolls on the hidden test set </p>",
  "messages": [
    {
      "id": "2220713",
      "postDate": "04/13/2023 16:05:07",
      "content": "<p>Do we know what is the % of the public test data, we know it is a small subset but could we have an idea? 1%, 5%, 10%, 20%. Maybe this is part of the challenge and we can't know 😅. If it is 1% leaderboard is useless 🙃</p>\n<p>Is there a difference between the hidden test set and the public test, I want to add a logic to do different thing between the submission and the private re-run? For example in some competition you could do it by the length of the submission file but best guess is that we also have 2 scrolls on the hidden test set </p>",
      "rawMarkdown": "Do we know what is the % of the public test data, we know it is a small subset but could we have an idea? 1%, 5%, 10%, 20%. Maybe this is part of the challenge and we can't know 😅. If it is 1% leaderboard is useless 🙃\n\nIs there a difference between the hidden test set and the public test, I want to add a logic to do different thing between the submission and the private re-run? For example in some competition you could do it by the length of the submission file but best guess is that we also have 2 scrolls on the hidden test set",
      "votes": null
    },
    {
      "id": "2220763",
      "postDate": "04/13/2023 16:55:21",
      "content": "<p>I don't have the exact number handy but I believe it's somewhere around 10%. At least that order of magnitude. So the leaderboard is somewhat useful but definitely not completely representative. Be sure to do k-fold measurements on the public fragments!</p>",
      "rawMarkdown": "I don't have the exact number handy but I believe it's somewhere around 10%. At least that order of magnitude. So the leaderboard is somewhat useful but definitely not completely representative. Be sure to do k-fold measurements on the public fragments!",
      "votes": null
    },
    {
      "id": "2220784",
      "postDate": "04/13/2023 17:20:08",
      "content": "<p>Ahh great, thanks 👍</p>",
      "rawMarkdown": "Ahh great, thanks 👍",
      "votes": null
    },
    {
      "id": "2221489",
      "postDate": "04/14/2023 10:07:47",
      "content": "<p>Can you tell us what is the size of the fragments in hidden test set </p>",
      "rawMarkdown": "Can you tell us what is the size of the fragments in hidden test set",
      "votes": null
    },
    {
      "id": "2221492",
      "postDate": "04/14/2023 10:09:23",
      "content": "<blockquote>\n  <p>I don't have the exact number handy but I believe it's somewhere around 10%.</p>\n</blockquote>\n<p>Does this mean that the number of pixels in the image used to calculate the public score is about 10% of the private one?</p>",
      "rawMarkdown": ">I don't have the exact number handy but I believe it's somewhere around 10%.\n\nDoes this mean that the number of pixels in the image used to calculate the public score is about 10% of the private one?",
      "votes": null
    },
    {
      "id": "2221856",
      "postDate": "04/14/2023 16:27:29",
      "content": "<p>Yeah something like that, maybe a little more actually but that ballpark</p>",
      "rawMarkdown": "Yeah something like that, maybe a little more actually but that ballpark",
      "votes": null
    },
    {
      "id": "2221857",
      "postDate": "04/14/2023 16:27:38",
      "content": "<p>Similar to public fragments</p>",
      "rawMarkdown": "Similar to public fragments",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2220763,
      "author_name": "jpposma",
      "author_url": "",
      "post_date": "04/13/2023 16:55:21",
      "content": "<p>I don't have the exact number handy but I believe it's somewhere around 10%. At least that order of magnitude. So the leaderboard is somewhat useful but definitely not completely representative. Be sure to do k-fold measurements on the public fragments!</p>",
      "votes": null,
      "replies": [
        {
          "id": 2220784,
          "author_name": "ragnar123",
          "author_url": "",
          "post_date": "04/13/2023 17:20:08",
          "content": "<p>Ahh great, thanks 👍</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 2221489,
          "author_name": "myppka",
          "author_url": "",
          "post_date": "04/14/2023 10:07:47",
          "content": "<p>Can you tell us what is the size of the fragments in hidden test set </p>",
          "votes": null,
          "replies": [
            {
              "id": 2221857,
              "author_name": "jpposma",
              "author_url": "",
              "post_date": "04/14/2023 16:27:38",
              "content": "<p>Similar to public fragments</p>",
              "votes": null,
              "replies": []
            }
          ]
        },
        {
          "id": 2221492,
          "author_name": "tattaka",
          "author_url": "",
          "post_date": "04/14/2023 10:09:23",
          "content": "<blockquote>\n  <p>I don't have the exact number handy but I believe it's somewhere around 10%.</p>\n</blockquote>\n<p>Does this mean that the number of pixels in the image used to calculate the public score is about 10% of the private one?</p>",
          "votes": null,
          "replies": [
            {
              "id": 2221856,
              "author_name": "jpposma",
              "author_url": "",
              "post_date": "04/14/2023 16:27:29",
              "content": "<p>Yeah something like that, maybe a little more actually but that ballpark</p>",
              "votes": null,
              "replies": []
            }
          ]
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2220713": "Do we know what is the % of the public test data, we know it is a small subset but could we have an idea? 1%, 5%, 10%, 20%. Maybe this is part of the challenge and we can't know 😅. If it is 1% leaderboard is useless 🙃\n\nIs there a difference between the hidden test set and the public test, I want to add a logic to do different thing between the submission and the private re-run? For example in some competition you could do it by the length of the submission file but best guess is that we also have 2 scrolls on the hidden test set",
    "2220763": "I don't have the exact number handy but I believe it's somewhere around 10%. At least that order of magnitude. So the leaderboard is somewhat useful but definitely not completely representative. Be sure to do k-fold measurements on the public fragments!",
    "2220784": "Ahh great, thanks 👍",
    "2221489": "Can you tell us what is the size of the fragments in hidden test set",
    "2221492": ">I don't have the exact number handy but I believe it's somewhere around 10%.\n\nDoes this mean that the number of pixels in the image used to calculate the public score is about 10% of the private one?",
    "2221856": "Yeah something like that, maybe a little more actually but that ballpark",
    "2221857": "Similar to public fragments"
  },
  "source": "meta"
}