{
  "id": 103488,
  "title": "Important information about the challenge data and evaluation",
  "url": "/competitions/open-images-2019-object-detection/discussion/103488",
  "author_name": "",
  "post_date": "2019-08-09T14:30:12.326705700Z",
  "votes": 6,
  "comment_count": 7,
  "views": 0,
  "content": "<p>Based on the results of the previous year's challenge and the questions we received, we would like to stress the following points:</p>\n\n<ul>\n<li><p>The images are annotated with positive image-level labels, indicating certain object classes are present, and with negative image-level labels, indicating certain classes are absent. These negative labels are all reliable and can be used during training, e.g. for hard-negative mining which is important when training object detectors.</p></li>\n<li><p>We have annotated bounding boxes for human body parts only for 95,335 images in the training set, due to the overwhelming number of instances (see also the full description). You can use this list of images to make sure you use the data correctly during training (as there might be a positive image label for a human body part, and yet no boxes). Instead, on the validation and Challenge sets, we annotated human body parts on all images for which we have a positive label.</p></li>\n<li><p>The public leaderboard on the Kaggle website is the most reliable indicator of your performance, as it is based on images and annotations distributed identically to the hidden test set on which participants will be ranked for final evaluation. The validation set is also indicative of performance on the Challenge set, but is not exactly identically distributed. The best reference is the public leaderboard.</p></li>\n</ul>",
  "messages": [
    {
      "id": "595675",
      "postDate": "08/09/2019 14:30:12",
      "content": "<p>Based on the results of the previous year's challenge and the questions we received, we would like to stress the following points:</p>\n\n<ul>\n<li><p>The images are annotated with positive image-level labels, indicating certain object classes are present, and with negative image-level labels, indicating certain classes are absent. These negative labels are all reliable and can be used during training, e.g. for hard-negative mining which is important when training object detectors.</p></li>\n<li><p>We have annotated bounding boxes for human body parts only for 95,335 images in the training set, due to the overwhelming number of instances (see also the full description). You can use this list of images to make sure you use the data correctly during training (as there might be a positive image label for a human body part, and yet no boxes). Instead, on the validation and Challenge sets, we annotated human body parts on all images for which we have a positive label.</p></li>\n<li><p>The public leaderboard on the Kaggle website is the most reliable indicator of your performance, as it is based on images and annotations distributed identically to the hidden test set on which participants will be ranked for final evaluation. The validation set is also indicative of performance on the Challenge set, but is not exactly identically distributed. The best reference is the public leaderboard.</p></li>\n</ul>",
      "rawMarkdown": "Based on the results of the previous year's challenge and the questions we received, we would like to stress the following points:\n\n- The images are annotated with positive image-level labels, indicating certain object classes are present, and with negative image-level labels, indicating certain classes are absent. These negative labels are all reliable and can be used during training, e.g. for hard-negative mining which is important when training object detectors.\n\n- We have annotated bounding boxes for human body parts only for 95,335 images in the training set, due to the overwhelming number of instances (see also the full description). You can use this list of images to make sure you use the data correctly during training (as there might be a positive image label for a human body part, and yet no boxes). Instead, on the validation and Challenge sets, we annotated human body parts on all images for which we have a positive label.\n\n- The public leaderboard on the Kaggle website is the most reliable indicator of your performance, as it is based on images and annotations distributed identically to the hidden test set on which participants will be ranked for final evaluation. The validation set is also indicative of performance on the Challenge set, but is not exactly identically distributed. The best reference is the public leaderboard.",
      "votes": null
    },
    {
      "id": "596103",
      "postDate": "08/10/2019 07:02:37",
      "content": "<p>I didn't join the last year's challenge so I don't know how many images in the hidden test set. Does it have 99,999 images? Thanks!</p>",
      "rawMarkdown": "I didn't join the last year's challenge so I don't know how many images in the hidden test set. Does it have 99,999 images? Thanks!",
      "votes": null
    },
    {
      "id": "602608",
      "postDate": "08/19/2019 09:04:11",
      "content": "<p>&gt; We have annotated bounding boxes for human body parts only for 95,335 images in the training set,</p>\n\n<p><a href=\"/akuznetsa\">@akuznetsa</a> where can i find the list of those files?</p>",
      "rawMarkdown": "&gt; We have annotated bounding boxes for human body parts only for 95,335 images in the training set,\n\n\n@akuznetsa where can i find the list of those files?",
      "votes": null
    },
    {
      "id": "603396",
      "postDate": "08/20/2019 08:05:06",
      "content": "<p>We have not released a separate list for that. Any image that has boxes for human body parts was annotated for those human body parts, and each image that has positive image-level labels for human body parts but no boxes was not annotated</p>",
      "rawMarkdown": "We have not released a separate list for that. Any image that has boxes for human body parts was annotated for those human body parts, and each image that has positive image-level labels for human body parts but no boxes was not annotated",
      "votes": null
    },
    {
      "id": "603444",
      "postDate": "08/20/2019 09:19:58",
      "content": "<p>Would it be possible to release this list? Can be handy. Obviously you know what files are these.</p>\n\n<blockquote>\n  <p>Any image that has boxes for human body parts was annotated for those human body parts</p>\n</blockquote>\n\n<p>Does that means that if a nose was annotated, all noses in the image were annotated but not necessarily ears?\nCan you at least release a list of body part labels that works by this rule? This will allow us to reproduce your list, hopefully</p>",
      "rawMarkdown": "Would it be possible to release this list? Can be handy. Obviously you know what files are these.\n\n&gt;Any image that has boxes for human body parts was annotated for those human body parts\n\nDoes that means that if a nose was annotated, all noses in the image were annotated but not necessarily ears?\nCan you at least release a list of body part labels that works by this rule? This will allow us to reproduce your list, hopefully",
      "votes": null
    },
    {
      "id": "603460",
      "postDate": "08/20/2019 09:48:56",
      "content": "<p>Sorry, actually we already released the list of images here:</p>\n\n<p><a href=\"https://storage.googleapis.com/openimages/2018_04/train/train-image-ids-with-human-parts-and-mammal-boxes.txt\">https://storage.googleapis.com/openimages/2018_04/train/train-image-ids-with-human-parts-and-mammal-boxes.txt</a></p>\n\n<p>The list of classes is here:</p>\n\n<p><a href=\"https://storage.googleapis.com/openimages/2018_04/class-ids-human-body-parts-and-mammal.txt\">https://storage.googleapis.com/openimages/2018_04/class-ids-human-body-parts-and-mammal.txt</a></p>",
      "rawMarkdown": "Sorry, actually we already released the list of images here:\n\nhttps://storage.googleapis.com/openimages/2018_04/train/train-image-ids-with-human-parts-and-mammal-boxes.txt\n\nThe list of classes is here:\n\nhttps://storage.googleapis.com/openimages/2018_04/class-ids-human-body-parts-and-mammal.txt",
      "votes": null
    },
    {
      "id": "607364",
      "postDate": "08/25/2019 05:01:13",
      "content": "<p>Thanks for this <a href=\"/akuznetsa\">@akuznetsa</a>. The ID list does have 95335 entries but if you cross the training gt with the list of images with the list of classes, you only get 84425 images. Is this right or am I missing anything?</p>",
      "rawMarkdown": "Thanks for this @akuznetsa. The ID list does have 95335 entries but if you cross the training gt with the list of images with the list of classes, you only get 84425 images. Is this right or am I missing anything?",
      "votes": null
    },
    {
      "id": "611557",
      "postDate": "08/29/2019 10:18:56",
      "content": "<p>nvm, two labels are not in the 500</p>",
      "rawMarkdown": "nvm, two labels are not in the 500",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 596103,
      "author_name": "vu8691",
      "author_url": "",
      "post_date": "08/10/2019 07:02:37",
      "content": "<p>I didn't join the last year's challenge so I don't know how many images in the hidden test set. Does it have 99,999 images? Thanks!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 602608,
      "author_name": "moshel",
      "author_url": "",
      "post_date": "08/19/2019 09:04:11",
      "content": "<p>&gt; We have annotated bounding boxes for human body parts only for 95,335 images in the training set,</p>\n\n<p><a href=\"/akuznetsa\">@akuznetsa</a> where can i find the list of those files?</p>",
      "votes": null,
      "replies": [
        {
          "id": 603396,
          "author_name": "akuznetsa",
          "author_url": "",
          "post_date": "08/20/2019 08:05:06",
          "content": "<p>We have not released a separate list for that. Any image that has boxes for human body parts was annotated for those human body parts, and each image that has positive image-level labels for human body parts but no boxes was not annotated</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 603444,
          "author_name": "moshel",
          "author_url": "",
          "post_date": "08/20/2019 09:19:58",
          "content": "<p>Would it be possible to release this list? Can be handy. Obviously you know what files are these.</p>\n\n<blockquote>\n  <p>Any image that has boxes for human body parts was annotated for those human body parts</p>\n</blockquote>\n\n<p>Does that means that if a nose was annotated, all noses in the image were annotated but not necessarily ears?\nCan you at least release a list of body part labels that works by this rule? This will allow us to reproduce your list, hopefully</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 603460,
          "author_name": "akuznetsa",
          "author_url": "",
          "post_date": "08/20/2019 09:48:56",
          "content": "<p>Sorry, actually we already released the list of images here:</p>\n\n<p><a href=\"https://storage.googleapis.com/openimages/2018_04/train/train-image-ids-with-human-parts-and-mammal-boxes.txt\">https://storage.googleapis.com/openimages/2018_04/train/train-image-ids-with-human-parts-and-mammal-boxes.txt</a></p>\n\n<p>The list of classes is here:</p>\n\n<p><a href=\"https://storage.googleapis.com/openimages/2018_04/class-ids-human-body-parts-and-mammal.txt\">https://storage.googleapis.com/openimages/2018_04/class-ids-human-body-parts-and-mammal.txt</a></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 607364,
          "author_name": "moshel",
          "author_url": "",
          "post_date": "08/25/2019 05:01:13",
          "content": "<p>Thanks for this <a href=\"/akuznetsa\">@akuznetsa</a>. The ID list does have 95335 entries but if you cross the training gt with the list of images with the list of classes, you only get 84425 images. Is this right or am I missing anything?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 611557,
          "author_name": "moshel",
          "author_url": "",
          "post_date": "08/29/2019 10:18:56",
          "content": "<p>nvm, two labels are not in the 500</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "595675": "Based on the results of the previous year's challenge and the questions we received, we would like to stress the following points:\n\n- The images are annotated with positive image-level labels, indicating certain object classes are present, and with negative image-level labels, indicating certain classes are absent. These negative labels are all reliable and can be used during training, e.g. for hard-negative mining which is important when training object detectors.\n\n- We have annotated bounding boxes for human body parts only for 95,335 images in the training set, due to the overwhelming number of instances (see also the full description). You can use this list of images to make sure you use the data correctly during training (as there might be a positive image label for a human body part, and yet no boxes). Instead, on the validation and Challenge sets, we annotated human body parts on all images for which we have a positive label.\n\n- The public leaderboard on the Kaggle website is the most reliable indicator of your performance, as it is based on images and annotations distributed identically to the hidden test set on which participants will be ranked for final evaluation. The validation set is also indicative of performance on the Challenge set, but is not exactly identically distributed. The best reference is the public leaderboard.",
    "596103": "I didn't join the last year's challenge so I don't know how many images in the hidden test set. Does it have 99,999 images? Thanks!",
    "602608": "&gt; We have annotated bounding boxes for human body parts only for 95,335 images in the training set,\n\n\n@akuznetsa where can i find the list of those files?",
    "603396": "We have not released a separate list for that. Any image that has boxes for human body parts was annotated for those human body parts, and each image that has positive image-level labels for human body parts but no boxes was not annotated",
    "603444": "Would it be possible to release this list? Can be handy. Obviously you know what files are these.\n\n&gt;Any image that has boxes for human body parts was annotated for those human body parts\n\nDoes that means that if a nose was annotated, all noses in the image were annotated but not necessarily ears?\nCan you at least release a list of body part labels that works by this rule? This will allow us to reproduce your list, hopefully",
    "603460": "Sorry, actually we already released the list of images here:\n\nhttps://storage.googleapis.com/openimages/2018_04/train/train-image-ids-with-human-parts-and-mammal-boxes.txt\n\nThe list of classes is here:\n\nhttps://storage.googleapis.com/openimages/2018_04/class-ids-human-body-parts-and-mammal.txt",
    "607364": "Thanks for this @akuznetsa. The ID list does have 95335 entries but if you cross the training gt with the list of images with the list of classes, you only get 84425 images. Is this right or am I missing anything?",
    "611557": "nvm, two labels are not in the 500"
  },
  "source": "meta"
}