I am reviewing now some images (from the validation set, as it is much more manageable)\nI have encountered some wrong labeling. For example, in image ID 0049a724f5dc20e4 all the musical instruments are labeled cello but they are actually violins\nin 007f71665b0812a7, all the computer monitors are tvs and the person on tv is not boxed. on the other hand in 001997021f01f208 is a drawing of anime characters with tails etc and they are marked as person with lots of details (clothes, face, etc).\nid 0035a4bfeda1b637 is not labeled at all although it is definitely food. same with 004545770f4770c7\nand the list goes on and on (I am just examining 100 photos, so this is a large percentage)\nAnyways, it is what it is but just don't expect to get 100% even if your model detects 100%...
", "messages": [ { "id": 361628, "postDate": "2018-07-24T20:06:18.973Z", "content": "Hi,\nYes, annotation is impossible to get completely correct, especially on a large scale. Some clarifications:
\n\nI am reviewing now some images (from the validation set, as it is much more manageable)\nI have encountered some wrong labeling. For example, in image ID 0049a724f5dc20e4 all the musical instruments are labeled cello but they are actually violins\nin 007f71665b0812a7, all the computer monitors are tvs and the person on tv is not boxed. on the other hand in 001997021f01f208 is a drawing of anime characters with tails etc and they are marked as person with lots of details (clothes, face, etc).\nid 0035a4bfeda1b637 is not labeled at all although it is definitely food. same with 004545770f4770c7\nand the list goes on and on (I am just examining 100 photos, so this is a large percentage)\nAnyways, it is what it is but just don't expect to get 100% even if your model detects 100%...
", "rawMarkdown": "I am reviewing now some images (from the validation set, as it is much more manageable)\nI have encountered some wrong labeling. For example, in image ID 0049a724f5dc20e4 all the musical instruments are labeled cello but they are actually violins\nin 007f71665b0812a7, all the computer monitors are tvs and the person on tv is not boxed. on the other hand in 001997021f01f208 is a drawing of anime characters with tails etc and they are marked as person with lots of details (clothes, face, etc).\nid 0035a4bfeda1b637 is not labeled at all although it is definitely food. same with 004545770f4770c7\nand the list goes on and on (I am just examining 100 photos, so this is a large percentage)\nAnyways, it is what it is but just don't expect to get 100% even if your model detects 100%...\n\n", "votes": 2 }, { "id": 361433, "postDate": "2018-07-24T13:26:13.340Z", "content": "If I am not wrong all the images are not human verified.
", "rawMarkdown": "If I am not wrong all the images are not human verified." }, { "id": 361454, "postDate": "2018-07-24T14:11:45.300Z", "content": "Yup, I've explored some of the labels in the training set and it seems there's a lot of labelling errors as well. In particular, I found a few cases where the labelled object was either occluded or in a group, but the flag didn't indicate so. The flag does work though, because some of them ARE properly flagged.
", "rawMarkdown": "Yup, I've explored some of the labels in the training set and it seems there's a lot of labelling errors as well. In particular, I found a few cases where the labelled object was either occluded or in a group, but the flag didn't indicate so. The flag does work though, because some of them ARE properly flagged.", "isDeleted": true } ], "comments": [ { "id": 361628, "author_name": "Jasper Uijlings", "author_url": "", "post_date": "2018-07-24T20:06:18.973000", "content": "Hi,\nYes, annotation is impossible to get completely correct, especially on a large scale. Some clarifications:
\n\nIf I am not wrong all the images are not human verified.
", "votes": 0, "replies": [] }, { "id": 361454, "author_name": "", "author_url": "", "post_date": "2018-07-24T14:11:45.300000", "content": "Yup, I've explored some of the labels in the training set and it seems there's a lot of labelling errors as well. In particular, I found a few cases where the labelled object was either occluded or in a group, but the flag didn't indicate so. The flag does work though, because some of them ARE properly flagged.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "361628": "Hi,\nYes, annotation is impossible to get completely correct, especially on a large scale. Some clarifications:\n\n - all image-level labels which scored high according to an in-house classifier have been humanly verified. Due to our annotation process (described in more detail here: https://storage.googleapis.com/openimages/web/factsfigures.html), we have very few false positives but we do have missing objects. If we miss an object category in an image, we miss all its instances. Instead, if an object category is present, all its instances are boxed. Missing objects are taken into account into our evaluation protocol.\n - Due to cultural differences some categories such as violins and cellos have a higher confusion. But these errors are rare and affect everyone. We are performing an in-depth error analysis for our upcoming paper.\n - Depictions are valid objects. Depictions are recorded in the attributes of the annotations.\n - Not even humans will ever get 100% accuracy (many recent dataset papers have statistics on this).", "361200": "I am reviewing now some images (from the validation set, as it is much more manageable)\nI have encountered some wrong labeling. For example, in image ID 0049a724f5dc20e4 all the musical instruments are labeled cello but they are actually violins\nin 007f71665b0812a7, all the computer monitors are tvs and the person on tv is not boxed. on the other hand in 001997021f01f208 is a drawing of anime characters with tails etc and they are marked as person with lots of details (clothes, face, etc).\nid 0035a4bfeda1b637 is not labeled at all although it is definitely food. same with 004545770f4770c7\nand the list goes on and on (I am just examining 100 photos, so this is a large percentage)\nAnyways, it is what it is but just don't expect to get 100% even if your model detects 100%...\n\n", "361433": "If I am not wrong all the images are not human verified.", "361454": "Yup, I've explored some of the labels in the training set and it seems there's a lot of labelling errors as well. In particular, I found a few cases where the labelled object was either occluded or in a group, but the flag didn't indicate so. The flag does work though, because some of them ARE properly flagged." } }