{
  "id": 64341,
  "title": "what about true negatives in evaluation?",
  "url": "/competitions/rsna-pneumonia-detection-challenge/discussion/64341",
  "author_name": "",
  "post_date": "2018-08-28T11:48:34.288507600Z",
  "votes": 4,
  "comment_count": 5,
  "views": 0,
  "content": "<p>Hi,</p>\n\n<p>how is an image scored, that has zero ground truth boxes, and zero predictions on it?\nWill it be omitted for the mean over images, or count in with a score of 1?</p>\n\n<p>Also is there a certain reason why you compute the final score as a mean over images and not the average precision over the entire test set like it is done in well established detection challenges like coco or pascal voc?</p>",
  "messages": [
    {
      "id": "376960",
      "postDate": "08/28/2018 11:48:34",
      "content": "<p>Hi,</p>\n\n<p>how is an image scored, that has zero ground truth boxes, and zero predictions on it?\nWill it be omitted for the mean over images, or count in with a score of 1?</p>\n\n<p>Also is there a certain reason why you compute the final score as a mean over images and not the average precision over the entire test set like it is done in well established detection challenges like coco or pascal voc?</p>",
      "rawMarkdown": "Hi,\n\nhow is an image scored, that has zero ground truth boxes, and zero predictions on it?\nWill it be omitted for the mean over images, or count in with a score of 1?\n\nAlso is there a certain reason why you compute the final score as a mean over images and not the average precision over the entire test set like it is done in well established detection challenges like coco or pascal voc?",
      "votes": null
    },
    {
      "id": "377187",
      "postDate": "08/28/2018 17:46:28",
      "content": "<p>If there are zero ground truth boxes and zero predictions, it is omitted from the mean.</p>\n\n<p>The per-sample average allowed us to explicitly address the large number of true negatives - giving us options for dealing with the above case.  PASCAL VOC and COCO detection metrics do not.</p>",
      "rawMarkdown": "If there are zero ground truth boxes and zero predictions, it is omitted from the mean.\n\nThe per-sample average allowed us to explicitly address the large number of true negatives - giving us options for dealing with the above case.  PASCAL VOC and COCO detection metrics do not.",
      "votes": null
    },
    {
      "id": "378999",
      "postDate": "08/30/2018 18:31:09",
      "content": "<p>Thanks Phil. What happens to false positives in case of zero ground truth boxes? Will that gets ignored too? Or does it add a score of zero?</p>",
      "rawMarkdown": "Thanks Phil. What happens to false positives in case of zero ground truth boxes? Will that gets ignored too? Or does it add a score of zero?",
      "votes": null
    },
    {
      "id": "379027",
      "postDate": "08/30/2018 19:02:29",
      "content": "<p>If there are zero ground truth boxes and greater than zero predictions, a score of zero is added to the mean.</p>",
      "rawMarkdown": "If there are zero ground truth boxes and greater than zero predictions, a score of zero is added to the mean.",
      "votes": null
    },
    {
      "id": "379030",
      "postDate": "08/30/2018 19:09:02",
      "content": "<p>Got it. thank you.</p>",
      "rawMarkdown": "Got it. thank you.",
      "votes": null
    },
    {
      "id": "379035",
      "postDate": "08/30/2018 19:10:02",
      "content": "<p>Happy to help!  Good questions.</p>",
      "rawMarkdown": "Happy to help!  Good questions.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 377187,
      "author_name": "philculliton",
      "author_url": "",
      "post_date": "08/28/2018 17:46:28",
      "content": "<p>If there are zero ground truth boxes and zero predictions, it is omitted from the mean.</p>\n\n<p>The per-sample average allowed us to explicitly address the large number of true negatives - giving us options for dealing with the above case.  PASCAL VOC and COCO detection metrics do not.</p>",
      "votes": null,
      "replies": [
        {
          "id": 378999,
          "author_name": "vingpan",
          "author_url": "",
          "post_date": "08/30/2018 18:31:09",
          "content": "<p>Thanks Phil. What happens to false positives in case of zero ground truth boxes? Will that gets ignored too? Or does it add a score of zero?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 379027,
          "author_name": "philculliton",
          "author_url": "",
          "post_date": "08/30/2018 19:02:29",
          "content": "<p>If there are zero ground truth boxes and greater than zero predictions, a score of zero is added to the mean.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 379030,
          "author_name": "vingpan",
          "author_url": "",
          "post_date": "08/30/2018 19:09:02",
          "content": "<p>Got it. thank you.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 379035,
          "author_name": "philculliton",
          "author_url": "",
          "post_date": "08/30/2018 19:10:02",
          "content": "<p>Happy to help!  Good questions.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "376960": "Hi,\n\nhow is an image scored, that has zero ground truth boxes, and zero predictions on it?\nWill it be omitted for the mean over images, or count in with a score of 1?\n\nAlso is there a certain reason why you compute the final score as a mean over images and not the average precision over the entire test set like it is done in well established detection challenges like coco or pascal voc?",
    "377187": "If there are zero ground truth boxes and zero predictions, it is omitted from the mean.\n\nThe per-sample average allowed us to explicitly address the large number of true negatives - giving us options for dealing with the above case.  PASCAL VOC and COCO detection metrics do not.",
    "378999": "Thanks Phil. What happens to false positives in case of zero ground truth boxes? Will that gets ignored too? Or does it add a score of zero?",
    "379027": "If there are zero ground truth boxes and greater than zero predictions, a score of zero is added to the mean.",
    "379030": "Got it. thank you.",
    "379035": "Happy to help!  Good questions."
  },
  "source": "meta"
}