{
  "id": 19187,
  "title": "Errors in 'ground truth' a.k.a training data?",
  "url": "/competitions/second-annual-data-science-bowl/discussion/19187",
  "author_name": "",
  "post_date": "2016-02-25T20:53:18.023Z",
  "votes": null,
  "comment_count": 3,
  "views": 509,
  "content": "<p>While I'm not a cardiologist by any means, it seems like some of the data in the training file seems incorrect. For MRIs that are visually very distinct and of clearly different sizes, the Systole and Diastole settings are very similar in the training set (but my algorithm gave the larger predictions for the larger sized MRIs). </p>\n\n<p>For instance, patients 26 and 20 have very similar values for sys/dia - (76/158 and 72/162). But when you visually inspect the MRIs, and take a look at heart, you can clearly tell that 26 is at least twice as big as 20! </p>\n\n<p>Could it be that some of the images are re-sized from the original?? </p>\n\n<p>I found similar anomalies in the training dataset where MRIs of similar sizes show widely varying values in the training file! </p>",
  "messages": [
    {
      "id": "109410",
      "postDate": "02/25/2016 20:53:18",
      "content": "<p>While I'm not a cardiologist by any means, it seems like some of the data in the training file seems incorrect. For MRIs that are visually very distinct and of clearly different sizes, the Systole and Diastole settings are very similar in the training set (but my algorithm gave the larger predictions for the larger sized MRIs). </p>\n\n<p>For instance, patients 26 and 20 have very similar values for sys/dia - (76/158 and 72/162). But when you visually inspect the MRIs, and take a look at heart, you can clearly tell that 26 is at least twice as big as 20! </p>\n\n<p>Could it be that some of the images are re-sized from the original?? </p>\n\n<p>I found similar anomalies in the training dataset where MRIs of similar sizes show widely varying values in the training file! </p>",
      "rawMarkdown": "While I'm not a cardiologist by any means, it seems like some of the data in the training file seems incorrect. For MRIs that are visually very distinct and of clearly different sizes, the Systole and Diastole settings are very similar in the training set (but my algorithm gave the larger predictions for the larger sized MRIs). \r\n\r\nFor instance, patients 26 and 20 have very similar values for sys/dia - (76/158 and 72/162). But when you visually inspect the MRIs, and take a look at heart, you can clearly tell that 26 is at least twice as big as 20! \r\n\r\nCould it be that some of the images are re-sized from the original?? \r\n\r\nI found similar anomalies in the training dataset where MRIs of similar sizes show widely varying values in the training file!",
      "votes": null
    },
    {
      "id": "109411",
      "postDate": "02/25/2016 20:55:50",
      "content": "<p>Slice thickness, Slice Spacing and pixel resolutions (found in image header) of the two image sets could be different.  This is a major shortcoming of regression models that don't account for these metadata information.</p>",
      "rawMarkdown": "Slice thickness, Slice Spacing and pixel resolutions (found in image header) of the two image sets could be different.  This is a major shortcoming of regression models that don't account for these metadata information.",
      "votes": null
    },
    {
      "id": "109419",
      "postDate": "02/25/2016 21:43:38",
      "content": "<p>Are you saying, MRI slices for 2 patients that are of different by a factor of 2 can still have about the same systolic/diastolic volume, because the interpretations of the pixel data is somehow different (such as pixel resolution)?  That doesn't seem to make a lot of sense.  Slice thickness, spacing or the resolution shouldn't be re-sizing the actual size of the image itself. </p>\n\n<p>What you are saying is, in other words, if we were to image the same heart, with 2 different set of settings of slice thickness,spacing and pixel resolutions, the <em>size</em> of the images in  MRI slices would be drastically different for each of those different settings? </p>",
      "rawMarkdown": "Are you saying, MRI slices for 2 patients that are of different by a factor of 2 can still have about the same systolic/diastolic volume, because the interpretations of the pixel data is somehow different (such as pixel resolution)?  That doesn't seem to make a lot of sense.  Slice thickness, spacing or the resolution shouldn't be re-sizing the actual size of the image itself. \r\n\r\nWhat you are saying is, in other words, if we were to image the same heart, with 2 different set of settings of slice thickness,spacing and pixel resolutions, the *size* of the images in  MRI slices would be drastically different for each of those different settings?",
      "votes": null
    },
    {
      "id": "109424",
      "postDate": "02/25/2016 22:19:38",
      "content": "<p>I found the reason for the discrepancy. Looks like the PixelSpacing is twice the value for 26 compared to 20. But I wasn't expecting the sizes of the images to be varying by a factor of 2 - that threw me off. I have to use the PixelSpacing value to correct whatever output I'm getting. Life's good now.  </p>",
      "rawMarkdown": "I found the reason for the discrepancy. Looks like the PixelSpacing is twice the value for 26 compared to 20. But I wasn't expecting the sizes of the images to be varying by a factor of 2 - that threw me off. I have to use the PixelSpacing value to correct whatever output I'm getting. Life's good now.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 109411,
      "author_name": "iasiam",
      "author_url": "",
      "post_date": "02/25/2016 20:55:50",
      "content": "<p>Slice thickness, Slice Spacing and pixel resolutions (found in image header) of the two image sets could be different.  This is a major shortcoming of regression models that don't account for these metadata information.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 109419,
      "author_name": "arfoss",
      "author_url": "",
      "post_date": "02/25/2016 21:43:38",
      "content": "<p>Are you saying, MRI slices for 2 patients that are of different by a factor of 2 can still have about the same systolic/diastolic volume, because the interpretations of the pixel data is somehow different (such as pixel resolution)?  That doesn't seem to make a lot of sense.  Slice thickness, spacing or the resolution shouldn't be re-sizing the actual size of the image itself. </p>\n\n<p>What you are saying is, in other words, if we were to image the same heart, with 2 different set of settings of slice thickness,spacing and pixel resolutions, the <em>size</em> of the images in  MRI slices would be drastically different for each of those different settings? </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 109424,
      "author_name": "arfoss",
      "author_url": "",
      "post_date": "02/25/2016 22:19:38",
      "content": "<p>I found the reason for the discrepancy. Looks like the PixelSpacing is twice the value for 26 compared to 20. But I wasn't expecting the sizes of the images to be varying by a factor of 2 - that threw me off. I have to use the PixelSpacing value to correct whatever output I'm getting. Life's good now.  </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "109410": "While I'm not a cardiologist by any means, it seems like some of the data in the training file seems incorrect. For MRIs that are visually very distinct and of clearly different sizes, the Systole and Diastole settings are very similar in the training set (but my algorithm gave the larger predictions for the larger sized MRIs). \r\n\r\nFor instance, patients 26 and 20 have very similar values for sys/dia - (76/158 and 72/162). But when you visually inspect the MRIs, and take a look at heart, you can clearly tell that 26 is at least twice as big as 20! \r\n\r\nCould it be that some of the images are re-sized from the original?? \r\n\r\nI found similar anomalies in the training dataset where MRIs of similar sizes show widely varying values in the training file!",
    "109411": "Slice thickness, Slice Spacing and pixel resolutions (found in image header) of the two image sets could be different.  This is a major shortcoming of regression models that don't account for these metadata information.",
    "109419": "Are you saying, MRI slices for 2 patients that are of different by a factor of 2 can still have about the same systolic/diastolic volume, because the interpretations of the pixel data is somehow different (such as pixel resolution)?  That doesn't seem to make a lot of sense.  Slice thickness, spacing or the resolution shouldn't be re-sizing the actual size of the image itself. \r\n\r\nWhat you are saying is, in other words, if we were to image the same heart, with 2 different set of settings of slice thickness,spacing and pixel resolutions, the *size* of the images in  MRI slices would be drastically different for each of those different settings?",
    "109424": "I found the reason for the discrepancy. Looks like the PixelSpacing is twice the value for 26 compared to 20. But I wasn't expecting the sizes of the images to be varying by a factor of 2 - that threw me off. I have to use the PixelSpacing value to correct whatever output I'm getting. Life's good now."
  },
  "source": "meta"
}