{
  "id": 189229,
  "title": "Image features good for CV but disastrous for private... what should I do?",
  "url": "/competitions/osic-pulmonary-fibrosis-progression/discussion/189229",
  "author_name": "",
  "post_date": "2020-10-07T03:02:02.137846700Z",
  "votes": 4,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Our CV without image features was around -6.98 but private was around -6.85 ~ -6.88.<br>\nThen we added image features like lung volume, lung skew, kurtosis… CV went to around -6.9, but public score went to around -6.90 ~ -6.93.</p>\n<p>Why did we suffer from the giant shakedown? Should we do some analysis to discard or preprocess image features? Or just we are so unlucky?</p>",
  "messages": [
    {
      "id": "1040207",
      "postDate": "10/07/2020 03:02:02",
      "content": "<p>Our CV without image features was around -6.98 but private was around -6.85 ~ -6.88.<br>\nThen we added image features like lung volume, lung skew, kurtosis… CV went to around -6.9, but public score went to around -6.90 ~ -6.93.</p>\n<p>Why did we suffer from the giant shakedown? Should we do some analysis to discard or preprocess image features? Or just we are so unlucky?</p>",
      "rawMarkdown": "Our CV without image features was around -6.98 but private was around -6.85 ~ -6.88.\nThen we added image features like lung volume, lung skew, kurtosis... CV went to around -6.9, but public score went to around -6.90 ~ -6.93.\n\nWhy did we suffer from the giant shakedown? Should we do some analysis to discard or preprocess image features? Or just we are so unlucky?",
      "votes": null
    },
    {
      "id": "1040692",
      "postDate": "10/07/2020 09:44:22",
      "content": "<p>My guess was that those image features (e.g. lung volume, lung skew, kurtosis) were actually very hard to compute accurately due to the quality of the data. The quality varied a lot across patients, including ones in the private. So there might be no guarantees that those features can be computed accurately in the private test sets. </p>\n<p>You might want to plot histogram comparing your image features between train and (private) test. Do they overlap well?</p>\n<p>I was wondering about it and eventually let my 2DCNN extract whatever features from the image relevant to the decline in FVC. I expected that this was more robust than manually computing presumably controversial image statistics. </p>",
      "rawMarkdown": "My guess was that those image features (e.g. lung volume, lung skew, kurtosis) were actually very hard to compute accurately due to the quality of the data. The quality varied a lot across patients, including ones in the private. So there might be no guarantees that those features can be computed accurately in the private test sets. \n\nYou might want to plot histogram comparing your image features between train and (private) test. Do they overlap well?\n\nI was wondering about it and eventually let my 2DCNN extract whatever features from the image relevant to the decline in FVC. I expected that this was more robust than manually computing presumably controversial image statistics.",
      "votes": null
    },
    {
      "id": "1040732",
      "postDate": "10/07/2020 10:11:25",
      "content": "<p>Because of data quality? I never noticed… so the solution should be robust to varying data quality, I learned.<br>\nAnd I should see the distribution of image features… I trusted image feats because of high correlation with FVC (0.5~0.85).</p>",
      "rawMarkdown": "Because of data quality? I never noticed... so the solution should be robust to varying data quality, I learned.\nAnd I should see the distribution of image features... I trusted image feats because of high correlation with FVC (0.5~0.85).",
      "votes": null
    },
    {
      "id": "1041221",
      "postDate": "10/07/2020 16:14:47",
      "content": "<p>Did your CV partition patients, or just available observations?</p>",
      "rawMarkdown": "Did your CV partition patients, or just available observations?",
      "votes": null
    },
    {
      "id": "1042611",
      "postDate": "10/08/2020 11:04:33",
      "content": "<p>CV is partition of patients.</p>",
      "rawMarkdown": "CV is partition of patients.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1040692,
      "author_name": "code1110",
      "author_url": "",
      "post_date": "10/07/2020 09:44:22",
      "content": "<p>My guess was that those image features (e.g. lung volume, lung skew, kurtosis) were actually very hard to compute accurately due to the quality of the data. The quality varied a lot across patients, including ones in the private. So there might be no guarantees that those features can be computed accurately in the private test sets. </p>\n<p>You might want to plot histogram comparing your image features between train and (private) test. Do they overlap well?</p>\n<p>I was wondering about it and eventually let my 2DCNN extract whatever features from the image relevant to the decline in FVC. I expected that this was more robust than manually computing presumably controversial image statistics. </p>",
      "votes": null,
      "replies": [
        {
          "id": 1040732,
          "author_name": "resistance0108",
          "author_url": "",
          "post_date": "10/07/2020 10:11:25",
          "content": "<p>Because of data quality? I never noticed… so the solution should be robust to varying data quality, I learned.<br>\nAnd I should see the distribution of image features… I trusted image feats because of high correlation with FVC (0.5~0.85).</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1041221,
      "author_name": "archimedus",
      "author_url": "",
      "post_date": "10/07/2020 16:14:47",
      "content": "<p>Did your CV partition patients, or just available observations?</p>",
      "votes": null,
      "replies": [
        {
          "id": 1042611,
          "author_name": "resistance0108",
          "author_url": "",
          "post_date": "10/08/2020 11:04:33",
          "content": "<p>CV is partition of patients.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1040207": "Our CV without image features was around -6.98 but private was around -6.85 ~ -6.88.\nThen we added image features like lung volume, lung skew, kurtosis... CV went to around -6.9, but public score went to around -6.90 ~ -6.93.\n\nWhy did we suffer from the giant shakedown? Should we do some analysis to discard or preprocess image features? Or just we are so unlucky?",
    "1040692": "My guess was that those image features (e.g. lung volume, lung skew, kurtosis) were actually very hard to compute accurately due to the quality of the data. The quality varied a lot across patients, including ones in the private. So there might be no guarantees that those features can be computed accurately in the private test sets. \n\nYou might want to plot histogram comparing your image features between train and (private) test. Do they overlap well?\n\nI was wondering about it and eventually let my 2DCNN extract whatever features from the image relevant to the decline in FVC. I expected that this was more robust than manually computing presumably controversial image statistics.",
    "1040732": "Because of data quality? I never noticed... so the solution should be robust to varying data quality, I learned.\nAnd I should see the distribution of image features... I trusted image feats because of high correlation with FVC (0.5~0.85).",
    "1041221": "Did your CV partition patients, or just available observations?",
    "1042611": "CV is partition of patients."
  },
  "source": "meta"
}