{
  "id": 77260,
  "title": "My one trick for image preprocessing, score + ~0.01",
  "url": "/competitions/human-protein-atlas-image-classification/discussion/77260",
  "author_name": "",
  "post_date": "2019-01-11T01:24:21.222155700Z",
  "votes": 11,
  "comment_count": 2,
  "views": 0,
  "content": "<p>TL; DR: Applying a non-linear image preprocessing step on the images increased my macro F1-score for CV, public and private LB by about 0.01. </p>\n\n<p>I come from an optics/imaging hardware background. When I looked at the train and test image sets, one of the first things that jumped out at me was that the background black area was completely black (0). In any types of imaging systems, the 8-bit pixel values inside the images comes from some sort of A/D conversion unit that has inherent noise, so normally you would see the background areas not entirely consisting of 0s, you would see some zeros, some ones, maybe some twos. In order to get a completely black background, either a bias was added to the A/D conversion, or a digital threshold was applied afterwards. Either way, this means that the zeros were not true zeros, in reality they should have been a negative number. Therefore in the images, the difference between the 8-bit pixel values 0 and 1 is more significant than the difference between 1 and 2.</p>\n\n<p>One way to exploit this is to apply a non-linear function on the images before feeding them to the neural networks. I simply applied a square root function. This boosted my macro F1 scores by about 0.01 across all cv and LBs. There might be more effective or efficient ways to exploit this little piece of information. Maybe you guys would come up with better ideas. </p>",
  "messages": [
    {
      "id": "453952",
      "postDate": "01/11/2019 01:24:21",
      "content": "<p>TL; DR: Applying a non-linear image preprocessing step on the images increased my macro F1-score for CV, public and private LB by about 0.01. </p>\n\n<p>I come from an optics/imaging hardware background. When I looked at the train and test image sets, one of the first things that jumped out at me was that the background black area was completely black (0). In any types of imaging systems, the 8-bit pixel values inside the images comes from some sort of A/D conversion unit that has inherent noise, so normally you would see the background areas not entirely consisting of 0s, you would see some zeros, some ones, maybe some twos. In order to get a completely black background, either a bias was added to the A/D conversion, or a digital threshold was applied afterwards. Either way, this means that the zeros were not true zeros, in reality they should have been a negative number. Therefore in the images, the difference between the 8-bit pixel values 0 and 1 is more significant than the difference between 1 and 2.</p>\n\n<p>One way to exploit this is to apply a non-linear function on the images before feeding them to the neural networks. I simply applied a square root function. This boosted my macro F1 scores by about 0.01 across all cv and LBs. There might be more effective or efficient ways to exploit this little piece of information. Maybe you guys would come up with better ideas. </p>",
      "rawMarkdown": "TL; DR: Applying a non-linear image preprocessing step on the images increased my macro F1-score for CV, public and private LB by about 0.01. \n\nI come from an optics/imaging hardware background. When I looked at the train and test image sets, one of the first things that jumped out at me was that the background black area was completely black (0). In any types of imaging systems, the 8-bit pixel values inside the images comes from some sort of A/D conversion unit that has inherent noise, so normally you would see the background areas not entirely consisting of 0s, you would see some zeros, some ones, maybe some twos. In order to get a completely black background, either a bias was added to the A/D conversion, or a digital threshold was applied afterwards. Either way, this means that the zeros were not true zeros, in reality they should have been a negative number. Therefore in the images, the difference between the 8-bit pixel values 0 and 1 is more significant than the difference between 1 and 2.\n\nOne way to exploit this is to apply a non-linear function on the images before feeding them to the neural networks. I simply applied a square root function. This boosted my macro F1 scores by about 0.01 across all cv and LBs. There might be more effective or efficient ways to exploit this little piece of information. Maybe you guys would come up with better ideas.",
      "votes": null
    },
    {
      "id": "453960",
      "postDate": "01/11/2019 01:38:53",
      "content": "<p>I tried to do gamma correction of images, but not very successfully...</p>",
      "rawMarkdown": "I tried to do gamma correction of images, but not very successfully...",
      "votes": null
    },
    {
      "id": "454040",
      "postDate": "01/11/2019 04:03:54",
      "content": "<p>Given others' comments on the influences of image brightness and contrast, now I'm not so sure if my square-root image preprocessing worked by amplifying the differences at low intensity levels, or by making the model less sensitive to brightness differences...</p>",
      "rawMarkdown": "Given others' comments on the influences of image brightness and contrast, now I'm not so sure if my square-root image preprocessing worked by amplifying the differences at low intensity levels, or by making the model less sensitive to brightness differences...",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 453960,
      "author_name": "iafoss",
      "author_url": "",
      "post_date": "01/11/2019 01:38:53",
      "content": "<p>I tried to do gamma correction of images, but not very successfully...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 454040,
      "author_name": "mingzhao03",
      "author_url": "",
      "post_date": "01/11/2019 04:03:54",
      "content": "<p>Given others' comments on the influences of image brightness and contrast, now I'm not so sure if my square-root image preprocessing worked by amplifying the differences at low intensity levels, or by making the model less sensitive to brightness differences...</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "453952": "TL; DR: Applying a non-linear image preprocessing step on the images increased my macro F1-score for CV, public and private LB by about 0.01. \n\nI come from an optics/imaging hardware background. When I looked at the train and test image sets, one of the first things that jumped out at me was that the background black area was completely black (0). In any types of imaging systems, the 8-bit pixel values inside the images comes from some sort of A/D conversion unit that has inherent noise, so normally you would see the background areas not entirely consisting of 0s, you would see some zeros, some ones, maybe some twos. In order to get a completely black background, either a bias was added to the A/D conversion, or a digital threshold was applied afterwards. Either way, this means that the zeros were not true zeros, in reality they should have been a negative number. Therefore in the images, the difference between the 8-bit pixel values 0 and 1 is more significant than the difference between 1 and 2.\n\nOne way to exploit this is to apply a non-linear function on the images before feeding them to the neural networks. I simply applied a square root function. This boosted my macro F1 scores by about 0.01 across all cv and LBs. There might be more effective or efficient ways to exploit this little piece of information. Maybe you guys would come up with better ideas.",
    "453960": "I tried to do gamma correction of images, but not very successfully...",
    "454040": "Given others' comments on the influences of image brightness and contrast, now I'm not so sure if my square-root image preprocessing worked by amplifying the differences at low intensity levels, or by making the model less sensitive to brightness differences..."
  },
  "source": "meta"
}