![](\"https://i.ibb.co/CWSsS7k/domain-shift1.jpg\")
PLEASE GIVE CREDIT TO THE ORIGINAL AUTHOR --> Heather D. Couture Ph.D.
\nI'm sharing it as I found it incredibly helpful and instead of posting the link I thought reiterating the content here would help more people
\nOne of the largest challenges in histopathology image analysis is creating models that are robust to the variations across different labs and imaging systems. These variations can be caused by:
\n
Figure 1: H&E image variations induced by different scanners [ref: Aubreville2021]
\nDifferent setups can produce images with different stain intensities or other changes, creating a domain shift between the source data that the model was trained on and the target data on which a deployed solution would need to operate. When the domain shift is too large, a model trained on one type of data will fail on another, often in unpredictable ways.
\nI’ve [Heather D. Couture Ph.D.] been working with a client who is interested in selecting the best object detection model for their use case. But the images their model will examine once deployed come from a different lab and a variety of scanners.
\n![\"domain-shift2\"](\"https://i.ibb.co/7bJ3yKT/domain-shift2.jpg\")
Figure 2: Mitosis appearance in two different datasets: Radboudumc (top) and TUPAC (bottom) [ref: Tellez2018]
\nSo how do you handle the domain shift? There are a few different options:
\nSome of these approaches have opposing goals.
\n![\"domain-shift3\"](\"https://i.ibb.co/8Kd56k0/domain-shift3.jpg\")
Figure 3: The different goals of color augmentation, stain normalization, and domain alignment by with a GAN [ref: Stacke2020]
\nThis article [Kaggle Discussion Post] will review each of the five strategies, followed by a summary of studies revealing which method(s) work best.
\nDifferent labs and scanners can produce images with different color profiles for a particular stain. The goal of stain normalization is to standardize the appearance of these stains.
\nTraditionally, methods like color matching [Reinhard2001] and stain separation [Macenko2009, Khan2014, Vahadane2016] were used. However, these methods rely on selection of a single reference slide. Ren et al. have since shown that using an ensemble with different reference slides is one possible solution [Ren2019].
\n![\"domain-shift4\"](\"https://i.ibb.co/MVKpDX9/domain-shift4.jpg\")
Figure 4: H&E image (left) and deconvolved hematoxylin (middle) and eosin (right) [ref: Macenko2009]
\nThe larger problem is that these techniques do not consider spatial features, which can lead to tissue structure not being preserved.
\nGenerative Adversarial Networks (GANs) are the state-of-the-art in stain normalization today.
\nIf paired and aligned images from domains A and B are available, this setup performs well.
\nBut there is a simpler solution to obtaining paired images:
\n![\"domain-shift5\"](\"https://i.ibb.co/pjpknGP/domain-shift5.jpg\")
Figure 5: Stain style transfer from grayscale to H&E with a Conditional GAN [ref: Cho2017]
\nOne major advantage of this approach is that a restaining model trained for one particular domain may work for a variety of different labs and scanners as there is less variation in the input grayscale images than in color.
\nAn alternative approach when paired images are not available is a CycleGAN [Zhu2017].
\n![\"domain-shift6\"](\"https://i.ibb.co/3psgzCS/domain-shift6.jpg\")
Figure 6: Stain style transfer with a CycleGAN [ref: Lo2021]
\nStain normalization methods using deep learning have become increasingly complex. As a first pass to see if this type of standardization is helpful for your task, I [Heather D. Couture Ph.D.] suggest starting simple.
\n![\"domain-shift7\"](\"https://i.ibb.co/crDH8mx/domain-shift7.jpg\")
Figure 7: Comparison of stain normalization results on different datasets [ref: Salehi2020]
\nImage augmentation by applying random affine transforms or adding noise is one of the most common regularization techniques for combating overfitting. Similarly, variations in staining can be taken advantage of to increase the diversity of image appearance presented to the model during training.
\nFaryna et al. demonstrated the RandAugment technique on histopathology [Faryna2021].
![\"domain-shift8\"](\"https://i.ibb.co/Hdk0rrS/domain-shift8.jpg\")
Figure 8: Image augmentation techniques [ref: Faryna2021]
\nTellez et al. studied the effect of different augmentation techniques (individually and combined) on mitosis detection and proposed an H&E-specific transform [Tellez2018].
\n![\"domain-shift9\"](\"https://i.ibb.co/QfHDR7S/domain-shift9.jpg\")
Figure 9: Color augmentation on hematoxylin and eosin channels [ref: Tellez2018]
\nThe next technique for domain adaptation is domain adversarial training [Ganin2016]. This approach makes use of unlabeled images from the target domain.
\nA domain adversarial module is added to an existing model.
\nDuring training, labeled images from the source domain and unlabeled ones from the target domain are used.
\n![\"domain-shift10\"](\"https://i.ibb.co/qRmDYY7/domain-shift10.jpg\")
Figure 10: A domain adversarial module attached to a CNN classifier [ref: Ganin2016]
\nThis module can be added to a variety of deep learning models.
\n![\"domain-shift11\"](\"https://i.ibb.co/R9VpHhq/domain-shift11.jpg\")
Figure 11: Improved domain invariance with a domain adapted model [ref: Aubreville2020a]
\nInstead of accommodating domain shifts during training, the model may be modified at test time.
\nThese new statistics can be calculated over the whole test set and updated in the model before running inference.
\n![\"domain-shift12\"](\"https://i.ibb.co/9ZgCPy5/domain-shift12.jpg\")
Figure 12: Prediction-time batch normalization aligns the shifted activations with the training distribution [ref: Nado2020]
\nFinally, a model may be finetuned on a test set with a domain shift [Aubreville2020b].
\nColor augmentation and stain normalization are used extensively in pathology image applications, especially for whole slide H&E images. Domain adversarial training and model adaptation at test time are less studied thus far.
\nKhan et al. studied stain normalization and color augmentation when applied individually and together [Khan2020].
\nTellez et al. tested different combinations of image augmentation and normalization strategies for a variety of different histology classification tasks [Tellez2019].
\nRen et al. compared domain adversarial training with some stain normalization and color augmentation approaches, demonstrating that the domain adversarial approach was superior for generalizing to new image sets [Ren2019].
\nLarfarge et al. performed a similar study that also included domain adversarial training.
\nThey compared domain adversarial training with color augmentation and stain normalization on mitosis classification and nuclei segmentation tasks [Lafarge2019].
\nFor nuclei segmentation, the results were a bit different.
\nClearly domain adversarial training was beneficial for both types of domain shifts!
\nThe stain normalization methods tested in the above analysis of methods are only the traditional color matching and stain separation techniques.
\nTraditional stain normalization techniques are worth trying as a first pass as they’re easier to implement and often faster to run.
\nThe available data may also be a deciding factor in selecting appropriate techniques.
\nFor these reasons, stain normalization and color augmentation are often tried first, with adversarial domain adaptation included when needed.
\nMonitoring a deployed system for unexpected domain shifts is also critical.
\nPLEASE GIVE CREDIT TO THE ORIGINAL AUTHOR --> Heather D. Couture Ph.D.
\nI'm sharing it as I found it incredibly helpful and instead of posting the link I thought reiterating the content here would help more people
\nOne of the largest challenges in histopathology image analysis is creating models that are robust to the variations across different labs and imaging systems. These variations can be caused by:
\n
Figure 1: H&E image variations induced by different scanners [ref: Aubreville2021]
\nDifferent setups can produce images with different stain intensities or other changes, creating a domain shift between the source data that the model was trained on and the target data on which a deployed solution would need to operate. When the domain shift is too large, a model trained on one type of data will fail on another, often in unpredictable ways.
\nI’ve [Heather D. Couture Ph.D.] been working with a client who is interested in selecting the best object detection model for their use case. But the images their model will examine once deployed come from a different lab and a variety of scanners.
\n
Figure 2: Mitosis appearance in two different datasets: Radboudumc (top) and TUPAC (bottom) [ref: Tellez2018]
\nSo how do you handle the domain shift? There are a few different options:
\nSome of these approaches have opposing goals.
\n
Figure 3: The different goals of color augmentation, stain normalization, and domain alignment by with a GAN [ref: Stacke2020]
\nThis article [Kaggle Discussion Post] will review each of the five strategies, followed by a summary of studies revealing which method(s) work best.
\nDifferent labs and scanners can produce images with different color profiles for a particular stain. The goal of stain normalization is to standardize the appearance of these stains.
\nTraditionally, methods like color matching [Reinhard2001] and stain separation [Macenko2009, Khan2014, Vahadane2016] were used. However, these methods rely on selection of a single reference slide. Ren et al. have since shown that using an ensemble with different reference slides is one possible solution [Ren2019].
\n
Figure 4: H&E image (left) and deconvolved hematoxylin (middle) and eosin (right) [ref: Macenko2009]
\nThe larger problem is that these techniques do not consider spatial features, which can lead to tissue structure not being preserved.
\nGenerative Adversarial Networks (GANs) are the state-of-the-art in stain normalization today.
\nIf paired and aligned images from domains A and B are available, this setup performs well.
\nBut there is a simpler solution to obtaining paired images:
\n
Figure 5: Stain style transfer from grayscale to H&E with a Conditional GAN [ref: Cho2017]
\nOne major advantage of this approach is that a restaining model trained for one particular domain may work for a variety of different labs and scanners as there is less variation in the input grayscale images than in color.
\nAn alternative approach when paired images are not available is a CycleGAN [Zhu2017].
\n
Figure 6: Stain style transfer with a CycleGAN [ref: Lo2021]
\nStain normalization methods using deep learning have become increasingly complex. As a first pass to see if this type of standardization is helpful for your task, I [Heather D. Couture Ph.D.] suggest starting simple.
\n
Figure 7: Comparison of stain normalization results on different datasets [ref: Salehi2020]
\nImage augmentation by applying random affine transforms or adding noise is one of the most common regularization techniques for combating overfitting. Similarly, variations in staining can be taken advantage of to increase the diversity of image appearance presented to the model during training.
\nFaryna et al. demonstrated the RandAugment technique on histopathology [Faryna2021].
Figure 8: Image augmentation techniques [ref: Faryna2021]
\nTellez et al. studied the effect of different augmentation techniques (individually and combined) on mitosis detection and proposed an H&E-specific transform [Tellez2018].
\n
Figure 9: Color augmentation on hematoxylin and eosin channels [ref: Tellez2018]
\nThe next technique for domain adaptation is domain adversarial training [Ganin2016]. This approach makes use of unlabeled images from the target domain.
\nA domain adversarial module is added to an existing model.
\nDuring training, labeled images from the source domain and unlabeled ones from the target domain are used.
\n
Figure 10: A domain adversarial module attached to a CNN classifier [ref: Ganin2016]
\nThis module can be added to a variety of deep learning models.
\n
Figure 11: Improved domain invariance with a domain adapted model [ref: Aubreville2020a]
\nInstead of accommodating domain shifts during training, the model may be modified at test time.
\nThese new statistics can be calculated over the whole test set and updated in the model before running inference.
\n
Figure 12: Prediction-time batch normalization aligns the shifted activations with the training distribution [ref: Nado2020]
\nFinally, a model may be finetuned on a test set with a domain shift [Aubreville2020b].
\nColor augmentation and stain normalization are used extensively in pathology image applications, especially for whole slide H&E images. Domain adversarial training and model adaptation at test time are less studied thus far.
\nKhan et al. studied stain normalization and color augmentation when applied individually and together [Khan2020].
\nTellez et al. tested different combinations of image augmentation and normalization strategies for a variety of different histology classification tasks [Tellez2019].
\nRen et al. compared domain adversarial training with some stain normalization and color augmentation approaches, demonstrating that the domain adversarial approach was superior for generalizing to new image sets [Ren2019].
\nLarfarge et al. performed a similar study that also included domain adversarial training.
\nThey compared domain adversarial training with color augmentation and stain normalization on mitosis classification and nuclei segmentation tasks [Lafarge2019].
\nFor nuclei segmentation, the results were a bit different.
\nClearly domain adversarial training was beneficial for both types of domain shifts!
\nThe stain normalization methods tested in the above analysis of methods are only the traditional color matching and stain separation techniques.
\nTraditional stain normalization techniques are worth trying as a first pass as they’re easier to implement and often faster to run.
\nThe available data may also be a deciding factor in selecting appropriate techniques.
\nFor these reasons, stain normalization and color augmentation are often tried first, with adversarial domain adaptation included when needed.
\nMonitoring a deployed system for unexpected domain shifts is also critical.
\nFollow and/or visit the author Heather D. Couture Ph.D. on LinkedIn for other insightful posts and information related to the domain of Artificial Intelligence in Pathology (and much more!).
", "rawMarkdown": "Follow and/or visit the author [**Heather D. Couture**](https://www.linkedin.com/in/hdcouture/) Ph.D. on [**LinkedIn**](https://www.linkedin.com/in/hdcouture/) for other insightful posts and information related to the domain of Artificial Intelligence in Pathology (and much more!).", "votes": null }, { "id": "1862642", "postDate": "07/19/2022 21:37:08", "content": "Great work!
", "rawMarkdown": "Great work!", "votes": null }, { "id": "1862737", "postDate": "07/20/2022 00:32:26", "content": "Super! Thank you very much for this useful sharing!
", "rawMarkdown": "Super! Thank you very much for this useful sharing!", "votes": null }, { "id": "1867486", "postDate": "07/23/2022 08:46:25", "content": "thanks for sharing!
", "rawMarkdown": "thanks for sharing!", "votes": null }, { "id": "1892616", "postDate": "08/10/2022 07:40:31", "content": "I have implemented stain transfer using Pix2Pix (mentioned in Section 1 above) which uses gray/color image pairs. I used HPA images to train the GAN but external HuBMAP images can be used as well. Training with HuBMAP images will give HuBMAP (H&E) stained images.
", "rawMarkdown": "I have implemented [stain transfer using Pix2Pix](https://www.kaggle.com/code/shir0mani/hubmap-stain-transfer-w-pix2pix-pytorch) (mentioned in Section 1 above) which uses gray/color image pairs. I used HPA images to train the GAN but external HuBMAP images can be used as well. Training with HuBMAP images will give HuBMAP (H&E) stained images.", "votes": null } ], "comments": [ { "id": 1860952, "author_name": "dschettler8845", "author_url": "", "post_date": "07/18/2022 17:03:23", "content": "Follow and/or visit the author Heather D. Couture Ph.D. on LinkedIn for other insightful posts and information related to the domain of Artificial Intelligence in Pathology (and much more!).
", "votes": null, "replies": [] }, { "id": 1862642, "author_name": "muhammad4hmed", "author_url": "", "post_date": "07/19/2022 21:37:08", "content": "Great work!
", "votes": null, "replies": [] }, { "id": 1862737, "author_name": "mawanda", "author_url": "", "post_date": "07/20/2022 00:32:26", "content": "Super! Thank you very much for this useful sharing!
", "votes": null, "replies": [] }, { "id": 1867486, "author_name": "bridgeoverwater", "author_url": "", "post_date": "07/23/2022 08:46:25", "content": "thanks for sharing!
", "votes": null, "replies": [] }, { "id": 1892616, "author_name": "shir0mani", "author_url": "", "post_date": "08/10/2022 07:40:31", "content": "I have implemented stain transfer using Pix2Pix (mentioned in Section 1 above) which uses gray/color image pairs. I used HPA images to train the GAN but external HuBMAP images can be used as well. Training with HuBMAP images will give HuBMAP (H&E) stained images.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1860944": "