{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# HuBMAP deepflash2 Judge Price Submission\n\n> Outline of the deepflash2* approach\n\n***\n*deepflash2 is our kaggle team name and our [open source python library](https://matjesg.github.io/deepflash2/) that aims to combine state-of-the-art deep learning with a barrier-free environment for life science researchers.\n\n# Highlights\n\n> Super fast training: 30 minutes training to reach a public LB score of 0.922.\n- **Efficient Sampling**\n - no \"pre-tiling\" needed, the training data gets converted into .zarr files for efficient loading.\n - flexible tile dimensions (e.g. 1024, 512, 256) & downscaling (2, 3, 4x) at runtime \n - training focuses on the relevant regions (e.g., tiles that contain glomeruli and cortex)\n - during data augmentation we have no cropping artifacts during rotation etc.\n- **Standardized Workflows**\n - Using best practices for training schedules, architectures, augmentations and inference\n - leveraging the `pytorch` ecosystem: `deepflash2`, `fastai`, `Albumentations`, `segmentation-models.pytorch`\n - *production ready*: our apporach provides an outstanding prediction quality without stacking many different architectures and encoders.\n- **Uncertainty Estimation**\n - provides bayesian and energy based measures for uncertainty\n - enables human-in-the-loop refinement of difficult specimen\n- **Generalizability**\n - We applied our modelling technique to a new type of tissue and achived a cross validation dice score over 0.8 with only 3 annotated slices. \n- **Public Kernel Sharing and Open Source Libraries**\n - All our solutions are open source and run on Kaggle Servers. We provided public notebooks as an example to use our sampling techniques earlier in the challenge.\n\n| Public Kernels | | | |\n| --------------------| ------------------------|------------------------|------------------------|\n| [](https://www.kaggle.com/matjes/hubmap-zarr) | [](https://www.kaggle.com/matjes/hubmap-labels-pdf-0-5-0-25-0-01) | [](https://www.kaggle.com/matjes/hubmap-efficient-sampling-deepflash2-train) |[](https://www.kaggle.com/matjes/hubmap-efficient-sampling-deepflash2-sub) \n| [File Conversion](https://www.kaggle.com/matjes/hubmap-zarr) | [Efficient Samping](https://www.kaggle.com/matjes/hubmap-labels-pdf-0-5-0-25-0-01) | [Training](https://www.kaggle.com/matjes/hubmap-efficient-sampling-deepflash2-train) |[Inference](https://www.kaggle.com/matjes/hubmap-efficient-sampling-deepflash2-sub) |\n| 30 upvotes, 116 forks| 63 upvotes, 126 forks| 97 upvotes, 356 forks |113 upvotes, 561 forks |\n\n\n**Team**\n\n- Matthias Griebel, PhD Candidate & Research Associate, University of Würzburg \n- Philipp Sodmann, Physician & Research Assistant, University Hospital Würzburg\n- Thomas Lux, Physician & Clinical Scientist, University Hospital Würzburg\n\n***","metadata":{}},{"cell_type":"markdown","source":"# Table of Contents\n\n* [1. Methodology](#1)\n * [1.1. Efficient Sampling](#sampling)\n * [1.2. Training](#train)\n * [1.3. Validation](#val)\n * [1.4. Confidence estimation](#ce)\n * [1.5. Inference](#inf)\n* [2. Generalizability](#gen)\n* [3. Limitations and conclusion](#lim)","metadata":{}},{"cell_type":"markdown","source":"# 1. Methodology\n\nOur competition notebooks for the Judge Price cover the following workflow\n\n\n \n Figure 1: Proposed standard workflow\n\n\nThe corresponding kernels are \n- [File Conversion](https://www.kaggle.com/matjes/hubmap-zarr)\n- [Sampling Preparation](https://www.kaggle.com/matjes/hubmap-efficient-sampling-ii-deepflash2)\n- [Ensemble Training](https://www.kaggle.com/matjes/hubmap-deepflash2-train)\n- [Validation](https://www.kaggle.com/matjes/hubmap-deepflash2-validation)\n- [Inference (5-Fold-CV Ensemble)](https://www.kaggle.com/matjes/hubmap-deepflash2-sumbission)\n- [Inference (Ensemble at different scales)](https://www.kaggle.com/matjes/hubmap-deepflash2-scaled-ensemble-sumbission)\n\n\n# 1.1. Efficient Sampling\n\n[View our Kernel](https://www.kaggle.com/matjes/hubmap-efficient-sampling-ii-deepflash2)\n\nA common approach to deal with the very large (>500MB - 5GB) TIFF files in the dataset is to decompose the images in smaller patches/tiles, for instance by using a sliding window approach. However, whole slide images contain only few relevant regions while much of the image are either blank or contain tissue without the target class. \n\nInstead of preprocessing the images by saving them into fixed tiles, we combine two sampling approaches:\n\n> 1. Sampling tiles via center points in the proximity of every glomerulus - this ensures that each glomerulus is seen during one epoch of training at least once. \n> 2. Sampling random tiles based on region probabilities (e.g., medulla, cortex, other)\n\nWe use the provided anatomical information to train with more examples of the cortex than the medulla, because glomeruli have a higher abundance in this region. We also sampled a few examples not contained in the anatomic regions to ensure that our model can interpret these as well. The figure below depicts the sampling process of one image during one training epoch. \n\n\n \n Figure 2: Examplary sampling of one training epoch on image \"0486052bb\" \n Masks of the annotated glomeruli (left), anatomical regions (middle), sampled tiles during one epoch of training (right).\n \n\n\nThis kind of sampling is not only intuitive but also has strong theoretical support. Considering the exemplary batch of 16 images and the corresponding pixel distribution in the figures below, we can see that the data distribution clearly follows a normal distribution. This property is beneficial when using pre-trained models and generally speeds up learning and leads to faster convergence during the training of artificial neural networks. Moreover, the sampling is model-agnostic and can be used with any model on several other tasks (e.g., classification, object detection).\n\n\n \n Figure 3: One batch of 16 512x512 patches downscaled by factor 3, note that 14 out of 16 examples contain the foreground class \"glomerulus\".\n\n \n\n \n Figure 4: Pixel distribution after normalization of the batch above\n\n \nA beneficial side effect of our sampling method is that functional tissue units which were overlooked in the annotation will be rarely used in training. ","metadata":{}},{"cell_type":"markdown","source":"# 1.2. Training\n\n[View our Kernel](https://www.kaggle.com/matjes/hubmap-deepflash2-train)\n\nOur training methodology is based on best practices for training schedules (fastai), architectures (segmentation-models.pytorch), image augmentations (Albumentations) and inference (tile shift and gaussian weighting, see [nnunet](https://www.nature.com/articles/s41592-020-01008-z)). To ensure the objectivity, reliability, and validity of our model we tried to incorporate the principles outlined by [Segebarth, Griebel, et. al. (elife 2020)](https://elifesciences.org/articles/59780) by using model ensembles.\n\n**Negligible differences between various encoders and models**\n\nDuring this challenge, we trained and tested different encoders (resnet, efficientnetb0-b4) and architectures (U-Net, U-Net-plusplus, deeplab-v3) and found no significant difference in performance. Therefore, we decided to use a reasonable small encoder (efficientnet-b2) as well as a simple default U-Net. We tried the same experiment for different optimizers (SGD, AdamW, Ranger, Madgrad) and found that ranger performed most consistently.\nIn our experiments, the Dice-CrossEntropy loss worked well. We also tried more exotic loss functions like deep supervision but did not find any consistent benefit. \n\n**Resolution makes a difference**\n\nWe trained our model with different magnifications. While a higher resolution might be beneficial to identify the glomerular border (bowman's capsule) correctly, a reduced resolution (equal to less magnification) provides more context for the annotated area. We compared a resolution reduction of 2, 3, 4, 6, 8x in which a 3x resolution reduction resulted in the best dice score. For the same reason, we trained our model on an image size of 256 and 512. A smaller image size results in larger batch size and therefore in better estimates for batch norm. The larger image size resulted in better scores as well. We logged all our training runs using [wandb](https://wandb.ai). The results can be made publicly available on demand.\n\n**Final training setup**\n\n- Architecture: U-Net\n- Encoder: efficientnet-b2\n- Pretraining: imagenet\n- Loss: Dice-CrossEntropy\n- Optimizer: [ranger](https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer)\n- Learning rate: 1e-3\n- Batch Size: 16\n- Tile Size: 512x512\n- Resolultion Downscaling Factor: 3\n- Training iterations: 2500-3000 (best model selected on validation set)\n- Ensembling 1: 5 Models (5-fold cross validation)\n- Ensembling 2: 3 Models at scale 2,3, and 4 trained on all data\n- Augmentations: see training Notebook\n\nThe figure below summarizes a typical training run of one model with [one-cycle-training](https://arxiv.org/pdf/1803.09820.pdf).\n\n\n \n Figure 5: Training summary \n\n\nFurthermore, we tested the recently published [nnunet](https://www.nature.com/articles/s41592-020-01008-z), but we could not observe improved results. Because of the high computational cost (about 1 week training time for 5 folds), we did not run further experiments and discarded this approach.","metadata":{}},{"cell_type":"markdown","source":"## 1.3. Validation\n[View our Kernel](https://www.kaggle.com/matjes/hubmap-deepflash2-validation)\n\nTo evaluate the performance of our models, we trained and tested them in a five-fold cross validation. Each fold is trained on 12 and validated on 3 whole slice images.\n\n**Metrics**\n\n**Pixel level metrics during training (Figure 5)**:\nWe logged the dice f1, the precision and recall as well as the soft dice and loss for each epoch. The scores were consistent over all folds except for one image, in which our model found significantly more glomeruli than annotated in a poorly stained area.\n\n**Instance level metrics**: \nTo estimate the quality of our model on instance level (glomeruli level) instead of pixel level, we additionally calculated measures to account for the detection similarity. Similarly to [Segebarth, Griebel, et. al. (elife 2020)](https://elifesciences.org/articles/59780) we define the intersection over the union: \n\n$$\nM_{\\text{IoU}}(a,b) := \\frac{|a\\cap b|}{|a \\cup b|}\n$$\n\nA pair of objects (segmented glomeruli) with an IoU is above a threshold $t$ as correctly detected (true positive - $TP$). \nObjects that match with an IoU at or below $t$ or have no match at all are considered to be false negative ($FN$) for the source mask and false positive ($FP$) for the target mask. This allows us to calculate the Precision $M_{\\text{Precision}}$, Recall $M_{\\text{Recall}}$, and F1 score $M_{\\text{F1 score}}$ as the harmonic mean of $M_{\\text{Precision}}$ and $M_{\\text{Recall}}$:\n\n$$\nM_{\\text{Precision}}(t):=\\frac{TP(t)}{TP(t)+FP(t)}\n$$\n\n$$\nM_{\\text{Recall}}(t):=\\frac{TP(t)}{TP(t)+FN(t)}\n$$\n\n$$M_{\\text{F1 score}}(t) := 2 \\cdot \\frac{\\mathrm{M_{\\text{Precision}}(t)} \\cdot \\mathrm{M_{\\text{Recall}}(t)}}{\\mathrm{M_{\\text{Precision}}(t)} + \\mathrm{M_{\\text{Recall}}(t)}}\n$$\n\nwith $t\\in[0,1]$ as a fixed IoU threshold. If not indicated differently, we used $t=0.5$ in our calculations.\n\n**Results**\n\nThe cross validation results are depicted in Figure 6. The high recall indicates that the models can find almost all annotated glomeruli. The precision is slightly lower, which indicates that the models are sometimes misled by vessels or other tissue that look similar to a glomerulus. Overall, we think that these results are entirely convincing.\n\n\n \n Figure 6: Cross Validation Results\n","metadata":{}},{"cell_type":"markdown","source":"## 1.4. Confidence estimation\n[View our Kernel](https://www.kaggle.com/matjes/hubmap-deepflash2-validation)\n\nTo robustly estimate the confidence of our prediction, we developed an energy based approach for image segmentation that is based on the work of [Weitang et al. (2020)](https://proceedings.neurips.cc/paper/2020/file/f5496252609c43eb8a3d147ab9b9c006-Paper.pdf) for image classification. When applying a softmax prediction, neural networks often overestimate their confidence when predicting out of distribution data. Using the energy score can help to reduce the false positive rate \\[FPR\\]in such cases. The authors were able to reduce the FPR on CIFAR-10 by 18.03%. The formula for the energy is derived from the Helmholtz free energy and calculated as: $$E(x;f) = -T * log(\\sum_{i}^{K} e^{fi(x)/T})$$\nIn this equation, $K$ is the number of classes, $x$ the logits and $f$ the function (here the neural network). \nAs suggested by the authors, we chose the temperature parameter $T=1$. To allow a more intuitive interpretation of the energy (i.e., having mostly positive numbers), we calculated the *negative* energy score $-E$ in our experiments. Thus, our energy score always describes the *negative* energy in all our work.\n\n\n \n Figure 7: True positive example with high energy\n\n\n\n \n Figure 8: False positive example: using a thresholded softmax at 0.5, the network predicts a glomerulus. However, the energy is comparatively low indicating uncertainty in the prediction.\n\n\n\n \n Figure 9: Mean softmax activation values\n\n\n\nFigure 9 summarizes the softmax activations over the entire training cross-validation. Most of the glomeruli exhibit high mean softmax activations. Figure 10 displays the same data and its mean energy score. \nIn both Figures 9 and 10, softmax and energy show a positive correlation between the iou metric and the respective score, but the energy score is much more selective and allows differentiation of out-of-distribution data (positive classes which are unlike the examples the network has seen during training).\n\n\n \n Figure 10: Mean energy values\n\n\n**Pseudolabels and human-in-the-loop annotation refinement**\n\nTo extend the amount of training data, we predicted labels on the publicly available test-data as well as on the unused whole slide images published on [hubmap](https://portal.hubmapconsortium.org/search?q=pas&mapped_data_types[0]=PAS%20Stained%20Microscopy&entity_type[0]=Dataset), [the data is discussed in the challenge forums here](https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/233336).\nThe resulting predictions were manually refined by one physician in uncertain regions using qupath and a Wacom drawing tablet.\nThe uncertainty was estimated by computing the energy score on the logits. Positive instances with a low energy score were specifically reviewed.\nGlomeruli were excluded if more than half of their area was destroyed by artifacts.\nWe observed a steady improvement of our score with the number of glomeruli we found. \nTherefore, we reduced the threshold to a smaller value when reviewing the pseudo annotations to include all uncertain examples.\n\n\n \n Figure 11: Proposed human-in-the-loop annotation refinement\n\n\n**Finding reference glomeruli**\n\nOur approach also provides insights that are useful to generate reference glomeruli for inclusion into a Human Reference Atlas. \nTo identify typical glomeruli in an image, we utilize the energy score. A high score energy helps us to locate typical and artifact-free glomeruli on a whole slide image.\n\n\n \n Figure 12: Proposed reference glomeruli with a high energy score, one selected for each image provided in the training data.\n","metadata":{}},{"cell_type":"markdown","source":"### 1.5. Inference\n[View our Kernel](https://www.kaggle.com/matjes/hubmap-deepflash2-sumbission)\n\nFor inference, we combined several best practices:\n- Overlapping tiles (shift factor 0.8)\n- Gaussian weighting ([`nnunet`](https://www.nature.com/articles/s41592-020-01008-z))\n- Pre-filtering of empty tiles (thanks to @iafoss ([kernel](https://www.kaggle.com/iafoss/hubmap-pytorch-fast-ai-starter-sub))!\n- Test-time augmentation (horizontal and vertical flip)\n\nThese \"tricks\" removed almost any prediction artifacts such as half cut-off glomeruli or noise.\n\n**Post-processing delivers ambiguous results**\n\nWe compared different post-processing steps in our cross validation experiment and compared them with a 0.5 threshold value as our baseline.\nUsing a conditional random field had a negative impact on the dice score. We found no benefit in removing positive areas that were significantly smaller than the normal glomeruli size. This is most likely due to the negligible amount of pixels affected compared with the total amount of positive pixels.\nWe observed promising results when the softmax score was locally thresholded with Otsu's method. However, this did not improve the average dice when applied to all data. Thus, we did not use any post-processing in the final submission. \n\n\n\n \n Figure 13: Example of post-processing with Otsu's method\n\n\nHowever, we used post-processing to get a better approximation of our uncertainty scores based on the visual results during cross validation. We decided to post-process areas with less than 10k pixel (at scale 2) with Otsu's method and \"fill holes\" for all other regions. ","metadata":{}},{"cell_type":"markdown","source":"# 2. Generalizability: Experiments on pancreas specimen\n\n**Data**\n\nTo test for generalizability, we annotated pancreatic islets (langerhans) in three whole slice images \\[[1](https://pathology.cancerimagingarchive.net/pathdata/cptac_camicroscope/osdCamicroscope.php?tissueId=C3L-03371-25), [2](https://pathology.cancerimagingarchive.net/pathdata/cptac_camicroscope/osdCamicroscope.php?tissueId=C3L-01158-25), [3](https://pathology.cancerimagingarchive.net/pathdata/cptac_camicroscope/osdCamicroscope.php?tissueId=C3L-03350-24)\\] in the same way as the challenge data. Similar to the challenge data, pancreatic islands are a functional tissue unit of the pancreas. They function as the endocrinologic system of the pancreas and are producing insulin, these cells are destroyed in cases of diabetes type 1. A major difference to the challenge data is that the slides are stained with hematoxilin and eosin and not PAS.)\n\n\n \n Figure 14: Examplary data from the pancreas dataset\n\n\n\n\n \n Figure 15: Reference annotation for pancreatic islands\n\n\n\nThe data was downloaded from [cancerimagingarchive](https://www.cancerimagingarchive.net/) and only cancer-free tissue was included.\n\n**Our kernels**\n\n- [Pancreas image conversion notebook](https://www.kaggle.com/matjes/cptac-pda-to-zarr) \n- [Pancreas sampling notebook](https://www.kaggle.com/matjes/cptac-pda-pancreas-efficient-sampling-deepflash2) \n- [Pancreas training notebook](https://www.kaggle.com/matjes/cptac-pda-train-deepflash2) \n- [Pancreas validation notebook](https://www.kaggle.com/matjes/cptac-pda-deepflash2-validation/output)\n\n**Training**\n\nWe trained a three-fold cross validation with the same model architecture and training routine as for the kidney data. The only things to adjust for are the mean and standard deviation of the image data. \nIn about 10 epochs, each fold achieved a good dice metric for training (0.805, 0.808, 0.922), recall (0.848, 0.810, 0.916) and precision (0.827, 0.818, 0.949). \n\n\n \n Figure 16: Pancreas training summary \n\n\n\n \n Figure 17: Instance based cross validation metrics on the pancareas data set \n\n \n \n \n**Uncertainty Estimation**\n\nWe used the same code for validation and uncertainty estimation for the pancreas data. The figures below depict exemplary predictions.\nThe softmax and energy scores exhibit the same desired properties.\n\n \n Figure 18: True positive example with high energy\n\n \n\n\n \n Figure 19: Example with low energy\n\n \n \n\n \n Figure 20: Mean energy values\n\n\n# 3. Limitations and conclusion\n\n**Limitations**\n\nSince our models are not trained on the entire data, it is possible that rare artifacts like small air bubbles are not seen during training and get misclassified by the model for glomeruli. These regions have a lower energy score compared to normal glomeruli and can easily be found with human-in-the-loop quality control.\nWhen training the next model, these regions can be upsampled as well. \n \nFurthermore, we refrained from stacking different architectures or differently uptrained models with e.g. deviating loss functions. \nEven though this usually results in a marginally better test time performance, these models are hyper optimized for one use case and will not provide any benefit when trained on a new kind of tissue such as colon crypts.\n\n**Conclusion**\n\nOur sampling strategy is ideal to segment large volumes of whole slide images:\n- model agnostic sampling \n- model agnostic uncertainty scores \n- fast and reliable training and on standards GPUs\n\nWe have shown that our training pipeline can achieve competitive performance without the need to stack different models and can be easily transferred to new data without the need for many annotated examples. \n\nThe human-in-the-loop annotation process is ideal and helps to create more training data painlessly. It can be even further improved with custom plugins in qupath.","metadata":{}}]}
![\"workflow\"/](\"https://i.imgur.com/2mxXcJh.png\")
\n ![\"Examplary](\"https://i.imgur.com/nsEXX1M.png\")
\n ![\"One](\"https://i.imgur.com/CWpZq8Q.png\")
\n ![\"Pixel](\"https://i.imgur.com/uANDUPG.png\")
\n ![\"Pixel](\"https://i.imgur.com/kP9SCyn.png\")
\n ![\"Proposed](\"https://i.imgur.com/aT9ypRo.png\")
\n ![\"Proposed](\"https://i.imgur.com/ozVyAQT.png\")
\n ![\"Proposed](\"https://i.imgur.com/u861uOf.png\")
\n ![\"Proposed](\"https://i.imgur.com/n22iqp5.png\")
\n ![\"Proposed](\"https://i.imgur.com/WSWhuBn.png\")
\n ![\"Proposed](\"https://i.imgur.com/Zle7BDU.png\")
\n ![\"Proposed](\"https://i.imgur.com/eQPVrUj.png\")
\n ![\"Pixel](\"https://i.imgur.com/sZySuV9.png\")
\n ![\"Reference](\"https://i.imgur.com/Vmj6GRY.png\")
\n ![\"Reference](\"https://i.imgur.com/mfHEA0k.png\")
\n ![\"Pixel](\"https://i.imgur.com/4o9GyhN.png\")
\n ![\"Pixel](\"https://i.imgur.com/Sc4Rais.png\")
\n ![\"Proposed](\"https://i.imgur.com/1UqI91j.png\")
\n ![\"Proposed](\"https://i.imgur.com/BG74Yw7.png\")
\n ![\"Proposed](\"https://i.imgur.com/KqityVf.png\")
\n