{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# HuBMAP - Efficient Sampling Baseline (deepflash2, pytorch, fastai) [train]\n\n> Kernel for model training with efficient region based sampling.\n\nRequires deepflash2 (git version), zarr, and segmentation-models-pytorch\n","metadata":{"id":"OcsetTMwKXqC"}},{"cell_type":"markdown","source":"## Overview\n\n1. Installation and package loading\n2. Helper functions and patches\n3. Configuration\n4. Training\n\n### Inputs\n- https://www.kaggle.com/matjes/hubmap-zarr converted images (downscaled with factor 2)\n- https://www.kaggle.com/matjes/hubmap-labels-pdf-0-5-0-25-0-01 masks and weights for sampling\n\n### Versions\n- V7: Fixed augmentations in deepflash2 `RandomTileDataset` config (random zoom) - LB 0.913\n- V8: Adding *albumentations* transforms, switching to Cross-entropy loss","metadata":{}},{"cell_type":"markdown","source":"## Motivation\n\n### Background\n\nA glomerulus is a network of small blood vessels located at the beginning of a nephron in the kidney ([Wikipedia](https://en.wikipedia.org/wiki/Glomerulus_(kidney))\n)). Glomeruli are mainly found in the renal **cortex**, while the renal **medulla** contains mainly the renal tubule. Since we are dealing with biological structures, the separation is not not absolute and the transitions are not always perfectly sharp.\n\n![Diagram of a nephron](http://s3-us-west-2.amazonaws.com/courses-images/wp-content/uploads/sites/1842/2017/05/26234530/m9skcbftjqzrokkkopam.png)\n[Diagram of a nephron from libretexts.org, Introductory and General Biology](https://bio.libretexts.org/Bookshelves/Introductory_and_General_Biology/Book%3A_General_Biology_(Boundless)/41%3A_Osmotic_Regulation_and_the_Excretory_System/41.4%3A_Human_Osmoregulatory_and_Excretory_Systems/41.4B%3A_Nephron%3A_The_Functional_Unit_of_the_Kidney)\n\n### Key Idea\n\nA common approach to deal with the very large (>500MB - 5GB) TIFF files in the dataset is to decompose the images in smaller patches/tiles, for instance by using a sliding window apporach.\n> **Knowing that the glomeruli are mainly found in the cortex, we should focus on this region during training**. \n\nInstead of preprocessing the images and saving them into fixed tiles, we sample tiles from the entire images with a higher probability on tiles that contain glumeroli and cortex. Have a look at [this kernel](https://www.kaggle.com/matjes/hubmap-labels-pdf-0-5-0-25-0-01) for more details.\n\n\n## Advantages of this approach\n\nIn combination with [deepflash2](https://github.com/matjesg/deepflash2/tree/master/) and the deepflash2 [pytorch datasets](https://matjesg.github.io/deepflash2/data.html#Datasets) in particular, this approach has several advantages:\n- no preprocessing of the data (only saving them to .zarr files for memory efficient loading)\n    - flexible tile shapes (input shapes, e.g. 1024, 512, 256) at runtime\n    - flexible scaling (e.g., by facors of 2,3,4)\n- faster convergence during traing (~30 min for training a competitive model)\n    - focusing on the relevant regions (e.g., tiles that contain glumeroli and cortex)\n    - \"additional\" data augmentation from random sampling (compared to fixed windows)","metadata":{}},{"cell_type":"markdown","source":"### Installation and package loading","metadata":{}},{"cell_type":"code","source":"# Install deepflash2 and dependencies\nimport sys\nsys.path.append(\"../input/zarrkaggleinstall\")\nsys.path.append(\"../input/segmentation-models-pytorch-install\")\n!pip install -q --no-deps ../input/deepflash2-lfs\nimport cv2, torch, zarr, tifffile, pandas as pd, gc\nfrom fastai.vision.all import *\nfrom deepflash2.all import *\nimport segmentation_models_pytorch as smp\nimport albumentations as alb","metadata":{"execution":{"iopub.status.busy":"2022-09-02T15:58:48.011593Z","iopub.execute_input":"2022-09-02T15:58:48.012058Z","iopub.status.idle":"2022-09-02T15:58:51.888434Z","shell.execute_reply.started":"2022-09-02T15:58:48.012021Z","shell.execute_reply":"2022-09-02T15:58:51.887391Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# !pip install -q git+https://github.com/matjesg/deepflash2.git\n# !pip install segmentation_models_pytorch","metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-09-02T15:58:51.891072Z","iopub.execute_input":"2022-09-02T15:58:51.891809Z","iopub.status.idle":"2022-09-02T15:58:51.898317Z","shell.execute_reply.started":"2022-09-02T15:58:51.891765Z","shell.execute_reply":"2022-09-02T15:58:51.897294Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# imports\n# import zarr, cv2\n# import numpy as np, pandas as pd, segmentation_models_pytorch as smp\n# from fastai.vision.all import *\n# from deepflash2.all import *\n# import albumentations as alb","metadata":{"execution":{"iopub.status.busy":"2022-09-02T15:58:51.899899Z","iopub.execute_input":"2022-09-02T15:58:51.900791Z","iopub.status.idle":"2022-09-02T15:58:51.908288Z","shell.execute_reply.started":"2022-09-02T15:58:51.900754Z","shell.execute_reply":"2022-09-02T15:58:51.907320Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Helper functions and patches","metadata":{}},{"cell_type":"markdown","source":"![](http://)Patches for deepflash2 classes, see https://fastcore.fast.ai/basics.html#patch","metadata":{}},{"cell_type":"code","source":"@patch\ndef read_img(self:BaseDataset, file, *args, **kwargs):\n    return zarr.open(str(file), mode='r')\n\n@patch\ndef _name_fn(self:BaseDataset, g):\n    \"Name of preprocessed and compressed data.\"\n    return f'{g}'\n\n@patch\ndef apply(self:DeformationField, data, offset=(0, 0), pad=(0, 0), order=1):\n    \"Apply deformation field to image using interpolation\"\n    outshape = tuple(int(s - p) for (s, p) in zip(self.shape, pad))\n    coords = [np.squeeze(d).astype('float32').reshape(*outshape) for d in self.get(offset, pad)]\n    # Get slices to avoid loading all data (.zarr files)\n    sl = []\n    for i in range(len(coords)):\n        cmin, cmax = int(coords[i].min()), int(coords[i].max())\n        dmax = data.shape[i]\n        if cmin<0: \n            cmax = max(-cmin, cmax)\n            cmin = 0 \n        elif cmax>dmax:\n            cmin = min(cmin, 2*dmax-cmax)\n            cmax = dmax\n            coords[i] -= cmin\n        else: coords[i] -= cmin\n        sl.append(slice(cmin, cmax))    \n    if len(data.shape) == len(self.shape) + 1:\n        tile = np.empty((*outshape, data.shape[-1]))\n        for c in range(data.shape[-1]):\n            # Adding divide\n            tile[..., c] = cv2.remap(data[sl[0],sl[1], c]/255, coords[1],coords[0], interpolation=order, borderMode=cv2.BORDER_REFLECT)\n    else:\n        tile = cv2.remap(data[sl[0], sl[1]], coords[1], coords[0], interpolation=order, borderMode=cv2.BORDER_REFLECT)\n    return tile","metadata":{"execution":{"iopub.status.busy":"2022-09-02T15:58:51.910040Z","iopub.execute_input":"2022-09-02T15:58:51.910764Z","iopub.status.idle":"2022-09-02T15:58:51.928036Z","shell.execute_reply.started":"2022-09-02T15:58:51.910729Z","shell.execute_reply":"2022-09-02T15:58:51.927060Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Config","metadata":{}},{"cell_type":"code","source":"class CONFIG():\n    \n    # data paths\n    data_path = Path('../input/hubmap-kidney-segmentation')\n    data_path_zarr = Path('../input/with-external-hubmap-zarr/train_scale2')\n    mask_preproc_dir = '/kaggle/input/hubmap-labels-0-75/masks_scale2'\n    \n    # deepflash2 dataset\n    scale = 1.5 # data is already downscaled to 2, so absulute downscale is 3\n    tile_shape = (512, 512)\n    padding = (0,0) # Border overlap for prediction\n    n_jobs = 1\n    sample_mult = 100 # Sample 100 tiles from each image, per epoch\n    val_length = 500 # Randomly sample 500 validation tiles\n    stats = np.array([0.61561477, 0.5179343 , 0.64067212]), np.array([0.2915353 , 0.31549066, 0.28647661])\n    \n    # deepflash2 augmentation options\n    zoom_sigma = 0.1\n    flip = True\n    max_rotation = 360\n    deformation_grid_size = (150,150)\n    deformation_magnitude = (10,10)\n    \n\n    # pytorch model (segmentation_models_pytorch)\n    encoder_name = \"efficientnet-b4\"\n    encoder_weights = 'imagenet'\n    in_channels = 3\n    classes = 2\n    \n    # fastai Learner \n    mixed_precision_training = True\n    batch_size = 16\n    weight_decay = 0.01\n    loss_func = CrossEntropyLossFlat(axis=1)\n    metrics = [Iou(), Dice_f1()]\n    optimizer = ranger\n    max_learning_rate = 1e-3\n    epochs = 12\n    \ncfg = CONFIG()","metadata":{"execution":{"iopub.status.busy":"2022-09-02T15:58:51.931295Z","iopub.execute_input":"2022-09-02T15:58:51.932212Z","iopub.status.idle":"2022-09-02T15:58:51.944792Z","shell.execute_reply.started":"2022-09-02T15:58:51.932176Z","shell.execute_reply":"2022-09-02T15:58:51.943668Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Albumentations augmentations\n# Inspired by https://www.kaggle.com/iafoss/hubmap-pytorch-fast-ai-starter\n# deepflash2 augmentations are only affine transformations\ntfms = alb.OneOf([\n    alb.HueSaturationValue(10,15,10),\n    alb.CLAHE(clip_limit=2),\n    alb.RandomBrightnessContrast(),            \n    ], p=0.3)","metadata":{"execution":{"iopub.status.busy":"2022-09-02T15:58:51.946529Z","iopub.execute_input":"2022-09-02T15:58:51.947278Z","iopub.status.idle":"2022-09-02T15:58:51.957717Z","shell.execute_reply.started":"2022-09-02T15:58:51.947234Z","shell.execute_reply":"2022-09-02T15:58:51.956703Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train = pd.read_csv(cfg.data_path/'train.csv')\ndf_info = pd.read_csv(cfg.data_path/'HuBMAP-20-dataset_information.csv')\n\nfiles = [x for x in cfg.data_path_zarr.iterdir() if x.is_dir() if not x.name.startswith('.')]\nlabel_fn = lambda o: o","metadata":{"execution":{"iopub.status.busy":"2022-09-02T15:58:51.959439Z","iopub.execute_input":"2022-09-02T15:58:51.960219Z","iopub.status.idle":"2022-09-02T15:58:52.131926Z","shell.execute_reply.started":"2022-09-02T15:58:51.960182Z","shell.execute_reply":"2022-09-02T15:58:52.131138Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training","metadata":{}},{"cell_type":"code","source":"# Model\nmodel = smp.Unet(encoder_name=cfg.encoder_name, \n                 encoder_weights=cfg.encoder_weights, \n                 in_channels=cfg.in_channels, \n                 classes=cfg.classes)","metadata":{"execution":{"iopub.status.busy":"2022-09-02T15:58:52.134354Z","iopub.execute_input":"2022-09-02T15:58:52.135004Z","iopub.status.idle":"2022-09-02T15:58:52.517527Z","shell.execute_reply.started":"2022-09-02T15:58:52.134965Z","shell.execute_reply":"2022-09-02T15:58:52.516710Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Datasets\nds_kwargs = {\n    'tile_shape':cfg.tile_shape,\n    'padding':cfg.padding,\n    'scale': cfg.scale,\n    'n_jobs': cfg.n_jobs, \n    'preproc_dir': cfg.mask_preproc_dir, \n    'val_length':cfg.val_length, \n    'sample_mult':cfg.sample_mult,\n    'loss_weights':False,\n    'zoom_sigma': cfg.zoom_sigma,\n    'flip' : cfg.flip,\n    'max_rotation': cfg.max_rotation,\n    'deformation_grid_size' : cfg.deformation_grid_size,\n    'deformation_magnitude' : cfg.deformation_magnitude,\n    'albumentations_tfms': tfms\n}\n\ntrain_ds = RandomTileDataset(files, label_fn=label_fn, **ds_kwargs)\nvalid_ds = TileDataset(files, label_fn=label_fn, **ds_kwargs, is_zarr=True)","metadata":{"execution":{"iopub.status.busy":"2022-09-02T15:58:52.519977Z","iopub.execute_input":"2022-09-02T15:58:52.520656Z","iopub.status.idle":"2022-09-02T15:58:54.239458Z","shell.execute_reply.started":"2022-09-02T15:58:52.520598Z","shell.execute_reply":"2022-09-02T15:58:54.237666Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Dataloader and learner\ndls = DataLoaders.from_dsets(train_ds, valid_ds, bs=cfg.batch_size, after_batch=Normalize.from_stats(*cfg.stats))\nif torch.cuda.is_available(): dls.cuda(), model.cuda()\ncbs = [SaveModelCallback(monitor='iou'), ElasticDeformCallback]\nlearn = Learner(dls, model, metrics=cfg.metrics, wd=cfg.weight_decay, loss_func=cfg.loss_func, opt_func=ranger, cbs=cbs)\nif cfg.mixed_precision_training: learn.to_fp16()\n  ","metadata":{"execution":{"iopub.status.busy":"2022-09-02T15:58:54.240737Z","iopub.execute_input":"2022-09-02T15:58:54.241256Z","iopub.status.idle":"2022-09-02T15:58:58.539596Z","shell.execute_reply.started":"2022-09-02T15:58:54.241215Z","shell.execute_reply":"2022-09-02T15:58:58.538817Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Fit\nlearn.fit_one_cycle(cfg.epochs, lr_max=cfg.max_learning_rate)\nlearn.recorder.plot_metrics()","metadata":{"execution":{"iopub.status.busy":"2022-09-02T15:58:58.541486Z","iopub.execute_input":"2022-09-02T15:58:58.542005Z","iopub.status.idle":"2022-09-02T16:03:18.572178Z","shell.execute_reply.started":"2022-09-02T15:58:58.541966Z","shell.execute_reply":"2022-09-02T16:03:18.568294Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Fit\nlearn.fit_one_cycle(cfg.epochs, lr_max=cfg.max_learning_rate)\nlearn.recorder.plot_metrics()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Save Model\nstate = {'model': learn.model.state_dict(), 'stats':cfg.stats}\ntorch.save(state, f'unet_{cfg.encoder_name}.pth', pickle_protocol=2, _use_new_zipfile_serialization=False)","metadata":{"execution":{"iopub.status.busy":"2022-09-02T16:03:18.573457Z","iopub.status.idle":"2022-09-02T16:03:18.574237Z"},"trusted":true},"execution_count":null,"outputs":[]}]}