{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Submission notebook of HuBMAP - Pytorch smp Unet++ Inference\n\n\nTrain part is here.\n\nhttps://www.kaggle.com/nayuts/hubmap-pytorch-smp-unet"},{"metadata":{},"cell_type":"markdown","source":"--------------------------"},{"metadata":{"papermill":{"duration":0.018407,"end_time":"2020-11-29T03:22:16.658942","exception":false,"start_time":"2020-11-29T03:22:16.640535","status":"completed"},"tags":[]},"cell_type":"markdown","source":"I create train and inference notebook for Unet++ of [segmentation_models.pytorch](https://github.com/qubvel/segmentation_models.pytorch).\n\nI use only pytorch for framework.\n\nThe accuracy of the model is going to be tuned and improved in the future.\n\nI published this notebook for our reference as an example implementation using only pytorch."},{"metadata":{"papermill":{"duration":0.016666,"end_time":"2020-11-29T03:22:16.692812","exception":false,"start_time":"2020-11-29T03:22:16.676146","status":"completed"},"tags":[]},"cell_type":"markdown","source":"I refered following two great notebooks for training and inference.\n\n- https://www.kaggle.com/iafoss/hubmap-pytorch-fast-ai-starter\n\n- https://www.kaggle.com/curiosity806/hubmap-use-catalyst-smp-and-albumentations\n\nAnd refered following great notebook for dice loss.\n\n- https://www.kaggle.com/bigironsphere/loss-function-library-keras-pytorch"},{"metadata":{"papermill":{"duration":0.016657,"end_time":"2020-11-29T03:22:16.726414","exception":false,"start_time":"2020-11-29T03:22:16.709757","status":"completed"},"tags":[]},"cell_type":"markdown","source":"## Load libraries"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","execution":{"iopub.execute_input":"2020-11-29T03:22:37.174948Z","iopub.status.busy":"2020-11-29T03:22:37.173695Z","iopub.status.idle":"2020-11-29T03:22:41.664993Z","shell.execute_reply":"2020-11-29T03:22:41.663901Z"},"papermill":{"duration":4.529346,"end_time":"2020-11-29T03:22:41.66512","exception":false,"start_time":"2020-11-29T03:22:37.135774","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"import gc\nimport os\nimport random\nimport sys\nimport time\nimport warnings\nwarnings.simplefilter(\"ignore\")\n\n#import pdb\n#import zipfile\n#import pydicom\nfrom albumentations import *\nfrom albumentations.pytorch import ToTensor\nimport cv2\nfrom matplotlib import pyplot as plt\nimport numpy as np\nimport pandas as pd\nfrom sklearn.model_selection import KFold\nimport tifffile as tiff\nimport torch\nimport torch.backends.cudnn as cudnn\nimport torch.nn as nn\nfrom torch.nn import functional as F\nimport torch.optim as optim\nfrom torch.optim.lr_scheduler import ReduceLROnPlateau\nfrom torch.utils.data import DataLoader, Dataset, sampler\nfrom tqdm import tqdm_notebook as tqdm","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To use segmentation_models_pytorch in offline environment, I loaded modules I cloned and uploaded to dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"#https://www.kaggle.com/hfutybx/unet-densenet121-lung-of-segmentation/data\n\nsys.path.append('../input/efficientnetpytorchaug252020/EfficientNet-PyTorch-master')\nsys.path.append('../input/pretrainedmodels/pretrainedmodels-0.7.4/')\nsys.path.append('../input/pytorchimagemodelsoct302020/pytorch-image-models-master')\nsys.path.append('../input/segmentation-models-pytorch0-1-2/segmentation_models.pytorch-master')\nimport segmentation_models_pytorch as smp","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.027134,"end_time":"2020-11-29T03:22:41.720618","exception":false,"start_time":"2020-11-29T03:22:41.693484","status":"completed"},"tags":[]},"cell_type":"markdown","source":"##  Set parameters"},{"metadata":{"execution":{"iopub.execute_input":"2020-11-29T03:22:41.781737Z","iopub.status.busy":"2020-11-29T03:22:41.781003Z","iopub.status.idle":"2020-11-29T03:22:41.786621Z","shell.execute_reply":"2020-11-29T03:22:41.786119Z"},"papermill":{"duration":0.038393,"end_time":"2020-11-29T03:22:41.786754","exception":false,"start_time":"2020-11-29T03:22:41.748361","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def set_seed(seed=2**3):\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    np.random.seed(seed)\n    random.seed(seed)\n    torch.backends.cudnn.deterministic = True\nset_seed(121)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.027878,"end_time":"2020-11-29T03:22:41.843067","exception":false,"start_time":"2020-11-29T03:22:41.815189","status":"completed"},"tags":[]},"cell_type":"markdown","source":"To save time, the number of \"folds\" is smaller. \nThese days, depending on the model, I think it's more common to use 5 ~ 10."},{"metadata":{"execution":{"iopub.execute_input":"2020-11-29T03:22:42.261044Z","iopub.status.busy":"2020-11-29T03:22:42.259963Z","iopub.status.idle":"2020-11-29T03:22:42.268072Z","shell.execute_reply":"2020-11-29T03:22:42.267426Z"},"papermill":{"duration":0.395969,"end_time":"2020-11-29T03:22:42.268205","exception":false,"start_time":"2020-11-29T03:22:41.872236","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"fold = 0\nnfolds = 3\nreduce = 4\nsz = 256\n\nBATCH_SIZE = 16\nDEVICE = ('cuda' if torch.cuda.is_available() else 'cpu')\nNUM_WORKERS = 4\nSEED = 2020\nTH = 0.50  #threshold for positive predictions\n\nDATA = '../input/hubmap-kidney-segmentation/test/'\nLABELS = '../input/hubmap-kidney-segmentation/train.csv'\nMASKS = '../input/hubmap-256x256/masks/'\nTRAIN = '../input/hubmap-256x256/train/'\ndf_sample = pd.read_csv('../input/hubmap-kidney-segmentation/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.029423,"end_time":"2020-11-29T03:22:42.329459","exception":false,"start_time":"2020-11-29T03:22:42.300036","status":"completed"},"tags":[]},"cell_type":"markdown","source":"## Util functions"},{"metadata":{"execution":{"iopub.execute_input":"2020-11-29T03:22:42.393494Z","iopub.status.busy":"2020-11-29T03:22:42.392768Z","iopub.status.idle":"2020-11-29T03:22:42.39574Z","shell.execute_reply":"2020-11-29T03:22:42.395262Z"},"papermill":{"duration":0.038315,"end_time":"2020-11-29T03:22:42.395843","exception":false,"start_time":"2020-11-29T03:22:42.357528","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"#https://www.kaggle.com/bguberfain/memory-aware-rle-encoding\n#with bug fix\ndef rle_encode_less_memory(img):\n    #watch out for the bug\n    pixels = img.T.flatten()\n    \n    # This simplified method requires first and last pixel to be zero\n    pixels[0] = 0\n    pixels[-1] = 0\n    runs = np.where(pixels[1:] != pixels[:-1])[0] + 2\n    runs[1::2] -= runs[::2]\n    \n    return ' '.join(str(x) for x in runs)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.028236,"end_time":"2020-11-29T03:22:42.451856","exception":false,"start_time":"2020-11-29T03:22:42.42362","status":"completed"},"tags":[]},"cell_type":"markdown","source":"## Dataset"},{"metadata":{"execution":{"iopub.execute_input":"2020-11-29T03:22:42.593715Z","iopub.status.busy":"2020-11-29T03:22:42.593035Z","iopub.status.idle":"2020-11-29T03:22:42.596222Z","shell.execute_reply":"2020-11-29T03:22:42.595755Z"},"papermill":{"duration":0.039197,"end_time":"2020-11-29T03:22:42.596336","exception":false,"start_time":"2020-11-29T03:22:42.557139","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def get_aug(p=1.0):\n    return Compose([\n        HorizontalFlip(),\n        VerticalFlip(),\n        RandomRotate90(),\n        ShiftScaleRotate(shift_limit=0.0625, scale_limit=0.2, rotate_limit=15, p=0.9, \n                         border_mode=cv2.BORDER_REFLECT),\n        OneOf([\n            OpticalDistortion(p=0.3),\n            GridDistortion(p=.1),\n            IAAPiecewiseAffine(p=0.3),\n        ], p=0.3),\n        OneOf([\n            HueSaturationValue(10,15,10),\n            CLAHE(clip_limit=2),\n            RandomBrightnessContrast(),            \n        ], p=0.3),\n    ], p=p)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.105237,"end_time":"2020-11-29T03:22:57.078897","exception":false,"start_time":"2020-11-29T03:22:56.97366","status":"completed"},"tags":[]},"cell_type":"markdown","source":"## Model"},{"metadata":{"execution":{"iopub.execute_input":"2020-11-29T03:22:57.291884Z","iopub.status.busy":"2020-11-29T03:22:57.290989Z","iopub.status.idle":"2020-11-29T03:22:57.292889Z","shell.execute_reply":"2020-11-29T03:22:57.293397Z"},"papermill":{"duration":0.110928,"end_time":"2020-11-29T03:22:57.293516","exception":false,"start_time":"2020-11-29T03:22:57.182588","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def get_UnetPlusPlus():\n    model =  smp.UnetPlusPlus(\n                 encoder_name='efficientnet-b3',\n                 encoder_weights=None,\n                 in_channels=3,\n                 classes=1)\n    return model","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.074016,"end_time":"2020-11-29T03:54:36.68492","exception":false,"start_time":"2020-11-29T03:54:36.610904","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Inference"},{"metadata":{"execution":{"iopub.execute_input":"2020-11-29T03:54:36.840684Z","iopub.status.busy":"2020-11-29T03:54:36.840073Z","iopub.status.idle":"2020-11-29T03:54:36.844203Z","shell.execute_reply":"2020-11-29T03:54:36.843568Z"},"papermill":{"duration":0.085068,"end_time":"2020-11-29T03:54:36.84431","exception":false,"start_time":"2020-11-29T03:54:36.759242","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"# https://www.kaggle.com/iafoss/256x256-images\nmean = np.array([0.65459856,0.48386562,0.69428385])\nstd = np.array([0.15167958,0.23584107,0.13146145])\n\ndef img2tensor(img,dtype:np.dtype=np.float32):\n    if img.ndim==2 : img = np.expand_dims(img,2)\n    img = np.transpose(img,(2,0,1))\n    return torch.from_numpy(img.astype(dtype, copy=False))\n\nclass HuBMAPTestDataset(Dataset):\n    def __init__(self, imgs, idxs):\n        self.imgs = imgs\n        self.fnames = idxs\n        \n    def __len__(self):\n        return len(self.fnames)\n    \n    def __getitem__(self, idx):\n        return img2tensor((self.imgs[idx]/255.0 - mean)/std)","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-11-29T03:54:37.001296Z","iopub.status.busy":"2020-11-29T03:54:37.00039Z","iopub.status.idle":"2020-11-29T03:54:38.169302Z","shell.execute_reply":"2020-11-29T03:54:38.168749Z"},"papermill":{"duration":1.250809,"end_time":"2020-11-29T03:54:38.169407","exception":false,"start_time":"2020-11-29T03:54:36.918598","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"models = []\n\n\"\"\"for fold in range(nfolds):\n\n    model = get_UnetPlusPlus().to(DEVICE)\n    model.load_state_dict(torch.load(f\"../input/hubmap-pytorch-smp-unet/FOLD{fold}_.pth\"))\n    models.append(model)\"\"\"\nmodel = smp.Unet('se_resnext50_32x4d', encoder_weights=None, classes=1).cuda()\nmodel.load_state_dict(torch.load(\"../input/testngerror/FOLD-2-model.pth\"))\nmodels.append(model)\n","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-11-29T03:54:38.336349Z","iopub.status.busy":"2020-11-29T03:54:38.335584Z","iopub.status.idle":"2020-11-29T03:54:38.338836Z","shell.execute_reply":"2020-11-29T03:54:38.339264Z"},"papermill":{"duration":0.094455,"end_time":"2020-11-29T03:54:38.339388","exception":false,"start_time":"2020-11-29T03:54:38.244933","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"#iterator like wrapper that returns predicted masks\nclass Model_pred:\n    def __init__(self, models, dl, tta:bool=True, half:bool=False):\n        self.models = models\n        self.dl = dl\n        self.tta = tta\n        self.half = half\n        \n    def __iter__(self):\n        count=0\n        with torch.no_grad():\n            for x in self.dl: #iter(self.dl):\n                x = x.to(DEVICE)\n                if self.half: x = x.half()\n                py = None\n                for model in self.models:\n                    p = model(x)\n                    p = torch.sigmoid(p).detach()\n                    if py is None: py = p\n                    else: py += p\n                if self.tta:\n                    #x,y,xy flips as TTA\n                    flips = [[-1],[-2],[-2,-1]]\n                    for f in flips:\n                        xf = torch.flip(x,f)\n                        for model in self.models:\n                            p = model(xf)\n                            p = torch.flip(p,f)\n                            py += torch.sigmoid(p).detach()\n                    py /= (1+len(flips))        \n                py /= len(self.models)\n                    \n                py = F.upsample(py, scale_factor=reduce, mode=\"bilinear\")\n                py = py.permute(0,2,3,1).float().cpu()\n                batch_size = len(py)\n                for i in range(batch_size):\n                    yield py[i]\n                    count += 1\n                    \n    def __len__(self):\n        return len(self.dl.dataset)","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-11-29T03:54:38.563616Z","iopub.status.busy":"2020-11-29T03:54:38.560133Z","iopub.status.idle":"2020-11-29T03:54:38.571478Z","shell.execute_reply":"2020-11-29T03:54:38.572616Z"},"papermill":{"duration":0.147643,"end_time":"2020-11-29T03:54:38.572812","exception":false,"start_time":"2020-11-29T03:54:38.425169","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"#Somehow I cannot resolve the submission error with consideration of the\n#private LB data, and the submission error doesn't give an informative\n#output. So, for now I share the notbook that makes a submission only\n#to the public LB, and later I'll try to resolve the issue.\n#IMPORTANT: This notebook doesn't perform predictions for the private LB.\nnames,preds = [],[]\nsamples = ['b9a3865fc','b2dc8411c','26dc41664','c68fe75ea','afa5e8098']\nsamples_n = [id for id in df_sample.id if id not in samples]\n\nnames += samples_n\npreds += [np.NaN]*len(samples_n)\ndf_sample = df_sample.loc[df_sample.id.isin(samples)]","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-11-29T03:54:38.826332Z","iopub.status.busy":"2020-11-29T03:54:38.825462Z","iopub.status.idle":"2020-11-29T04:04:29.322493Z","shell.execute_reply":"2020-11-29T04:04:29.318621Z"},"papermill":{"duration":590.629497,"end_time":"2020-11-29T04:04:29.322675","exception":false,"start_time":"2020-11-29T03:54:38.693178","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"#https://www.kaggle.com/iafoss/256x256-images\ns_th = 40  #saturation blancking threshold\np_th = 200*sz//256 #threshold for the minimum number of pixels\n#names,preds = [],[]\nfor idx,row in tqdm(df_sample.iterrows(),total=len(df_sample)):\n    idx = row['id']\n    #read image\n    img = tiff.imread(os.path.join(DATA,idx+'.tiff'))\n    if len(img.shape) == 5: img = np.transpose(img.squeeze(), (1,2,0))\n    \n    #add padding to make the image dividable into tiles\n    img_shape = img.shape\n    pad0 = (reduce*sz - img_shape[0]%(reduce*sz))%(reduce*sz)\n    pad1 = (reduce*sz - img_shape[1]%(reduce*sz))%(reduce*sz)\n    img = np.pad(img,[[pad0//2,pad0-pad0//2],[pad1//2,pad1-pad1//2],[0,0]],\n                 constant_values=0)\n\n    #split image into tiles using the reshape+transpose trick\n    if reduce != 1:\n        img = cv2.resize(img,(img.shape[1]//reduce,img.shape[0]//reduce),\n                     interpolation = cv2.INTER_AREA)\n    img_shape_p = img.shape\n    img = img.reshape(img.shape[0]//sz,sz,img.shape[1]//sz,sz,3)\n    img = img.transpose(0,2,1,3,4).reshape(-1,sz,sz,3)\n\n    #select tiles for running the model\n    imgs,idxs = [],[]\n    for i,im in enumerate(img):\n        #remove black or gray images based on saturation check\n        hsv = cv2.cvtColor(im, cv2.COLOR_BGR2HSV)\n        h, s, v = cv2.split(hsv)\n        if (s>s_th).sum() <= p_th or im.sum() <= p_th: continue\n        imgs.append(im)\n        idxs.append(i)\n    #tile dataset\n    ds = HuBMAPTestDataset(imgs,idxs)\n    dl = DataLoader(ds,batch_size=BATCH_SIZE, shuffle=False)\n    #dl = DataLoader(ds,BATCH_SIZE,num_workers=NUM_WORKERS,shuffle=False,pin_memory=True)\n    mp = Model_pred(models,dl)\n    \n    #generate masks\n    mask = torch.zeros(img.shape[0],sz*reduce,sz*reduce,dtype=torch.int8)\n    for i,p in zip(idxs,iter(mp)): mask[i] = p.squeeze(-1) > TH\n    \n    #reshape tiled masks into a single mask and crop padding\n    mask = mask.view(img_shape_p[0]//sz,img_shape_p[1]//sz,sz*reduce,sz*reduce).\\\n        permute(0,2,1,3).reshape(img_shape_p[0]*reduce,img_shape_p[1]*reduce)\n    mask = mask[pad0//2:-(pad0-pad0//2) if pad0 > 0 else img_shape_p[0]*reduce,\n        pad1//2:-(pad1-pad1//2) if pad1 > 0 else img_shape_p[1]*reduce]\n    \n    #convert to rle\n    #https://www.kaggle.com/bguberfain/memory-aware-rle-encoding\n    rle = rle_encode_less_memory(mask.numpy())\n    names.append(idx)\n    preds.append(rle)\n    gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-11-29T04:04:29.489161Z","iopub.status.busy":"2020-11-29T04:04:29.488492Z","iopub.status.idle":"2020-11-29T04:04:30.76147Z","shell.execute_reply":"2020-11-29T04:04:30.760627Z"},"papermill":{"duration":1.363307,"end_time":"2020-11-29T04:04:30.761583","exception":false,"start_time":"2020-11-29T04:04:29.398276","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"df = pd.DataFrame({'id':names,'predicted':preds})\ndf.to_csv('submission.csv',index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}