{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Train on Crops - Predict on Full - CV 0.60+\nIn this kernel we use a trick that we learned in Kaggle's Steel Competition. Since segmentation neural networks are all convolutions, you can train with one input size and predict with a different input size (explained in detail [here][1]). First we will resize all training and test images into size `700x1050` from their original `1400x2100`. Next we will train with random `352x512` crops and then feed full test images (`700x1050` images) into our network and get full segmentation masks!\n\n[1]: https://www.kaggle.com/c/severstal-steel-defect-detection/discussion/114321\n"},{"metadata":{},"cell_type":"markdown","source":"# Load Data\nKaggle has recently upgraded to TensorFlow 2.0. This is causing memory issues, so we will install TensorFlow 1.14 here. Next we will load and restructure the `train.csv` dataframe."},{"metadata":{"_kg_hide-input":false,"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"!pip install tensorflow-gpu==1.14.0\n!pip install keras==2.2.4","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":false},"cell_type":"code","source":"import pandas as pd, numpy as np, os\nfrom PIL import Image \nimport cv2, keras, gc\nimport keras.backend as K\nfrom keras import layers\nfrom keras.models import Model\nfrom keras.models import load_model\nfrom keras.callbacks import LearningRateScheduler\nfrom sklearn.model_selection import KFold\nimport matplotlib.pyplot as plt, time\nfrom sklearn.metrics import roc_auc_score, accuracy_score\n#os.listdir('../input/understanding_cloud_organization/')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"sub = pd.read_csv('../input/understanding_cloud_organization/sample_submission.csv')\nsub['Image'] = sub['Image_Label'].map(lambda x: x.split('.')[0])\n\nPATH = '../input/understanding_cloud_organization/train_images/'\ntrain = pd.read_csv('../input/understanding_cloud_organization/train.csv')\ntrain['Image'] = train['Image_Label'].map(lambda x: x.split('.')[0])\ntrain['Label'] = train['Image_Label'].map(lambda x: x.split('_')[1])\ntrain2 = pd.DataFrame({'Image':train['Image'][::4]})\ntrain2['e1'] = train['EncodedPixels'][::4].values\ntrain2['e2'] = train['EncodedPixels'][1::4].values\ntrain2['e3'] = train['EncodedPixels'][2::4].values\ntrain2['e4'] = train['EncodedPixels'][3::4].values\ntrain2.set_index('Image',inplace=True,drop=True)\ntrain2.fillna('',inplace=True); train2.head()\ntrain2[['d1','d2','d3','d4']] = (train2[['e1','e2','e3','e4']]!='').astype('int8')\ntrain2.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Helper Functions\nBelow are functions to manipulate `rle` masks. Click \"code\" to the right to see the code."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def mask2rleX(img0, shape=(1050,700), shrink=2):\n    # USAGE: embeds into size shape, then shrinks, then outputs rle\n    # EXAMPLE: img0 can be 600x1000. It will center load into\n    # a mask of 700x1050 then the mask is downsampled to 350x525\n    # finally the rle is outputted. \n    a = (shape[1]-img0.shape[0])//2\n    b = (shape[0]-img0.shape[1])//2\n    img = np.zeros((shape[1],shape[0]))\n    img[a:a+img0.shape[0],b:b+img0.shape[1]] = img0\n    img = img[::shrink,::shrink]\n    \n    pixels= img.T.flatten()\n    pixels = np.concatenate([[0], pixels, [0]])\n    runs = np.where(pixels[1:] != pixels[:-1])[0] + 1\n    runs[1::2] -= runs[::2]\n    return ' '.join(str(x) for x in runs)\n\ndef rle2maskX(mask_rle, shape=(2100,1400), shrink=1):\n    # Converts rle to mask size shape then downsamples by shrink\n    s = mask_rle.split()\n    starts, lengths = [np.asarray(x, dtype=int) for x in (s[0:][::2], s[1:][::2])]\n    starts -= 1\n    ends = starts + lengths\n    img = np.zeros(shape[0]*shape[1], dtype=np.uint8)\n    for lo, hi in zip(starts, ends):\n        img[lo:hi] = 1\n    return img.reshape(shape).T[::shrink,::shrink]\n\ndef mask2contour(mask, width=5):\n    w = mask.shape[1]\n    h = mask.shape[0]\n    mask2 = np.concatenate([mask[:,width:],np.zeros((h,width))],axis=1)\n    mask2 = np.logical_xor(mask,mask2)\n    mask3 = np.concatenate([mask[width:,:],np.zeros((width,w))],axis=0)\n    mask3 = np.logical_xor(mask,mask3)\n    return np.logical_or(mask2,mask3) \n\ndef clean(rle,sz=20000):\n    if rle=='': return ''\n    mask = rle2maskX(rle,shape=(525,350))\n    num_component, component = cv2.connectedComponents(np.uint8(mask))\n    mask2 = np.zeros((350,525))\n    for i in range(1,num_component):\n        y = (component==i)\n        if np.sum(y)>=sz: mask2 += y\n    return mask2rleX(mask2,shape=(525,350), shrink=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data Generator\nThis data generator outputs random crops of size `352x512`. These crops are taken from the original `1400x2100` training images after they are resized to `700x1050` pixels. The masks are cropped to match the image crops. Also we have horizontal and vertical flip augmentation.\n\nBelow we display examples. The image on the left is the original image. The yellow rectangle is an original mask. The black rectangle is a random crop. The image on the right is the random crop outputted by the data generator. Notice how the original mask is cropped too."},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"class DataGenerator(keras.utils.Sequence):\n    # USES GLOBAL VARIABLE TRAIN2 COLUMNS E1, E2, E3, E4\n    'Generates data for Keras'\n    def __init__(self, list_IDs, batch_size=8, shuffle=False, width=512, height=352, scale=1/128., sub=1., mode='train_seg',\n                 path='../input/understanding_cloud_organization/train_images/', ext='.jpg', flips=False, shrink=2):\n        'Initialization'\n        self.list_IDs = list_IDs\n        self.shuffle = shuffle\n        self.batch_size = batch_size\n        self.path = path\n        self.scale = scale\n        self.sub = sub\n        self.path = path\n        self.ext = ext\n        self.width = width\n        self.height = height\n        self.mode = mode\n        self.flips = flips\n        self.shrink = shrink\n        self.on_epoch_end()\n        \n    def __len__(self):\n        'Denotes the number of batches per epoch'\n        ct = int(np.floor( len(self.list_IDs) / self.batch_size))\n        if len(self.list_IDs)>ct*self.batch_size: ct += 1\n        return int(ct)\n\n    def __getitem__(self, index):\n        'Generate one batch of data'\n        indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]\n        X, y, msk, crp = self.__data_generation(indexes)\n        if (self.mode=='display'): return X, msk, crp\n        elif (self.mode=='train_seg')|(self.mode=='validate_seg'): return X, msk\n        elif (self.mode=='train')|(self.mode=='validate'): return X, y\n        else: return X\n\n    def on_epoch_end(self):\n        'Updates indexes after each epoch'\n        self.indexes = np.arange(int( len(self.list_IDs) ))\n        if self.shuffle: np.random.shuffle(self.indexes)\n\n    def __data_generation(self, indexes):\n        'Generates data containing batch_size samples' \n        # Initialization\n        lnn = len(indexes)\n        X = np.empty((lnn,self.height,self.width,3),dtype=np.float32)\n        msk = np.empty((lnn,self.height,self.width,4),dtype=np.int8)\n        crp = np.zeros((lnn,2),dtype=np.int16)\n        y = np.zeros((lnn,4),dtype=np.int8)\n        \n        # Generate data\n        for k in range(lnn):\n            img = cv2.imread(self.path + self.list_IDs[indexes[k]] + self.ext)\n            img = cv2.resize(img,(2100//self.shrink,1400//self.shrink),interpolation = cv2.INTER_AREA)\n            # AUGMENTATION FLIPS\n            hflip = False; vflip = False\n            if (self.flips):\n                if np.random.uniform(0,1)>0.5: hflip=True\n                if np.random.uniform(0,1)>0.5: vflip=True\n            if vflip: img = cv2.flip(img,0) # vertical\n            if hflip: img = cv2.flip(img,1) # horizontal\n            # RANDOM CROP\n            a = np.random.randint(0,2100//self.shrink-self.width+1)\n            b = np.random.randint(0,1400//self.shrink-self.height+1)\n            if (self.mode=='predict'):\n                a = (2100//self.shrink-self.width)//2\n                b = (1400//self.shrink-self.height)//2\n            img = img[b:self.height+b,a:self.width+a]\n            # NORMALIZE IMAGES\n            X[k,] = img*self.scale - self.sub      \n            # LABELS\n            if (self.mode!='predict'):\n                for j in range(1,5):\n                    rle = train2.loc[self.list_IDs[indexes[k]],'e'+str(j)]\n                    mask = rle2maskX(rle,shrink=self.shrink)\n                    if vflip: mask = np.flip(mask,axis=0)\n                    if hflip: mask = np.flip(mask,axis=1)\n                    msk[k,:,:,j-1] = mask[b:self.height+b,a:self.width+a]\n                    if (self.mode=='train')|(self.mode=='validate'):\n                        if np.sum( msk[k,:,:,j-1] )>0: y[k,j-1]=1\n            if (self.mode=='display'):\n                crp[k,0] = a; crp[k,1] = b\n\n        return X, y, msk, crp","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"types = ['Fish','Flower','Gravel','Sugar']\ntrain_batch = DataGenerator(train2.index[:8], mode='display',batch_size=1,scale=1,sub=0)\nfor k,image in enumerate(train_batch):\n    plt.figure(figsize=(15,8))\n    \n    # RANDOMLY PICK CLOUD TYPE TO DISPLAY FROM NON-EMPTY MASKS\n    idx = np.argwhere( train2.loc[train2.index[k],['d1','d2','d3','d4']].values==1 ).flatten()\n    d = np.random.choice(idx)+1\n    \n    # DISPLAY ORIGINAL\n    img = Image.open(PATH+train2.index[k]+'.jpg'); img=np.array(img)\n    mask = rle2maskX( train2.loc[train2.index[k],'e'+str(d)] )\n    contour = mask2contour( mask,10 )\n    img[contour==1,:2]=255\n    diff = np.ones((1400,2100,3),dtype=np.int)*255-img.astype(int)\n    img=img.astype(int); img[mask==1,:] += diff[mask==1,:]//6\n    mask = np.zeros((1400,2100))\n    a = image[2][0,1]*2\n    b = image[2][0,0]*2\n    mask[a:a+2*352,b:b+2*512]=1\n    mask = mask2contour(mask,20)\n    img[mask==1,:]=0\n    plt.subplot(1,2,1); \n    plt.title('Original - '+train2.index[k]+'.jpg - '+types[d-1])\n    plt.imshow(img);\n    \n    # DISPLAY RANDOM CROP\n    img = image[0][0,]\n    mask = image[1][0,:,:,d-1]\n    contour = mask2contour( mask )\n    img[contour==1,:2]=255\n    diff = np.ones((352,512,3),dtype=np.int)*255-img.astype(int)\n    img=img.astype(int); img[mask==1,:] += diff[mask==1,:]//6\n    plt.subplot(1,2,2)\n    plt.title('Training Crop - '+train2.index[k]+'.jpg - '+types[d-1])\n    plt.imshow( img.astype(int) ); \n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Build Segmentation Model\nWe will build a segmentation model using Qubvel's Keras Segmentation models [here][1]. Our architecture will be FPN (feature pyramid network) and our backbone will be Efficientnetb2. We will use Jaccard loss and Adam optimizer with learning rate 1e-4. Our metric will be Dice coef.\n\n[1]: https://github.com/qubvel/segmentation_models"},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"! pip install segmentation-models\n\nfrom segmentation_models import Unet,FPN\nfrom segmentation_models.losses import bce_jaccard_loss, jaccard_loss\nfrom keras.optimizers import Adam\n\ndef build_model():\n    #model = Unet('resnet34', input_shape=(None,None,3), classes=4, activation='sigmoid')\n    model = FPN('efficientnetb2', input_shape=(None, None, 3), classes=4, activation='sigmoid')\n    #model = FPN('inceptionv3', input_shape=(None, None, 3), classes=4, activation='sigmoid')\n\n    #model.compile(optimizer=Adam(lr=0.001), loss='binary_crossentropy', metrics=[dice_coef])\n    #model.compile(optimizer=Adam(lr=0.0001), loss=bce_jaccard_loss, metrics=[dice_coef])\n    model.compile(optimizer=Adam(lr=0.0001), loss=jaccard_loss, metrics=[dice_coef])\n    #model.compile(optimizer=Adam(lr=0.0001), loss=dice_coef_loss, metrics=[dice_coef])\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras import backend as K\n\ndef dice_coef_loss(y_true, y_pred):\n    return 1 - dice_coef(y_true, y_pred)\n\ndef dice_coef(y_true, y_pred, smooth=1):\n    y_true_f = K.flatten(y_true)\n    y_pred_f = K.flatten(y_pred)\n    intersection = K.sum(y_true_f * y_pred_f)\n    return (2. * intersection + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Train Segmentation Model on Crops\nWe will train with `352x512` crops, `batch_size=8`, and use 3-Fold validation. During training we predict OOF. We predict `test.csv` afterwards by saving the 3 models from the 3 folds. We will train each fold for 4 epochs. In our predicted oof masks, all pixel probabilities over 0.4 will be converted to 1 and less than 0.4 to 0. Any mask with fewer than `4*20000` pixels (predicted on `700x1050` image) will be regarded as no mask."},{"metadata":{"trusted":true},"cell_type":"code","source":"oof = np.empty_like(train2[['e1','e2','e3','e4']].values)\n\n# K-FOLD MODELS\nskf = KFold(n_splits=3, shuffle=True, random_state=42)\nfor k, (idxT, idxV) in enumerate( skf.split(train2) ):\n        \n    # TRAIN MODEL\n    print(); print('#'*10,'FOLD',k,'#'*10)\n    print('Train on',len(idxT),'Validate on',len(idxV))\n    model = build_model()        \n    train_gen = DataGenerator(train2.index[idxT],flips=True, shuffle=True)\n    val_gen = DataGenerator(train2.index[idxV])\n    h = model.fit_generator(train_gen, epochs = 4, verbose=2, validation_data = val_gen)\n        \n    # PREDICT OOF\n    print('Predict OOF: ',end='')\n    oof_gen = DataGenerator(train2.index[idxV], width=1024, height=672, batch_size=2, mode='predict')\n    for b,batch in enumerate(oof_gen):\n        btc = model.predict_on_batch(batch)\n        for j in range(btc.shape[0]):\n            for i in range(btc.shape[-1]):\n                mask = (btc[j,:,:,i]>0.4).astype(int); rle =''\n                if np.sum(mask)>4*20000: rle = mask2rleX( mask )\n                oof[idxV[2*b+j],i] = rle\n        if b%50==0: print(2*b,', ',end='')\n        \n    # SAVE MODEL AND FREE GPU MEMORY \n    model.save('Seg_'+str(k)+'.h5', overwrite=True)\n    del train_gen, val_gen, oof_gen, model, h, idxT, idxV, btc, batch, b\n    K.clear_session(); x=gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Evaluate Segmentation Model using OOF\nWe will evaluate OOF using Kaggle's Dice metric. For post processing, we will remove any contiguous piece of predicted mask with fewer than 20000 pixels (predicted on `350x525` image). And we will remove any mask with less than 0.5 probability as determined by our cloud classifer from our previous notebook [here][1]\n\n[1]: https://www.kaggle.com/cdeotte/cloud-bounding-boxes-cv-0-58"},{"metadata":{"trusted":true},"cell_type":"code","source":"def dice_coef6(y_true_rle, y_pred_prob, y_pred_rle, th):\n    if y_pred_prob<th:\n        if y_true_rle=='': return 1\n        else: return 0\n    else:\n        y_true_f = rle2maskX(y_true_rle,shrink=4)\n        y_pred_f = rle2maskX(y_pred_rle,shape=(525,350))\n        union = np.sum(y_true_f) + np.sum(y_pred_f)\n        if union==0: return 1\n        intersection = np.sum(y_true_f * y_pred_f)\n        return 2. * intersection / union","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# LOAD CLASSIFICATION PREDICTIONS FROM PREVIOUS KERNEL\n# https://www.kaggle.com/cdeotte/cloud-bounding-boxes-cv-0-58\nfor k in range(1,5): train2['o'+str(k)] = 0\ntrain2[['o1','o2','o3','o4']] = np.load('../input/cloudpred1/oof.npy')[:len(train2),]\n\n# LOAD OOF SEGMENTATION PREDICTIONS FROM 3-FOLD ABOVE\nfor k in range(1,5): train2['ee'+str(k)] = ''\ntrain2[['ee1','ee2','ee3','ee4']] = oof\nfor k in range(1,5): train2['ee'+str(k)] = train2['ee'+str(k)].map(clean)\n\n# COMPUTE KAGGLE DICE\nth = [0.5,0.5,0.5,0.5]\nfor k in range(1,5):\n    train2['ss'+str(k)] = train2.apply(lambda x:dice_coef6(x['e'+str(k)],x['o'+str(k)],x['ee'+str(k)],th[k-1]),axis=1)\n    dice = np.round( train2['ss'+str(k)].mean(),3 )\n    print(types[k-1],': Kaggle Dice =',dice)\ndice = np.round( np.mean( train2[['ss1','ss2','ss3','ss4']].values ),3 )\nprint('Overall : Kaggle Dice =',dice)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# View OOF Examples\nBelow yellow outlines are true masks and blue outlines (with shaded insides) are predicted masks. The Dice score for each predicted mask is displayed above each image."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"for d in range(1,5):\n    print('#'*27); print('#'*5,types[d-1].upper(),'CLOUDS','#'*7); print('#'*27)\n    plt.figure(figsize=(20,15)); k=0\n    for kk in range(9):\n        plt.subplot(3,3,kk+1)\n        while (train2.loc[train2.index[k],'e'+str(d)]==''): k += 1\n        f = train2.index[k]+'.jpg'\n        img = Image.open(PATH+f); img = img.resize((525,350)); img = np.array(img)\n        rle1 = train2.loc[train2.index[k],'e'+str(d)]; mask = rle2maskX(rle1,shrink=4)\n        contour = mask2contour(mask,5); img[contour==1,:2] = 255\n        rle2 = train2.loc[train2.index[k],'ee'+str(d)]; mask = rle2maskX(rle2,shape=(525,350))\n        contour = mask2contour(mask,5); img[contour==1,2] = 255\n        diff = np.ones((350,525,3),dtype=np.int)*255-img\n        img=img.astype(int); img[mask==1,:] += diff[mask==1,:]//4\n        dice = np.round( dice_coef6(rle1,1,rle2,0),3 )\n        plt.title(f+'  Dice = '+str(dice)+'   Yellow true, Blue predicted')\n        plt.imshow(img); k += 1\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Predict Test Images\nWe load the models from our 3 folds above and use them to predict `test.csv`. Note that we must choose an input size that is divisble by 32. Therefore we choose `672x1024` instead of `700x1050`. We segment the middle of each test image ignoring the 14 pixel wide border around the edge."},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.models import load_model\nmodel1 = load_model('Seg_0.h5',custom_objects={'dice_coef':dice_coef,'jaccard_loss':jaccard_loss})\nmodel2 = load_model('Seg_1.h5',custom_objects={'dice_coef':dice_coef,'jaccard_loss':jaccard_loss})\nmodel3 = load_model('Seg_2.h5',custom_objects={'dice_coef':dice_coef,'jaccard_loss':jaccard_loss})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Computing masks for',len(sub)//4,'test images with 3 models'); sub.EncodedPixels = ''\nPTH = '../input/understanding_cloud_organization/test_images/'\ntest_gen = DataGenerator(sub.Image[::4].values, width=1024, height=672, batch_size=2, mode='predict',path=PTH)\n\nfor b,batch in enumerate(test_gen):\n    btc = model1.predict_on_batch(batch)\n    btc += model2.predict_on_batch(batch)\n    btc += model3.predict_on_batch(batch)\n    btc /= 3.0\n    for j in range(btc.shape[0]):\n        for i in range(btc.shape[-1]):\n            mask = (btc[j,:,:,i]>0.4).astype(int); rle = ''\n            if np.sum(mask)>4*20000: rle = mask2rleX( mask )\n            sub.iloc[4*(2*b+j)+i,1] = rle\n    if b%50==0: print(b*2,', ',end='')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Submit to Kaggle\nUsing our classification predictions from our previous notebook [here][1], we will remove any segmentation mask with probability less than 0.5\n\n[1]: https://www.kaggle.com/cdeotte/cloud-bounding-boxes-cv-0-58"},{"metadata":{"trusted":true},"cell_type":"code","source":"# LOAD CLASSIFICATION PREDICTIONS FROM PREVIOUS KERNEL\n# https://www.kaggle.com/cdeotte/cloud-bounding-boxes-cv-0-58\nsub['p'] = np.load('../input/cloudpred1/preds.npy').reshape((-1))[:len(sub)]\nsub.loc[sub.p<0.5,'EncodedPixels'] = ''\n\nsub.EncodedPixels = sub.EncodedPixels.map(clean)\nsub[['Image_Label','EncodedPixels']].to_csv('submission.csv',index=False)\nsub.head(25)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}