{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Cloud Bounding Boxes - CV 0.58\nIn this kernel we will attempt to create bounding boxes instead of segmentation. Currently the CV is a low 0.58 but perhaps this kernel can be improved to score higher.\n\nThere is a variable `TRAIN_MODELS` below. If you set this variable to `False` then you can load `oof` and `preds` from offline training and predicting. Do this to save your GPU quota. If you want to see the models train and predict, then turn GPU on and set variable `TRAIN_MODELS` to `True`."},{"metadata":{},"cell_type":"markdown","source":"# Load Data\nWe will load `train.csv` and restructure it to work more efficiently with our data generator"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import pandas as pd, numpy as np, os\nfrom PIL import Image \nimport cv2, keras, gc\nimport keras.backend as K\nfrom keras import layers\nfrom keras.models import Model\nfrom keras.models import load_model\nfrom keras.callbacks import LearningRateScheduler\nfrom sklearn.model_selection import KFold\nimport matplotlib.pyplot as plt, time\nfrom sklearn.metrics import roc_auc_score, accuracy_score","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"TRAIN_MODELS = False\n\nsub = pd.read_csv('../input/understanding_cloud_organization/sample_submission.csv')\nsub['Image'] = sub['Image_Label'].map(lambda x: x.split('.')[0])\n\ntrain = pd.read_csv('../input/understanding_cloud_organization/train.csv')\ntrain['Image'] = train['Image_Label'].map(lambda x: x.split('.')[0])\ntrain['Label'] = train['Image_Label'].map(lambda x: x.split('_')[1])\ntrain2 = pd.DataFrame({'Image':train['Image'][::4]})\ntrain2['e1'] = train['EncodedPixels'][::4].values\ntrain2['e2'] = train['EncodedPixels'][1::4].values\ntrain2['e3'] = train['EncodedPixels'][2::4].values\ntrain2['e4'] = train['EncodedPixels'][3::4].values\ntrain2.set_index('Image',inplace=True,drop=True)\ntrain2.fillna('',inplace=True); train2.head()\ntrain2[['d1','d2','d3','d4']] = (train2[['e1','e2','e3','e4']]!='').astype('int8')\ntrain2.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data Generator\nThis data generator outputs resized full images when `mode = train` with class labels or with bounding boxes when `mode = train_bb`. Additionally there are parameters to apply data augmentation flips and shake. Also images can be normalized with parameters `scale` and `sub` such that `output = img * scale - sub`."},{"metadata":{"trusted":true},"cell_type":"code","source":"class DataGenerator(keras.utils.Sequence):\n # USES GLOBAL VARIABLE TRAIN2 COLUMNS E1, E2, E3, E4\n 'Generates data for Keras'\n def __init__(self, list_IDs, batch_size=8, shuffle=False, width=525, height=350, scale=1/128., sub=1., mode='train',\n path='../input/understanding_cloud_organization/train_images/', ext='.jpg', shake=0, flips=False, dft=1):\n 'Initialization'\n self.list_IDs = list_IDs\n self.shuffle = shuffle\n self.batch_size = batch_size\n self.path = path\n self.scale = scale\n self.sub = sub\n self.path = path\n self.ext = ext\n self.width = width\n self.height = height\n self.mode = mode\n self.shake = shake\n self.flips = flips\n self.dft = dft\n self.on_epoch_end()\n if (mode=='train_bb')&((width!=525)|(height!=350)|(shake>0)):\n print('ERROR: wrong size or shake for train_bb')\n \n def __len__(self):\n 'Denotes the number of batches per epoch'\n ct = int(np.floor( len(self.list_IDs) / self.batch_size))\n if len(self.list_IDs)>ct*self.batch_size: ct += 1\n return int(ct)\n\n def __getitem__(self, index):\n 'Generate one batch of data'\n indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]\n X, y, bb = self.__data_generation(indexes)\n if (self.mode=='train')|(self.mode=='validate'): return X, y\n elif self.mode=='train_bb': return X, bb\n else: return X\n\n def on_epoch_end(self):\n 'Updates indexes after each epoch'\n self.indexes = np.arange(int( len(self.list_IDs) ))\n if self.shuffle: np.random.shuffle(self.indexes)\n\n def __data_generation(self, indexes):\n 'Generates data containing batch_size samples' \n # Initialization\n lnn = len(indexes)\n X = np.empty((lnn,self.height-self.shake,self.width-self.shake,3),dtype=np.float32)\n y = np.zeros((lnn,4),dtype=np.int8)\n bb = np.zeros((lnn,4),dtype=np.int16)\n \n # Generate data\n for k in range(lnn):\n img = cv2.imread(self.path + self.list_IDs[indexes[k]] + self.ext)\n img = cv2.resize(img,(self.width,self.height),interpolation = cv2.INTER_AREA)\n # AUGMENTATION FLIPS\n hflip = False; vflip = False\n if (self.flips):\n if np.random.uniform(0,1)>0.5: hflip=True\n if np.random.uniform(0,1)>0.5: vflip=True\n if vflip: img = cv2.flip(img,0) # vertical\n if hflip: img = cv2.flip(img,1) # horizontal\n # AUGMENTATION SHAKE\n a = 0; b = 0\n if self.shake>0:\n a = np.random.randint(0,self.shake+1)\n b = np.random.randint(0,self.shake+1)\n if (self.mode=='predict')|(self.mode=='validate'):\n a = self.shake//2; b = self.shake//2\n img = img[b:(self.height-self.shake)+b,a:(self.width-self.shake)+a]\n # NORMALIZE IMAGES\n X[k,] = img*self.scale - self.sub \n # LABELS\n if (self.mode=='train')|(self.mode=='validate'):\n y[k,] = train2.loc[self.list_IDs[indexes[k]],['d1','d2','d3','d4']].values\n # BOUNDING BOXES\n if (self.mode=='train_bb'):\n bb[k,] = train2.loc[self.list_IDs[indexes[k]],'b'+str(self.dft)]\n bb[k,] = bb[k,]//4\n if vflip: bb[k,0] = (350-bb[k,0])-bb[k,2]\n if hflip: bb[k,1] = (525-bb[k,1])-bb[k,3]\n \n return X, y, bb","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Build Classifier Model\nWill will use pretrained Xception model with 4 `sigmoid` outputs and `binary_crossentropy` loss to classify our cloud images."},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras import applications\n#! pip install keras_efficientnets\n#from keras_efficientnets import EfficientNetB4\n\ndef build_model():\n base_model = applications.Xception(weights='imagenet', input_shape=(350-32, 525-32, 3), include_top=False)\n #base_model = EfficientNetB4(input_shape=(350-32, 525-32, 3), include_top=False, weights='imagenet')\n base_model.trainable = False\n\n x = base_model.output\n x = layers.Flatten()(x)\n x = layers.Dense(256, activation=\"relu\")(x)\n #x = layers.Dropout(0.4)(x)\n x = layers.Dense(256, activation=\"relu\")(x)\n #x = layers.Dropout(0.4)(x)\n\n pred = layers.Dense(4, activation=\"sigmoid\")(x)\n model = Model(inputs = base_model.input, outputs = pred)\n model.compile(loss='binary_crossentropy', optimizer = \"adam\", metrics=['accuracy'])\n \n return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Train Classifier Model\nWe will training our Xception model on 3 folds using resized full images and data augmentation. Augmentation includes horizontal flip, vertical flip, and shake (random crops that are slighly smaller than full size)."},{"metadata":{"trusted":true},"cell_type":"code","source":"if TRAIN_MODELS:\n oof = np.zeros((train2.shape[0],4))\n preds = np.zeros((sub.Image.values[::4].shape[0],4))\n\n skf = KFold(n_splits=3, shuffle=True, random_state=42)\n for k, (idxT, idxV) in enumerate( skf.split(train2) ):\n\n model = build_model()\n train_gen = DataGenerator(train2.index[idxT],shake=32,flips=True,shuffle=True)\n val_gen = DataGenerator(train2.index[idxV],shake=32,mode='validate')\n \n print()\n print('#'*10,'FOLD',k,'#'*10)\n print('#'*10,'TRAIN','#'*10)\n h = model.fit_generator(train_gen, epochs = 3, verbose=1, validation_data = val_gen)\n \n print('#'*10,'PREDICT','#'*10)\n test_gen = DataGenerator(sub.Image.values[::4], mode='predict', batch_size=8,\n path='../input/understanding_cloud_organization/test_images/', shake=32)\n preds += model.predict_generator(test_gen, verbose=1)\n oof[idxV,] = model.predict_generator(val_gen, verbose=1)\n \n # FREE GPU MEMORY (BEING EXTRA CAREFUL HERE)\n del train_gen, val_gen, test_gen, model, h, idxT, idxV\n K.clear_session(); x=gc.collect()\n \n preds /= skf.n_splits\nelse:\n oof = np.load('../input/cloudpred1/oof.npy')\n preds = np.load('../input/cloudpred1/preds.npy')\n print('Saving time by loading classification OOF and Preds')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Evaluate Classifier Model using OOF\nBelow shows the AUC and ACC for each type of prediction"},{"metadata":{"trusted":true},"cell_type":"code","source":"type = ['Fish','Flower','Gravel','Sugar']\nfor k in range(1,5): train2['o'+str(k)] = 0\ntrain2[['o1','o2','o3','o4']] = oof\n\nfor k in range(1,5):\n print(type[k-1],': ',end='')\n auc = np.round( roc_auc_score(train2['d'+str(k)].values,train2['o'+str(k)].values ),3 )\n acc = np.round( accuracy_score(train2['d'+str(k)].values,(train2['o'+str(k)].values>0.5).astype(int) ),3 )\n print('AUC =',auc,end='')\n print(', ACC =',acc) \nprint('OVERALL: ',end='')\nauc = np.round( roc_auc_score(train2[['d1','d2','d3','d4']].values.reshape((-1)),train2[['o1','o2','o3','o4']].values.reshape((-1)) ),3 )\nacc = np.round( accuracy_score(train2[['d1','d2','d3','d4']].values.reshape((-1)),(train2[['o1','o2','o3','o4']].values>0.5).astype(int).reshape((-1)) ),3 )\nprint('AUC =',auc, end='')\nprint(', ACC =',acc) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Create Bounding Boxes\nWe will naively create bounding boxes by finding the smallest bounding box that includes the entire given mask. Afterwards we compute the dice score of our created bounding box compared to original mask. Later we use this dice score to exclude any bounding box with dice less than 0.8 from our training."},{"metadata":{"trusted":true},"cell_type":"code","source":"def rle2mask(mask_rle, shape=(2100,1400), shrink=1):\n s = mask_rle.split()\n starts, lengths = [np.asarray(x, dtype=int) for x in (s[0:][::2], s[1:][::2])]\n starts -= 1\n ends = starts + lengths\n img = np.zeros(shape[0]*shape[1], dtype=np.uint8)\n for lo, hi in zip(starts, ends):\n img[lo:hi] = 1\n return img.reshape(shape).T[::shrink,::shrink]\n\ndef rle2bb(rle):\n if rle=='': return (0,0,0,0)\n mask = rle2mask(rle)\n z = np.argwhere(mask==1)\n mn_x = np.min( z[:,0] )\n mx_x = np.max( z[:,0] )\n mn_y = np.min( z[:,1] )\n mx_y = np.max( z[:,1] )\n return (mn_x,mn_y,mx_x-mn_x,mx_y-mn_y)\n\ndef bb2dice(rle,bb,shape=(2100,1400)):\n mask1 = rle2mask(rle,shape)\n mask2 = np.zeros((shape[1],shape[0]))\n mask2[bb[0]:bb[0]+bb[2],bb[1]:bb[1]+bb[3]]=1\n union = np.sum(mask1) + np.sum(mask2)\n if union==0: return 1\n intersection = np.sum(mask1*mask2)\n return 2.*intersection/union","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if TRAIN_MODELS:\n for k in range(1,5):\n print('Computing bb',k,'...',end='')\n train2['b'+str(k)] = train2['e'+str(k)].map(rle2bb)\n print('Computing dice_bb',k,'...',end='')\n train2['s'+str(k)] = train2.apply(lambda x: bb2dice(x['e'+str(k)],x['b'+str(k)]),axis=1)\n print('Done')\n train2.head()\nelse:\n for k in range(1,5):\n train2['b'+str(k)] = np.load('../input/cloudpred1/bb'+str(k)+'.npy',allow_pickle=True)[:,0]\n train2['s'+str(k)] = np.load('../input/cloudpred1/bb'+str(k)+'.npy',allow_pickle=True)[:,1]\n print('Saving time by loading computed bounding boxes')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Build Bounding Box Model\nWe will use pretrained Xception model with 4 linear outputs and `mean_squared_error` loss to model bounding boxes. Note that a bounding box is 4 numbers `(y, x, height, width)` where `(y,x)` is the coordinate of the top left corner."},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras import applications\n#! pip install keras_efficientnets\n#from keras_efficientnets import EfficientNetB4\n\ndef build_model():\n base_model = applications.Xception(weights='imagenet', input_shape=(350, 525, 3), include_top=False)\n #base_model = EfficientNetB4(input_shape=(350-32, 525-32, 3), include_top=False, weights='imagenet')\n base_model.trainable = False\n\n x = base_model.output\n x = layers.Flatten()(x)\n x = layers.Dense(256, activation=\"relu\")(x)\n #x = layers.Dropout(0.4)(x)\n x = layers.Dense(256, activation=\"relu\")(x)\n #x = layers.Dropout(0.4)(x)\n\n pred = layers.Dense(4, activation=\"linear\")(x)\n model = Model(inputs = base_model.input, outputs = pred)\n model.compile(loss='mean_squared_error', optimizer = \"adam\")\n \n return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Train Bounding Box Model\nWe will train 4 separate models for each cloud type. We will only train with images that have a bounding box that has dice score of 0.8 or larger when compared with the original masks. Furthermore we will only use images that our classification model above predicted a probability more than 0.5 for that class."},{"metadata":{"trusted":true},"cell_type":"code","source":"type2 = ['FISH','FLOWER','GRAVEL','SUGAR']\nif TRAIN_MODELS:\n oof_bb = np.zeros((train2.shape[0],4,4))\n preds_bb = np.zeros((sub.Image.values[::4].shape[0],4,4))\n\n skf = KFold(n_splits=3, shuffle=True, random_state=42)\n for k, (idxT, idxV) in enumerate( skf.split(train2) ):\n print(); print('#'*28)\n print('#'*10,'FOLD',k,'#'*10)\n print('#'*28)\n for j in [1,2,3,4]:\n model = build_model()\n one = train2[ (train2.index.isin(train2.index[idxT]))&(train2['d'+str(j)]==1)&(train2['s'+str(j)]>0.8)&(train2['o'+str(j)]>0.5) ]\n two = train2[ (train2.index.isin(train2.index[idxV]))&(train2['d'+str(j)]==1)&(train2['s'+str(j)]>0.8)&(train2['o'+str(j)]>0.5) ]\n train_gen = DataGenerator(one.index,mode='train_bb',shuffle=True,flips=True,dft=j)\n val_gen = DataGenerator(two.index,mode='train_bb',dft=j)\n \n print(); print('#'*10,'TRAIN CLOUD',type2[j-1],' (fold',str(k)+')','#'*10)\n print(' train on',len(one),', validate on',len(two))\n h = model.fit_generator(train_gen, epochs = 5, verbose=1, validation_data = val_gen)\n\n \n print('#'*10,'PREDICT OOF AND TEST','#'*10)\n test_gen = DataGenerator(sub.Image.values[::4], mode='predict', batch_size=8,\n path='test_images/')\n val_gen2 = DataGenerator(train2.index[idxV],mode='predict')\n preds_bb[:,:,j-1] += model.predict_generator(test_gen, verbose=1)\n oof_bb[idxV,:,j-1] = model.predict_generator(val_gen2, verbose=1)\n \n # FREE GPU MEMORY\n del train_gen, val_gen, val_gen2, test_gen, model, h, annealer, one, two\n K.clear_session(); x=gc.collect()\n \n preds_bb /= skf.n_splits\nelse:\n oof_bb = np.load('../input/cloudpred1/oof_bb.npy')\n preds_bb = np.load('../input/cloudpred1/preds_bb.npy')\n print('Saving time by loading Bounding Box OOF and Preds')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Evaluate Bounding Box Model using OOF\nUsing Kaggle's competition metric, we will compute the dice score of our predicted OOF bounding boxes (masks). Our bounding box model makes a bounding box for every image. Therefore using our classification model, we will predict no bounding box (i.e. empty mask) if we predicted less than 0.65 probabililty that a certain cloud type exists."},{"metadata":{"trusted":true},"cell_type":"code","source":"def bb2rle(b,shape=(525,350)):\n bb = [0,0,0,0]\n bb[0] = np.min(( np.max((int(b[0]),0)),shape[1] ))\n bb[1] = np.min(( np.max((int(b[1]),0)),shape[0] ))\n bb[2] = np.min(( np.max((int(b[2]),0)),shape[1]-bb[0] ))\n bb[3] = np.min(( np.max((int(b[3]),0)),shape[0]-bb[1] ))\n z = np.ones((bb[3]*2))*bb[2]\n z[::2] = np.arange(bb[3]) * shape[1] + (shape[1]*bb[1]+bb[0]+1)\n z = z.astype(int)\n return ' '.join(str(x) for x in z)\n\ndef dice_coef6(y_true_rle, y_pred_prob, y_pred_rle,th):\n if y_pred_prob