{"cells":[{"metadata":{},"cell_type":"markdown","source":"<h1>All About Efficiency</h1>\nOne of the problem that we encounter during this competition is submission time. In this kernel I will show you little trick on how to perform efficient inference for submission. In this kernel I will be using efficientnet B3 that is already pretrained on previous competition dataset. "},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import time\ns_time = time.time()\nimport os\n\nimport sys\nsys.path.append(os.path.abspath('../input/efficientnet/efficientnet-master/efficientnet-master/'))\nfrom efficientnet import EfficientNetB3","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport scipy as sp\n\n\nfrom keras.layers import Input, Dense, Dropout, GlobalAveragePooling2D, GlobalMaxPooling2D, Concatenate, BatchNormalization,PReLU\nfrom keras import backend as K\nfrom keras.models import Model, load_model\nfrom keras.utils import Sequence\nfrom keras.regularizers import l2\nfrom keras.optimizers import Adam\nfrom keras.callbacks import ReduceLROnPlateau, EarlyStopping\n\nfrom tqdm import tqdm\nimport glob\nfrom functools import partial\nfrom multiprocessing import Pool\nfrom matplotlib import pyplot as plt\nfrom efficientnet import EfficientNetB3 , preprocess_input\nfrom sklearn.metrics import cohen_kappa_score, mean_squared_error\nfrom albumentations import HorizontalFlip, Compose, RandomRotate90, RandomBrightness, Resize,OneOf, VerticalFlip,Rotate, RandomBrightnessContrast\n\nfrom sklearn.model_selection import StratifiedKFold\nimport cv2\n\nimport os\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"IMAGE_SIZE = 299\nbatch_size = 32\nTEST_DIR = 'test'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"data=pd.read_csv('../input/aptos2019-blindness-detection/test.csv')\n#data['id_code'] = data['id_code'].apply(lambda x : os.path.join('test', x)+'.png')\nsubm = pd.read_csv('../input/aptos2019-blindness-detection/sample_submission.csv')\ndata.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The key to inference efficiency is not to do redundant operations. In the case of using models that expect similar output shapes, the redundant operations are cropping and resizing (you can also add any other basic preprocessings). Hence, we will perform this preprocessing and save the file to the disk. However, there is a caveat, disk size is limited for kaggle kernel and there is a possibility we might overload the disk and we will get an error.\n\nThe solution to this problem is by splitting our inference task into several batches. After inference is done on a batch, saved images on the disk will be deleted, and we will repeat this to the amount of splits we want. In this kernel, In this kernel I will be splitting the job to 4 batches."},{"metadata":{"trusted":true},"cell_type":"code","source":"## First, we calculate how we split our inference to several batches\n\nif len(data) % 32 :\n    split_count = len(data)//32 + 1\nelse :\n    split_count = len(data) // 32\n    \nsplit_batch = np.array_split(np.arange(split_count) , 4)\nbatch_end =[min(len(data), (split_batch[i].max()+1)*batch_size) for i in range(len(split_batch))]\nbatch_start = [0] + batch_end[:-1]\nbatch_start, batch_end","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def load_cropped(path , image_size = (224,224)):\n    image = cv2.imread(path)\n    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n    image = crop_image_from_gray(image)\n    image = cv2.resize(image, image_size, interpolation = cv2.INTER_LANCZOS4)\n    return image\n\ndef save_image(path , directory = 'test' , image_size = (224,224)) :\n    filename = path.split('/')[-1]\n    image = load_cropped(path , image_size)\n    image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)\n    new_path = os.path.join(directory,filename)\n    cv2.imwrite(new_path, image)\n\n    \n##This function will do preprocessing and save file in a directory\ndef prepare_files(test_img_list, image_size = (224,224)) :\n\n    save_test = partial(save_image , directory = TEST_DIR, image_size = image_size)\n    if not(os.path.isdir(TEST_DIR)) :\n        os.mkdir(TEST_DIR)\n    with Pool(os.cpu_count()) as p :\n        list(tqdm(p.imap(save_test, test_img_list), total=len(test_img_list)))\n\ndef crop_image_from_gray(img,tol=7):\n    if img.ndim ==2:\n        mask = img>tol\n        return img[np.ix_(mask.any(1),mask.any(0))]\n    elif img.ndim==3:\n        gray_img = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)\n        mask = gray_img>tol\n        \n        check_shape = img[:,:,0][np.ix_(mask.any(1),mask.any(0))].shape[0]\n        if (check_shape == 0): # image is too dark so that we crop out everything,\n            return img # return original image\n        else:\n            img1=img[:,:,0][np.ix_(mask.any(1),mask.any(0))]\n            img2=img[:,:,1][np.ix_(mask.any(1),mask.any(0))]\n            img3=img[:,:,2][np.ix_(mask.any(1),mask.any(0))]\n    #         print(img1.shape,img2.shape,img3.shape)\n            img = np.stack([img1,img2,img3],axis=-1)\n    #         print(img.shape)\n        return img\n    \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_head(linear_size = [512,256,128] , probs_dropout = 0.5) :\n    inp = Input(shape = (None,None,base_model.output_shape[-1]))\n    x_avg = GlobalAveragePooling2D()(inp)\n    x_max = GlobalMaxPooling2D()(inp)\n    x = Concatenate()([x_avg,x_max])\n    x = BatchNormalization()(x)\n    x = Dropout(rate = probs_dropout)(x)\n    x = Dense(linear_size[0], activation = 'tanh', kernel_regularizer = l2(1e-5))(x)\n    #x = PReLU()(x)\n\n    for n in linear_size[1:] :\n        x = BatchNormalization()(x)\n        x = Dropout(rate = probs_dropout)(x)\n        x = Dense(n, kernel_regularizer = l2(1e-4))(x)\n        x = PReLU()(x)\n    x = Dense(1 , kernel_regularizer = l2(1e-4))(x)\n    return Model(inp,x)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We apply our train augmentations. On top of this, I also use randomized value of sigma for Ben's preprocessing."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_aug = Compose([\n    #Resize(IMAGE_SIZE,IMAGE_SIZE),\n    Rotate(360,border_mode = cv2.BORDER_CONSTANT, value = 0),\n    OneOf([\n        HorizontalFlip(),\n        VerticalFlip()\n    ]),\n    RandomBrightnessContrast(brightness_limit=0.3, contrast_limit=0.3,p=1)\n])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Our data loading will be performed all by this object, originally implemented for training but can be used for inference as well. Feel free to quote and reuse."},{"metadata":{"trusted":true},"cell_type":"code","source":"class ImageLoader(Sequence) :\n    def __init__(self , image_list, \n                        image_cat = None, \n                        batch_size = 32, \n                        shuffle = True, \n                        include_last = True, \n                        transforms = None,\n                        use_ben = True,\n                        sigma_ben = 15, #Default value sigma is not randomized\n                        randomized_sigma_ben_range = None, ##Apply randomized sigma for ben's preprocessing\n                        crop = True,\n                        resized = True) :\n\n        self.image_list = np.array(image_list)\n        self.crop = crop\n        self.use_ben  = use_ben\n        self.have_transform = False\n        if self.use_ben :\n            if not(randomized_sigma_ben_range is None) :\n                self.sigma_ben_range = randomized_sigma_ben_range\n                self.random_ben = True\n            else :\n                self.random_ben = False\n                self.sigma_ben = sigma_ben\n            \n        if transforms is None :\n            self.have_transform = False\n            if not(resized) :\n                self.have_transform = True\n                self.transforms = Resize(IMAGE_SIZE,IMAGE_SIZE)\n        else :\n            self.have_transform = True \n            self.transforms = transforms\n        \n        if len(self.image_list) == 0 :\n            print('List is empty please recheck')\n        else :\n            print('List contains {} images'.format(len(self.image_list)))\n        #print(self.image_list)\n        if image_cat is None :\n            self.image_cat = None\n        else :\n            self.image_cat = np.array(image_cat)\n            assert len(self.image_list) == len(self.image_cat) , 'Image List and Cat Mismatch'\n            \n        self.batch_size = batch_size \n        self.shuffle = shuffle\n        self.index = np.arange(len(self.image_list), dtype = int)\n        #print(type(image_list))\n        if shuffle :\n            self.shuffle_index()\n        self.include_last = include_last\n        \n    def __len__(self) :\n        if self.include_last :\n            if len(self.image_list) % self.batch_size :\n                return len(self.image_list) // self.batch_size + 1\n            else :\n                return len(self.image_list) // self.batch_size\n        else :\n            return len(self.image_list) // self.batch_size \n    \n    def __getitem__(self , index) :\n        \n        if self.include_last and (index == (len(self.image_list) // self.batch_size)) :\n            batch_count = len(self.image_list) % self.batch_size\n        else :\n            batch_count = self.batch_size \n        \n        X = np.empty((batch_count,IMAGE_SIZE,IMAGE_SIZE,3))\n        \n        idxs = np.arange(index*self.batch_size, (index*self.batch_size)+batch_count )\n        if self.use_ben :\n            if self.random_ben :\n                sigma = np.random.randint(self.sigma_ben_range[0], self.sigma_ben_range[1] , size = batch_count)\n            else :\n                sigma = [self.sigma_ben]*batch_count\n\n        for i , idx in enumerate(idxs) :\n            img = cv2.imread(self.image_list[idx])\n            img = cv2.cvtColor(img , cv2.COLOR_BGR2RGB)\n            if self.crop :\n                img = crop_image_from_gray(img)\n            if self.have_transform :\n                img = self.transforms(image=img)['image']\n            if self. use_ben :\n                img = cv2.addWeighted (img,4, cv2.GaussianBlur(img, (0,0) , sigma[i]) ,-4 ,128)\n            X[i] = img\n            \n        X = preprocess_input(X)\n        if not(self.image_cat is None) :\n            return X, self.image_cat[idxs]\n        else :\n            return X\n    \n    def shuffle_index(self) :\n        np.random.shuffle(self.index)\n            \n        self.image_list = self.image_list[self.index]\n            \n        if not(self.image_cat is None) :\n            self.image_cat = self.image_cat[self.index]\n            \n    def on_epoch_end(self) :\n        if self.shuffle :\n            self.shuffle_index()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We proceed to load our model to our kernels. You can load several small(but powerful) model in the same kernel without encountering any issues."},{"metadata":{"trusted":true},"cell_type":"code","source":"models = []\nK.clear_session()\nfor i in range(4) :\n    base_model = EfficientNetB3(weights = None , include_top = False , input_shape = (IMAGE_SIZE,IMAGE_SIZE,3))\n    head_model = get_head()\n    model = Model(base_model.input, head_model(base_model.output))\n    model.load_weights('../input/finetune-effnet/model_{}.h5'.format(i))\n    models.append(model)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Define some helper functions."},{"metadata":{"trusted":true},"cell_type":"code","source":"##Get the correct file path, from id\ndef get_path(path) :\n    def corrector(x) :\n        return os.path.join(path, x) + '.png'\n    return corrector\n\n##Function to clear directory where images are stored,\n#rmdir is not necessary but for sanity check since if there are still any remains it will throw an error.\ndef clean_images() :\n    if os.path.isdir('test') :\n        for f in glob.glob('test/*.png') :\n            os.remove(f)\n        os.rmdir('test')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"TTA_count = 8\npreds = np.zeros([len(data),4]) ##Array storing prediction\n\nfor i, (batch_s, batch_e) in enumerate(zip(batch_start,batch_end)) :\n    \n    test_img_list = data.iloc[batch_s:batch_e,0].apply(get_path('../input/aptos2019-blindness-detection/test_images/')).tolist() \n    prepare_files(test_img_list, image_size = (IMAGE_SIZE,IMAGE_SIZE))\n    \n    test_img_list = data.iloc[batch_s:batch_e,0].apply(get_path('test')).tolist()\n    \n    loader = ImageLoader(test_img_list,\n                         shuffle = False,\n                         crop = False,\n                         transforms=train_aug,\n                         resized = True)\n    for j in range(4) :\n        for k in range(TTA_count) :\n            preds[slice(batch_s,batch_e),j:j+1] += models[j].predict_generator(loader , \n                                                                              use_multiprocessing = True, \n                                                                              workers = os.cpu_count())/TTA_count\n    ##Clean image after each batch inference job VERY IMPORTANT!\n    clean_images()\n    \n    ##Sanity check that directory is cleaned\n    try :\n        print(len(os.listdir('test')))\n    except :\n        print('Batch {} Cleared'.format(i+1)) \n        print()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## Copied from predict method of optimized rounder\ndef get_labels(X , coef = [0.5,1.5,2.5,3.5]) :\n    X_p = np.copy(X)\n    for i, pred in enumerate(X_p):\n        if pred < coef[0]:\n            X_p[i] = 0\n        elif pred >= coef[0] and pred < coef[1]:\n            X_p[i] = 1\n        elif pred >= coef[1] and pred < coef[2]:\n            X_p[i] = 2\n        elif pred >= coef[2] and pred < coef[3]:\n            X_p[i] = 3\n        else:\n            X_p[i] = 4\n    return X_p\n\n## Predetermined coefficient, you can try optimize this value or use the default one\ncoef = [0.5353928773986922, 1.5749586701298883, 2.448495965240568, 3.393477738062221]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally we will finalized our submission."},{"metadata":{"trusted":true},"cell_type":"code","source":"### +2 Since I trained my network to output values centered at 0\nsubm['diagnosis'] = get_labels(preds.mean(axis = 1) + 2, coef).astype(int)\nsubm.to_csv('submission.csv', index = False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Kernel Runtime : {:.3f} minute'.format((time.time() - s_time) / 60 ))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Notice that I did 32 inferences per image but inference only takes less than half an hour for this kernel. For final inference time estimation simply multiply the kernel runtime by approx. 6-8 since it is mentioned that the size of private test set is around 13000 images.\n\nSorry for the poor presentation. I am not the best kernel writer, but hopefully the contents can be useful."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}