{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Regressor for the Blindness Detection challenge.\n\nThis kernel uses an **EfficientNetB0** regression model (initially **pre-trained** as a [classifier](https://www.kaggle.com/hmendonca/efficientnet-pytorch-ignite-amp-aptos19)),\nminimising the mean squared error (**MSE**) of the predicted blindness score \\[0-4\\] against the training labels.\n\nLater, it maximises the **Quadratic Weighted Kappa** by optimising the boundary coefficients as borrowed from [abhishek](https://www.kaggle.com/abhishek/optimizer-for-quadratic-weighted-kappa)'s kernel with a slightly cleaner code base, I hope ;)\n\nIt also uses a BatchSampler with weighted **OverSampling** to balance out the classes distribution during training, and test time augmentation (**TTA**) to improve the submission score.\n\n\nIf you please, upvote and leave questions or any constructive feedback below (for me and other kagglers learning).\nCheers!\n\n\nEfficientNet architecture and pre-trained weights from: https://www.kaggle.com/hmendonca/efficientnet-pytorch-ignite-aptos19 and [Ignite samples](https://github.com/pytorch/ignite/tree/master/examples):\n> Recently new ConvNets architectures have been proposed in \"[EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://arxiv.org/pdf/1905.11946.pdf)\" paper. According to the paper, model's compound scaling starting from a 'good' baseline provides an network that achieves state-of-the-art on ImageNet, while being 8.4x smaller and 6.1x faster on inference than the best existing ConvNet.\n![efficientnets](https://raw.githubusercontent.com/pytorch/ignite/c22609796031f5831f054036895696c7e4df07ce/examples/notebooks/assets/efficientnets.png)\n"},{"metadata":{},"cell_type":"markdown","source":"## Imports"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"# Ignore  the warnings\nimport warnings\nwarnings.filterwarnings('always')\nwarnings.filterwarnings('ignore')\n\n# data visualisation and manipulation\nimport numpy as np\nimport pandas as pd\nimport scipy as sp\nimport matplotlib.pyplot as plt\nfrom matplotlib import style\nimport seaborn as sns\n \n#configure\n# sets matplotlib to inline and displays graphs below the corressponding cell.\n%matplotlib inline  \nstyle.use('fivethirtyeight')\nsns.set(style='whitegrid', color_codes=True)\n\nfrom sklearn.metrics import confusion_matrix, cohen_kappa_score\n\n# specifically for manipulating zipped images and getting numpy arrays of pixel values of images.\nimport cv2                  \nimport numpy as np  \nfrom tqdm import tqdm, tqdm_notebook\nimport os, math\nfrom random import shuffle  \nfrom zipfile import ZipFile\nfrom PIL import Image\nfrom sklearn.utils import shuffle\n\n!ls ../input/*","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import fastai\nfrom fastai import *\nfrom fastai.vision import *\nfrom fastai.callbacks import *\n\nfastai.__version__","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# check if the kernel is running in interactive/edit/debug mode https://www.kaggle.com/masterscrat/detect-if-notebook-is-running-interactively\ndef is_interactive():\n   return 'runtime' in get_ipython().config.IPKernelApp.connection_file\n\nprint('Interactive?', is_interactive())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# from https://www.kaggle.com/chanhu/eye-inference-num-class-1-ver3\ndef seed_everything(seed):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\nseed_everything(42)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"package_path = '../input/efficientnet-pytorch/efficientnet-pytorch/EfficientNet-PyTorch-master'\nsys.path.append(package_path)\n\nfrom efficientnet_pytorch import EfficientNet","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# copy pretrained weights for resnet34 to the folder fastai will search by default\nPath('/tmp/.cache/torch/checkpoints/').mkdir(exist_ok=True, parents=True)\nmodel_path = '/tmp/.cache/torch/checkpoints/efficientNet.pth'\n!cp ../input/efficientnet*/efficientNet_*.pth {model_path}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Stratified Dataset"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"PATH = Path('../input/aptos2019-blindness-detection')\n\ndf_train = pd.read_csv(PATH/'train.csv', dtype={'id_code':str, 'diagnosis':int})\ndf_test = pd.read_csv(PATH/'test.csv')\n\n# if is_interactive():  # fast debug mode\n#     df_train = df_train.sample(1200)\n\n_ = df_train.hist()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# create Stratified validation split (10%)\nfrom sklearn.model_selection import StratifiedKFold\ncv = StratifiedKFold(n_splits=10, random_state=42)\ntr_ids, val_ids = next(cv.split(df_train.id_code, df_train.diagnosis))\nprint(len(tr_ids), len(val_ids))\n_ = df_train.loc[val_ids].hist()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# # append minority classes again to improve label balance\n# minority = df_train.loc[tr_ids] # don't touch validation set\n# minority = minority[(minority.diagnosis == 1) | (minority.diagnosis == 3) | (minority.diagnosis == 4)]\n# df_balanced = pd.concat([df_train, minority])\n\n# # def norm_data(Y): return (Y - 1.) / 4\n# # def denorm_data(Y_): return (Y_ * 4.) + 1.\n# # df_balanced.diagnosis = norm_data(df_balanced.diagnosis)\n\n# _ = df_balanced.hist()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# create image data bunch\n# aptos19_stats = ([0.42, 0.22, 0.075], [0.27, 0.15, 0.081])\nsrc = (ImageList.from_df(df=df_train, path=PATH, folder='train_images', suffix='.png')\n       .split_by_idx(val_ids)\n       .label_from_df(cols='diagnosis', label_cls=FloatList))\ndata = ImageDataBunch.create_from_ll(src,\n                                     ds_tfms=get_transforms(flip_vert=True,\n                                                            max_rotate=2., max_zoom=1.02,\n                                                            max_lighting=0.2, max_warp=0.05,\n                                                            p_affine=0.9, p_lighting=0.8),\n                                     size=256,\n                                     bs=32,\n                                     num_workers=os.cpu_count()\n                                    ).normalize(imagenet_stats)\ndata","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# show some sample images\ndata.show_batch(rows=3, figsize=(7,6), ds_type=DatasetType.Train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Define model"},{"metadata":{"trusted":true},"cell_type":"code","source":"def EfficientNetB4(pretrained=False):\n    \"\"\"Constructs a EfficientNetB0 model for FastAI.\n    Args:\n        pretrained (bool): If True, returns a model pre-trained on ImageNet\n    \"\"\"\n    model = EfficientNet.from_name('efficientnet-b4', override_params={'num_classes': 1 }) ## Regressor\n    if pretrained:\n        model_state = torch.load(model_path)\n        # load original weights apart from its head\n        if '_fc.weight' in model_state.keys():\n            model_state.pop('_fc.weight')\n            model_state.pop('_fc.bias')\n            res = model.load_state_dict(model_state, strict=False)\n            assert str(res.missing_keys) == str(['_fc.weight', '_fc.bias']), 'issue loading pretrained weights'\n        else:\n            # A basic remapping is required\n            from collections import OrderedDict\n            mapping = { i:o for i,o in zip(model_state.keys(), model.state_dict().keys()) }\n            mapped_model_state = OrderedDict([\n                (mapping[k], v) for k,v in model_state.items() if not mapping[k].startswith('_fc')\n            ])\n            res = model.load_state_dict(mapped_model_state, strict=False)\n            print(res)\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"# create model\nmodel = EfficientNetB4(pretrained=True)\n# print model structure (hidden)\nmodel","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Non-linear Kappa Optimizer"},{"metadata":{"trusted":true},"cell_type":"code","source":"# inspired by https://www.kaggle.com/tanlikesmath/intro-aptos-diabetic-retinopathy-eda-starter\nclass KappaOptimizer(nn.Module):\n    def __init__(self):\n        super().__init__()\n        self.coef = [0.5, 1.5, 2.5, 3.5]\n        # define score function:\n        self.func = self.quad_kappa\n    \n    def predict(self, preds):\n        return self._predict(self.coef, preds)\n\n    @classmethod\n    def _predict(cls, coef, preds):\n        if type(preds).__name__ == 'Tensor':\n            y_hat = preds.clone().view(-1)\n        else:\n            y_hat = torch.FloatTensor(preds).view(-1)\n\n        for i,pred in enumerate(y_hat):\n            if   pred < coef[0]: y_hat[i] = 0\n            elif pred < coef[1]: y_hat[i] = 1\n            elif pred < coef[2]: y_hat[i] = 2\n            elif pred < coef[3]: y_hat[i] = 3\n            else:                y_hat[i] = 4\n        return y_hat.int()\n    \n    def quad_kappa(self, preds, y):\n        return self._quad_kappa(self.coef, preds, y)\n\n    @classmethod\n    def _quad_kappa(cls, coef, preds, y):\n        y_hat = cls._predict(coef, preds)\n        return cohen_kappa_score(y, y_hat, weights='quadratic')\n\n    def fit(self, preds, y):\n        ''' maximize quad_kappa '''\n        print('Early score:', self.quad_kappa(preds, y))\n        neg_kappa = lambda coef: -self._quad_kappa(coef, preds, y)\n        opt_res = sp.optimize.minimize(neg_kappa, x0=self.coef, method='nelder-mead',\n                                       options={'maxiter':1000, 'fatol':1e-20, 'xatol':1e-20})\n        print(opt_res)\n        self.coef = opt_res.x\n        print('New score:', self.quad_kappa(preds, y))\n\n    def forward(self, preds, y):\n        ''' the pytorch loss function '''\n        return torch.tensor(self.quad_kappa(preds, y))\n\nkappa_opt = KappaOptimizer()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":false,"_kg_hide-output":false,"trusted":true},"cell_type":"code","source":"## some visual tests\npreds = [-2.5, 0.3, 0.4,  0.5, 1.2, 1.41,  1.5, 2.4, 2.42,  2.5, 3.4, 3.43,  3.5, 4, 9]\ny_hat = kappa_opt.predict(preds)\nprint(y_hat)\nprint('calc scores:', \n      kappa_opt(preds[::-1], y_hat.tolist()[::-1]), # inverse order\n      kappa_opt(preds, y_hat+1),\n      kappa_opt(preds, np.ones_like(y_hat)),\n      kappa_opt(preds, 4-y_hat))\n\n# shift predictions to test optimizer\nn_preds = np.array(preds)+0.11\nprint('goal Y:', y_hat)\nprint('before:', kappa_opt.predict(n_preds))\nkappa_opt.fit(n_preds, y_hat)\nprint('after: ', kappa_opt.predict(n_preds))\n\n# reset\nkappa_opt = KappaOptimizer()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Adapted from FastAI master\n# the weights of each label is 1/count of the labels of that class,\n# so the label distribution on each batch is uniform.\n# I.e. a batch of 100 images would have about 20 images of each class.\nfrom torch.utils.data.sampler import BatchSampler, WeightedRandomSampler\n\nclass OverSamplingCallback(LearnerCallback):\n    def __init__(self,learn:Learner, weights:torch.Tensor=None):\n        super().__init__(learn)\n        labels = self.learn.data.train_dl.dataset.y.items.astype(int)\n        _,counts = np.unique(labels, return_counts=True)\n#         counts = 1. / counts\n        counts = 1. / np.sqrt(counts)  # non-linear weights\n        self.weights = (weights if weights is not None else torch.DoubleTensor(counts[labels]))\n\n    def on_train_begin(self, **kwargs):\n        self.learn.data.train_dl.dl.batch_sampler = BatchSampler(\n            WeightedRandomSampler(self.weights, len(self.learn.data.train_dl.dataset)),\n                                  self.learn.data.train_dl.batch_size, False)\n\nclass StratifiedBatchSampler(Sampler):\n    def __init__(self, labels, batch_size):\n        self.labels = labels\n        self.len = math.ceil(len(labels) / batch_size)\n\n    def __len__(self): return self.len\n\n    def __iter__(self):\n        ids = np.arange(len(self.labels))\n        folds = StratifiedKFold(n_splits=self.len, shuffle=True).split(ids, self.labels)\n        for _,batch in folds:\n            yield batch.tolist()\n\nclass StratifiedBatchCallback(LearnerCallback):\n    def on_train_begin(self, **kwargs):\n        self.learn.data.train_dl.dl.batch_sampler = StratifiedBatchSampler(\n            self.learn.data.train_dl.dataset.y.items.astype(int),\n            self.learn.data.train_dl.batch_size)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Define Task"},{"metadata":{"trusted":true},"cell_type":"code","source":"# build model (using EfficientNetB0)\nlearn = Learner(data, model,\n                loss_func=MSELossFlat(),\n                metrics=[kappa_opt],\n                callback_fns=[BnFreeze,\n#                               StratifiedBatchCallback,\n#                               partial(GradientClipping, clip=0.2),\n                              partial(SaveModelCallback, monitor='quad_kappa', name='bestmodel')]\n               )\nlearn.split( lambda m: (model._conv_head,) )\nlearn.freeze()\nlearn.model_dir = '/tmp/'","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"# learn.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Training"},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.lr_find(end_lr=0.5)\nlearn.recorder.plot(suggestion=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# train head first\nlearn.fit_one_cycle(2, max_lr=5e-3, div_factor=15)\nlearn.save('stage-1')\nlearn.recorder.plot_losses()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# unfreeze and search appropriate learning rate for full training\nlearn.unfreeze()\nlearn.lr_find(start_lr=1e-10, wd=1e-3)\nlearn.recorder.plot(suggestion=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# train all layers\nlearn.fit_one_cycle(6, max_lr=slice(1e-5, 1e-3), div_factor=10, wd=1e-3)\nlearn.save('stage-2')\nlearn.recorder.plot_losses()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# schedule of the lr (left) and momentum (right) that the 1cycle policy uses\nlearn.recorder.plot_lr(show_moms=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# kappa scores\nlearn.recorder.plot_metrics()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# reload best model so far and look for a new learning rate\nlearn.load('bestmodel')\nlearn.lr_find(start_lr=1e-10)\nlearn.recorder.plot(suggestion=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# train all layers, now with some weight decay\nlearn.fit_one_cycle(12, max_lr=slice(2e-6, 2e-4), div_factor=20)\nlearn.save('stage-3')\nlearn.recorder.plot_losses()\n# # schedule of the lr (left) and momentum (right) that the 1cycle policy uses\n# learn.recorder.plot_lr(show_moms=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Find best coefficients"},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.load('bestmodel')\n\n# remove zoom from FastAI TTA\ntta_params = {'beta':0.12, 'scale':1.0}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"valid_preds = learn.get_preds(ds_type=DatasetType.Valid)\n# valid_preds = learn.TTA(ds_type=DatasetType.Valid, **tta_params)\n_ = pd.DataFrame(valid_preds[0].numpy().flatten()).hist()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"kappa_opt.fit(valid_preds[0], valid_preds[1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('New coefficients:', kappa_opt.coef)\nnew_valid = kappa_opt.predict(valid_preds[0]).numpy()\n_ = pd.DataFrame(new_valid).hist()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## TTA and submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_df = pd.read_csv('../input/aptos2019-blindness-detection/sample_submission.csv')\nsample_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# create test dataset\nlearn.data.add_test(ImageList.from_df(\n    sample_df, PATH,\n    folder='test_images',\n    suffix='.png'\n))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# test time augmentation\ntest_preds,_ = learn.get_preds(ds_type=DatasetType.Test)\n# test_preds,_ = learn.TTA(ds_type=DatasetType.Test, **tta_params)\n_ = pd.Series(test_preds.squeeze().tolist()).hist()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# apply optimised coefficients\nsample_df.diagnosis = kappa_opt.predict(test_preds)\nsample_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# save sub\nsample_df.to_csv('submission.csv', index=False)\n_ = sample_df.hist()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"#move models back to root folder\n!mv {learn.model_dir}/*.pth .\nos.listdir()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}