{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"# Ranzcr Clip - Catheter and Line Position Challenge\n\n##  Fastai + Bayesian Optimization (Albumentation)\n\nIn short: It's all about identifing malpositioned lines and tubes in patients. More information about the challenge can be found [here](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/overview)\n\nIn this notebook we will use Bayesian Optimization (with [hyperopt](https://github.com/hyperopt/hyperopt)) to optimize [Albumentations'](https://github.com/albumentations-team/albumentations) parameters in a [fastai](https://docs.fast.ai/) environment. \n\nNote that this notebook is just a small working example and serves only as a guildeline for the use of Bayesian Optimization with hyperopt. It should give you a feeling of how to use hyperopt + fastai + albumentation. Parameter values and selected Albumentation methods are chosen at random."},{"metadata":{},"cell_type":"markdown","source":"### Imports"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"from fastai.vision.all import *\nfrom fastai.metrics import accuracy_multi, RocAucMulti\nimport cv2\nimport pandas as pd\nfrom sklearn.metrics import roc_auc_score\nimport albumentations\nimport warnings\nwarnings.simplefilter(\"ignore\", FutureWarning)\nfrom hyperopt import fmin, hp, tpe, Trials, STATUS_OK\nimport gc","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%cd ../input/efficientnetpytorch/EfficientNet-PyTorch-master\nfrom efficientnet_pytorch import EfficientNet\n%cd -","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Paths"},{"metadata":{"trusted":true},"cell_type":"code","source":"data_path = \"../input/ranzcr-clip-catheter-line-classification/\"\ntrain_folder = data_path + \"train/\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Variables"},{"metadata":{"trusted":true},"cell_type":"code","source":"COL_NAMES = ['UID', 'ETTA','ETTB','ETTN','NGTA','NGTB','NGTI','NGTN','CVCA','CVCB','CVCN','SGCP', 'PatientID']\n\n# Albumentation\nRRC_SIZE = 512\nRRC_MIN_SCALE = 0.75\nRRC_RATIO = (1., 1.)\nBRIGHTNESS_LIMIT = (-0.15,0.15)\n\n# Augmentation\nAUG_TRANS_SIZE = 256\nAUG_TRANS_WARP = 0\nAUG_TRANS_FLIP = True\nAUG_TRANS_ROTATE = 20\nAUG_TRANS_ZOOM = 1.2\nAUG_TRANS_LIGHTNING = 0\n\n# Model\nEFFICIENTNET_PARAMS = [\"efficientnet-b4\", 1792]\n\n# DataLoader\nBS = 32\n\n# Callbacks\nPATIENCE_EARLY_STOPPING = 5\n\n# Training. TODO: I chose such a small number of EPOCHS and FREEZE_EPOCHS just to create a small example.\nEPOCHS = 2\nFREEZE_EPOCHS = 2","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 1. Utils"},{"metadata":{},"cell_type":"markdown","source":"### 1.1 Custom Transform\n\n<u>RandomResizedCrop</u>\nTransform images to same size by 1. resizing and 2. random crop\n\n<u>Coarse Dropout / Cutout</u><br>\nIn short: Randomly remove rectangles from a given image. Where coarse dropout is removing many small rectangles of similar size and cutout is removing 1 large rectangle of random size\n\n<u>Random Brightness</u><br>\nRandomly change brightness of the image.\n\nNote that we use different Transforms for Training and Validation (Testing). We will only apply a RandomResizedCrop on the Validation/ Testing Data."},{"metadata":{"trusted":true},"cell_type":"code","source":"class AlbumentationsTransform(RandTransform):\n    \"\"\"\n    A transform handler for multiple Albumentations transforms distinguishing between training \n    \"\"\"\n    split_idx, order = None, 2\n    def __init__(self, train_aug, valid_aug):\n        store_attr()\n        \n    def before_call(self, b, split_idx):\n        self.idx = split_idx\n    \n    def encodes(self, img: PILImage):\n        if self.idx == 0:\n            aug_img = self.train_aug(image=np.array(img))['image']\n        else:\n            aug_img = self.valid_aug(image=np.array(img))['image']\n        return PILImage.create(aug_img)\n    \ndef get_train_aug(brightness_prob, coarse_prob, cutout_prob): \n    return albumentations.Compose([\n        albumentations.RandomResizedCrop( \n            RRC_SIZE, RRC_SIZE,            \n            scale=(RRC_MIN_SCALE, 1.0),\n            ratio=RRC_RATIO,\n            p=1.0\n        ),\n        albumentations.RandomBrightness(\n            limit=BRIGHTNESS_LIMIT,\n            p=brightness_prob\n        ),\n        albumentations.CoarseDropout(p=coarse_prob),\n        albumentations.Cutout(p=cutout_prob)\n    ])\n\ndef get_valid_aug(): \n    return albumentations.Compose([\n        albumentations.Resize(RRC_SIZE, RRC_SIZE, p=1.0)\n    ])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Short note on *aug_transforms*. It's a utility function which applies a list of transforms such as rotation, flipping etc **only** on the Training images (e.g. notice how *dls.valid.show_batch(nrows=1, ncols=5)* does not rotate and flip the images)"},{"metadata":{"trusted":true},"cell_type":"code","source":"batch_tfms = [*aug_transforms(size=AUG_TRANS_SIZE, max_warp=AUG_TRANS_WARP, do_flip=AUG_TRANS_FLIP,\n                              max_rotate=AUG_TRANS_ROTATE, max_zoom=AUG_TRANS_ZOOM,\n                              max_lighting=AUG_TRANS_LIGHTNING),\n              Normalize.from_stats(*imagenet_stats)]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 1.2 Metric\n\nLet's define our own metric.\n\nFor each label/class we want to calculate the area under the receiver operating curve (here, in short: AUC). Further, we want another metric, taking the mean of each target's AUC score.\n\nProbs to RobertLangdonVinci. Thanks for sharing your [Notebook](https://www.kaggle.com/robertlangdonvinci/fastai-efficientnetb5-custom-metrics)"},{"metadata":{"trusted":true},"cell_type":"code","source":"def mean_auc(preds, targs, labels=range(len(COL_NAMES)-2)):\n    return np.mean([roc_auc_score(targs[:,i], preds[:,i]) for i in labels])\ndef ETTA_auc(*args):\n    return mean_auc(*args, labels=[0])\ndef ETTB_auc(*args):\n    return mean_auc(*args, labels=[1])\ndef ETTN_auc(*args):\n    return mean_auc(*args, labels=[2])\ndef NGTA_auc(*args):\n    return mean_auc(*args, labels=[3])\ndef NGTB_auc(*args):\n    return mean_auc(*args, labels=[4])\ndef NGTI_auc(*args):\n    return mean_auc(*args, labels=[5])\ndef NGTN_auc(*args):\n    return mean_auc(*args, labels=[6])\ndef CVCA_auc(*args):\n    return mean_auc(*args, labels=[7])\ndef CVCB_auc(*args):\n    return mean_auc(*args, labels=[8])\ndef CVCN_auc(*args):\n    return mean_auc(*args, labels=[9])\ndef SGCP_auc(*args):\n    return mean_auc(*args, labels=[10])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"metrics = [ AccumMetric(mean_auc, flatten=False),\n            AccumMetric(ETTA_auc, flatten=False),\n            AccumMetric(ETTB_auc, flatten=False),\n            AccumMetric(ETTN_auc, flatten=False),\n            AccumMetric(NGTA_auc, flatten=False),\n            AccumMetric(NGTB_auc, flatten=False),\n            AccumMetric(NGTI_auc, flatten=False),\n            AccumMetric(NGTN_auc, flatten=False),\n            AccumMetric(CVCA_auc, flatten=False),\n            AccumMetric(CVCB_auc, flatten=False),\n            AccumMetric(CVCN_auc, flatten=False),\n            AccumMetric(SGCP_auc, flatten=False), \n            accuracy_multi]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 1.3 Model\n\nCreate a Learner based on a pretrained EfficientNet-B4 model."},{"metadata":{"trusted":true},"cell_type":"code","source":"class RanzerModel(Module):\n    def __init__(self, num_classes):\n        self.effnet = EfficientNet.from_pretrained(EFFICIENTNET_PARAMS[0], weights_path=None, include_top=False)\n        self.dropout = nn.Dropout(0.1)\n        self.out = nn.Linear(EFFICIENTNET_PARAMS[1], num_classes)\n\n    def forward(self, image):\n        batch_size, _, _, _ = image.shape\n        x = self.effnet.extract_features(image)\n        x = F.adaptive_avg_pool2d(x, 1).reshape(batch_size, -1)\n        outputs = self.out(self.dropout(x))\n        return outputs","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 1.4 Load Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Read training set\ndf_train = pd.read_csv(data_path + \"train.csv\")\ndf_train.columns = COL_NAMES\n\n# Add entire path such that DataLoader knows the path for each file\ndf_train['path'] = df_train['UID'].map(lambda x:str(train_folder + x)+'.jpg')\ndf_train = df_train.drop(columns=['UID'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 1.5 Callbacks\n\nMore information on Callbacks can be found [here](https://docs.fast.ai/callback.core.html)"},{"metadata":{"trusted":true},"cell_type":"code","source":"cb1 = SaveModelCallback(monitor='mean_auc', fname='best-model', comp=np.greater)\ncb2 = EarlyStoppingCallback(monitor='valid_loss', min_delta=0.0, patience=PATIENCE_EARLY_STOPPING)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2. Training"},{"metadata":{},"cell_type":"markdown","source":"### 2.1 Search Space\n\nDefine the search space. Provide parameters and ranges of values using hyperopts stochastic expressions such as <br></br>\n\n\n* hp.choice: Returns one of the options\n* hp.randint: Returns a random integer in the range [0, upper)\n* hp.uniform: Returns a value uniformly between two variables\n\nSee https://github.com/hyperopt/hyperopt/wiki/FMin#21-parameter-expressions"},{"metadata":{"trusted":true},"cell_type":"code","source":"search_space = hp.choice('classifier',[\n    {\n        'param': {'brightness_prob': hp.uniform('brightness_prob', 0.0, 1.0),\n                  'coarse_prob': hp.uniform('coarse_prob', 0.0, 1.0),\n                  'cutout_prob': hp.uniform('cutout_prob', 0.0, 1.0)\n                 }\n    }\n])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.2 Define Objective Function\n\nWe need to define a function which should be **minimized** (that is why we are returning the negative mean AUC). Just place your model training inside such a function."},{"metadata":{"trusted":true},"cell_type":"code","source":"def hyperparameter_tuning(params):\n    \n    \"\"\"\n    Objective function\n\n    It takes in hyperparameter settings, fits a model based on those settings,\n    evaluates the model, and returns the mean AUC score.\n\n    :param params: map specifying the hyperparameter settings to test\n    :return: mean AUC for the fitted model\n    \"\"\"\n    \n    print(\"Parameter: {}\".format(params['param']))\n    brightness_prob = params['param']['brightness_prob']\n    coarse_prob = params['param']['coarse_prob']\n    cutout_prob = params['param']['cutout_prob']\n    \n    item_tfms = AlbumentationsTransform(get_train_aug(brightness_prob, coarse_prob, cutout_prob),\n                                        get_valid_aug())\n    \n    data = DataBlock(blocks=(ImageBlock, MultiCategoryBlock(encoded=True, vocab=list(df_train.columns[:11]))),\n                 splitter = RandomSplitter(seed=123),\n                 get_x = ColReader(12),\n                 get_y = ColReader(list(range(11))),\n                 item_tfms = item_tfms,\n                 batch_tfms = batch_tfms,\n                )\n\n    dls = data.dataloaders(df_train, bs=BS)\n    efficientnet = RanzerModel(11)\n    learn = Learner(dls, efficientnet, metrics=metrics, opt_func=Adam, cbs=[cb1, cb2])\n    learn.to_native_fp16()\n    \n    learn.fine_tune(epochs=EPOCHS, base_lr=2e-3, freeze_epochs=FREEZE_EPOCHS)\n    \n    auc_mean = float(learn.validate(dl=dls.valid)[2])\n    \n    del learn\n    torch.cuda.empty_cache()\n    gc.collect()\n  \n\n    return {'loss': -auc_mean, 'status': STATUS_OK}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.3 Optimization Loop\n\nSave statistics with the help of a Trials object. More information can be found [here](https://github.com/hyperopt/hyperopt/wiki/FMin#12-attaching-extra-information-via-the-trials-object).\n\nIn short: We are using the Trials object to store information such as the AUC Mean score of each evaluation and whether the evaluation went well."},{"metadata":{"trusted":true},"cell_type":"code","source":"trials = Trials()\n\nargmin = fmin(fn=hyperparameter_tuning,  # Objective function\n              space=search_space,  # Search space\n              algo=tpe.suggest,  # Use the tree of Parzen estimators:(Bayesian optimization). Alternative: random.suggest (Random Search)\n              trials=trials, # Trials object\n              max_evals=2 # Maximum number of evaluations/trials (hyperparameter settings)\n             )\nprint(argmin)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3. Interpretation"},{"metadata":{},"cell_type":"markdown","source":"*argmin* contains the parameters for the model which lead to the highest auc mean.\n\nIn this case it is:\n* a brightness probability of 0.38\n* a coarse probability of 0.91\n* a cutout probability of 0.86"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(argmin)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}