{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Background\n\nRecently I got a chance to go through Abhishek Thakur's youtube video published on his channel. It introduced a common framework for image classification problems using DL.\n\nThought, I would share that with all of you by implementing efficientnet.\n\nHope you find it useful","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Below framework helps you in defining and streamlining forward, loss computing and early stopping functions among others. Helps you incorporate other models of your choice.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# References:\n1. Abhishek Thakur's youtube video: https://youtu.be/WaCFd-vL4HA\n2. WTFML library: https://github.com/abhishekkrthakur/wtfml\n3. EfficientNet: https://github.com/lukemelas/EfficientNet-PyTorch","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport os\nimport imageio\nimport pydicom\nimport matplotlib.pyplot as plt\nfrom tqdm import tqdm\nimport glob\nfrom PIL import Image, ImageFile\nfrom joblib import Parallel, delayed\n\nImageFile.LOAD_TRUNCATED_IMAGES = True","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import cv2\nimport albumentations\nfrom albumentations.pytorch.transforms import ToTensorV2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.utils import model_zoo\nfrom albumentations.pytorch.transforms import ToTensorV2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn import model_selection\nfrom sklearn import metrics","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install efficientnet_pytorch\nfrom efficientnet_pytorch import EfficientNet","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Lets create a methodology for dividing dataset into multiple folds","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class stratification:\n    def __init__(self,path_dir,num_splits):\n        self.input_path = path_dir\n        self.n_splits = num_splits\n        self.df = pd.read_csv(os.path.join(input_path,\"train.csv\"))\n        \n    def create_split(self):\n        self.df['kfold'] = -1\n        #Shuffling the csv file => to get a new shuffled dataframe\n        self.df = self.df.sample(frac=1).reset_index(drop=True)\n        #Target value\n        y=self.df.target.values\n        #Why stratified - because we want the ratio of +ve:-ve samples to be the same\n        kf = model_selection.StratifiedKFold(n_splits=self.n_splits)\n        \n        kfold_df_dict = {}\n        \n        for fold_, (train_idx, val_idx) in enumerate(kf.split(X=self.df,y=y)):\n            df_temp = pd.read_csv(os.path.join(input_path,\"train.csv\"))\n            df_temp['kfold'] = -1\n            df_temp['dataset_type'] = 'train'\n            df_temp.loc[:,'kfold']=fold_\n            df_temp.loc[val_idx,'dataset_type'] = 'val'\n            kfold_df_dict[fold_]=df_temp\n        \n        df_comb_fold = pd.concat(kfold_df_dict[k] for (k,v) in kfold_df_dict.items())\n        \n        return df_comb_fold","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"input_path = \"/kaggle/input/siim-isic-melanoma-classification\"\nnum_splits = 2\ndf_actual_train = pd.read_csv(\"/kaggle/input/siim-isic-melanoma-classification/train.csv\")\ndf_kfold = stratification(input_path,num_splits).create_split()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Lets now define a class for the model with the aim to:\na.) defining the model with pre-trained weights\nb.) constructing a forward function that computes loss along with model output","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class model_efficientnetb7(torch.nn.Module):\n    def __init__(self, pretrained=True):\n        \n        super(model_efficientnetb7,self).__init__()\n        \n        self.model = EfficientNet.from_pretrained('efficientnet-b7')\n        ## Changing the last layer\n        num_ftrs = self.model._fc.in_features\n        self.model._fc = nn.Linear(num_ftrs, 1)\n        for param in self.model.parameters():\n            param.requires_grad = True\n    \n    def forward(self,image,targets):\n        # Arguments should match your dataloader arguments wrt dataset being passed\n        # in this case it is image, targets\n        \n        \n        out = self.model(image)\n        \n        loss = nn.BCEWithLogitsLoss()(\n            out, targets.reshape(-1,1).type_as(out)\n        )\n        # shape and datatype\n        return out,loss","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class ClassificationLoader:\n    def __init__(self, image_paths, targets, resize, augmentations):\n        self.image_paths = image_paths\n        self.targets = targets\n        self.resize = resize\n        self.augmentations = augmentations\n\n    def __len__(self):\n        return len(self.image_paths)\n\n    def __getitem__(self, item):\n        image = Image.open(self.image_paths[item])\n        targets = self.targets[item]\n        if self.resize is not None:\n            image = image.resize(\n                (self.resize[1], self.resize[0]), resample=Image.BILINEAR\n            )\n        image = np.array(image)\n        if self.augmentations is not None:\n            augmented = self.augmentations(image=image)\n            image = augmented[\"image\"]\n        image = np.transpose(image, (2, 0, 1)).astype(np.float32)\n        return {\n            \"image\": torch.tensor(image, dtype=torch.float),\n            \"targets\": torch.tensor(targets, dtype=torch.long),\n        }","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class AverageMeter:\n    \"\"\"\n    Computes and stores the average and current value\n    \"\"\"\n\n    def __init__(self):\n        self.val = 0\n        self.avg = 0\n        self.sum = 0\n        self.count = 0\n\n    def reset(self):\n        self.val = 0\n        self.avg = 0\n        self.sum = 0\n        self.count = 0\n\n    def update(self, val, n=1):\n        self.val = val\n        self.sum += val * n\n        self.count += n\n        self.avg = self.sum / self.count","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class EarlyStopping:\n    def __init__(self, patience=7, mode=\"max\", delta=0.0001):\n        self.patience = patience\n        self.counter = 0\n        self.mode = mode\n        self.best_score = None\n        self.early_stop = False\n        self.delta = delta\n        if self.mode == \"min\":\n            self.val_score = np.Inf\n        else:\n            self.val_score = -np.Inf\n\n    def __call__(self, epoch_score, model, model_path):\n        if self.mode == \"min\":\n            score = -1.0 * epoch_score\n        else:\n            score = np.copy(epoch_score)\n\n        if self.best_score is None:\n            self.best_score = score\n            self.save_checkpoint(epoch_score, model, model_path)\n        elif score < self.best_score + self.delta:\n            self.counter += 1\n            print(\n                \"EarlyStopping counter: {} out of {}\".format(\n                    self.counter, self.patience\n                )\n            )\n            if self.counter >= self.patience:\n                self.early_stop = True\n        else:\n            self.best_score = score\n            self.save_checkpoint(epoch_score, model, model_path)\n            self.counter = 0\n\n    def save_checkpoint(self, epoch_score, model, model_path):\n        if epoch_score not in [-np.inf, np.inf, -np.nan, np.nan]:\n            print(\n                \"Validation score improved ({} --> {}). Saving model!\".format(\n                    self.val_score, epoch_score\n                )\n            )\n            torch.save(model.state_dict(), model_path)\n        self.val_score = epoch_score","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class Engine:\n    @staticmethod\n    def train(\n        data_loader,\n        model,\n        optimizer,\n        device,\n        scheduler=None,\n        accumulation_steps=1\n    ):\n        \n        losses = AverageMeter()\n        predictions = []\n        model.train()\n        if accumulation_steps > 1:\n            optimizer.zero_grad()\n        #tk0 = tqdm(data_loader, total=len(data_loader), disable=use_tpu)\n        tk0 = tqdm(data_loader, total=len(data_loader),disable=False)\n        for b_idx, data in enumerate(tk0):\n            for key, value in data.items():\n                data[key] = value.to(device)\n            if accumulation_steps == 1 and b_idx == 0:\n                optimizer.zero_grad()\n            _, loss = model(**data)\n                   \n            with torch.set_grad_enabled(True):\n                loss.backward()\n                if (b_idx + 1) % accumulation_steps == 0:\n                    optimizer.step()\n                    if scheduler is not None:\n                        scheduler.step()\n                    if b_idx > 0:\n                        optimizer.zero_grad()\n            losses.update(loss.item(), data_loader.batch_size)\n            tk0.set_postfix(loss=losses.avg)\n        return losses.avg\n\n    @staticmethod\n    def evaluate(data_loader, model, device):\n        losses = AverageMeter()\n        final_predictions = []\n        model.eval()\n        with torch.no_grad():\n            #tk0 = tqdm(data_loader, total=len(data_loader), disable=use_tpu)\n            tk0 = tqdm(data_loader, total=len(data_loader), disable=False)\n            for b_idx, data in enumerate(tk0):\n                for key, value in data.items():\n                    data[key] = value.to(device)\n                predictions, loss = model(**data)\n                predictions = predictions.cpu()\n                losses.update(loss.item(), data_loader.batch_size)\n                final_predictions.append(predictions)\n                tk0.set_postfix(loss=losses.avg)\n        return final_predictions, losses.avg\n\n    @staticmethod\n    def predict(data_loader, model, device, use_tpu=False):\n        model.eval()\n        final_predictions = []\n        with torch.no_grad():\n            #tk0 = tqdm(data_loader, total=len(data_loader), disable=use_tpu)\n            tk0 = tqdm(data_loader, total=len(data_loader))\n            for b_idx, data in enumerate(tk0):\n                for key, value in data.items():\n                    data[key] = value.to(device)\n                predictions, _ = model(**data)\n                predictions = predictions.cpu()\n                final_predictions.append(predictions)\n        return final_predictions","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def train(fold):\n    print(f\"Starting Training for fold = {fold+1}\")\n    #Image size requirements per EfficientNet documentation\n    training_data_path = \"/kaggle/input/siic-isic-224x224-images/train\"\n    df = df_kfold[df_kfold['kfold']==fold]\n    device = 'cuda'\n    epochs = 50\n    train_bs = 4\n    val_bs = 4\n    \n    df_train = df.loc[df['dataset_type']=='train',list(df_actual_train.columns)]\n    df_val = df.loc[df['dataset_type']=='val',list(df_actual_train.columns)]\n    # Normalization needed as per EfficientNet documentation\n    mean = (0.485, 0.456, 0.406)\n    std = (0.229, 0.224, 0.225)\n    # add any extra augmentations here\n    train_aug = albumentations.Compose(\n        [\n            albumentations.Normalize(mean, std, max_pixel_value=255.0, always_apply=True,p=1.0)\n        ]\n    )\n    val_aug = albumentations.Compose(\n        [\n            albumentations.Normalize(mean, std, max_pixel_value=255.0, always_apply=True,p=1.0)\n        ]\n    )\n    train_images_list = df_train.image_name.values.tolist()\n    train_images = [os.path.join(training_data_path,i + '.png') for i in train_images_list]\n    train_targets = df_train.target.values\n    \n    val_images_list = df_val.image_name.values.tolist()\n    val_images = [os.path.join(training_data_path,i + '.png') for i in val_images_list]\n    val_targets = df_val.target.values\n    \n    train_dataset = ClassificationLoader(\n        image_paths = train_images,\n        targets= train_targets,\n        resize = None,\n        augmentations = train_aug\n    )\n    \n    train_loader = torch.utils.data.DataLoader(\n        train_dataset,\n        batch_size = train_bs,\n        shuffle = False,\n        num_workers=0\n    )\n    \n    val_dataset = ClassificationLoader(\n        image_paths = val_images,\n        targets= val_targets,\n        resize = None,\n        augmentations = val_aug\n    )\n    \n    val_loader = torch.utils.data.DataLoader(\n        val_dataset,\n        batch_size = val_bs,\n        shuffle = False,\n        num_workers=0\n    )\n    #Earlier defined class for model\n    #model = Model_Inception_v3(pretrained='imagenet')\n    model = model_efficientnetb7()\n    model.to(device)\n    \n    #Specify an optimizer\n    optimizer = torch.optim.Adam(model.parameters(), lr=1e-4)\n    \n    #Specify an scheduler\n    scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(\n        optimizer,\n        patience=3,\n        mode='max'\n    )\n    # why mode='max' becauase we will be using the metric of AUC\n    \n    # we would also need early stopping\n    es = EarlyStopping(patience=5, mode='max')\n    \n    for epoch in range(epochs):\n        training_loss = Engine.train(\n            train_loader,\n            model,\n            optimizer,\n            device\n        )\n        predictions, val_loss = Engine.evaluate(\n            val_loader,\n            model,\n            device\n        )\n        \n        predictions = np.vstack((predictions)).ravel()\n        # Ravel it because we have only one value\n        auc = metrics.roc_auc_score(val_targets, predictions)\n        # thats why val_loader shuffle was kept false\n        \n        scheduler.step(auc)\n        print(f\"epoch={epoch},auc={auc}\")\n        # Save it with .bin extension\n        model_path = f'efficient_model_fold{fold}.bin'\n        es(auc, model, model_path)\n        if es.early_stop:\n            print(\"Early Stopping\")\n            break","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def predict(fold):\n    print(f\"Generating Predictions for saved model, fold = {fold+1}\")\n    test_data_path = \"/kaggle/input/siic-isic-224x224-images/test\"\n    df_test = pd.read_csv(\"/kaggle/input/siim-isic-melanoma-classification/test.csv\")\n    df_test.loc[:,'target'] = 0\n    \n    #model_path = \"f'/kaggle/working/model_fold{fold}'\"\n    #model_path = '/kaggle/working/model_fold0_epoch0.bin'\n    model_path = f'/kaggle/working/efficient_model_fold{fold}.bin'\n    \n    device = 'cuda'\n    \n    test_bs = 16\n    \n    mean = (0.485, 0.456, 0.406)\n    std = (0.229, 0.224, 0.225)\n    \n    test_aug = albumentations.Compose(\n        [\n            albumentations.Normalize(mean, std, max_pixel_value=255.0, always_apply=True,p=1.0)\n        ]\n    )\n    test_images_list = df_test.image_name.values.tolist()\n    test_images = [os.path.join(test_data_path,i + '.png') for i in test_images_list]\n    test_targets = df_test.target.values\n    \n    test_dataset = ClassificationLoader(\n        image_paths = test_images,\n        targets= test_targets,\n        resize = None,\n        augmentations = test_aug\n    )\n    \n    test_loader = torch.utils.data.DataLoader(\n        test_dataset,\n        batch_size = test_bs,\n        shuffle = False,\n        num_workers=4\n    )\n    #Earlier defined class for model\n    model = model_efficientnetb7()\n    model.load_state_dict(torch.load(model_path))\n    model.to(device)\n    \n    predictions_op = Engine.predict(\n        test_loader,\n        model,\n        device\n    )\n    return np.vstack((predictions_op)).ravel()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Training\nfor fold_ in range(num_splits):\n    torch.cuda.empty_cache()\n    train(fold=fold_)\n    list_fold_pred.append(predict(fold_))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Generating Predictions\nlist_fold_pred = []\nfor fold_ in range(num_splits):    \n    list_fold_pred.append(predict(fold_))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred = np.mean(np.vstack(list(list_fold_pred[i] for i in range(1))),axis=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample = pd.read_csv(\"../input/siim-isic-melanoma-classification/sample_submission.csv\")\nsample.loc[:, \"target\"] = pred\nsample.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Going Forward\n1. Since the intent here was primarily to show how a framework can be leverage and you can incorporate other models with minimum adjustments needed for the model - probably basis the library you are using (e.g. pretrainedmodels, pytorch or any others), # of features in the last layers, normalization parameters etc.\n\n2. We can add extra augmentations to train and validate the model on a variety of type of images\n\n3. Incorporate patient data as well with this","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}