{"cells":[{"metadata":{},"cell_type":"markdown","source":"# About this Notebook\nHi fellow Kagglers,  \nI have alreadt written a detailed EDA and Baseline model kernel [here](https://www.kaggle.com/manabendrarout/detailed-eda-baseline-model-plant-pathology-21).  \nDo check it out before this kernel. That will set the context for what I am doing in this kernel and Why am I doing it that way.  \n\nThis kernel will mainly focus on model creation and getting a decent score. Contrary to the starter Kernel above, we will use Pytorch here just because of the simplicity with prototyping and iterations.\n\n# Imports"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"import sys\nsys.path.append('../input/pytorch-image-models/pytorch-image-models-master')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Asthetics\nimport warnings\nwarnings.filterwarnings('ignore', category=DeprecationWarning)\nwarnings.filterwarnings('ignore', category=FutureWarning)\n\n# General\nimport pandas as pd\npd.set_option('display.max_columns', None)\nimport numpy as np\nimport os\nfrom tqdm import tqdm\nfrom collections import defaultdict\n\n# Visialisation\nfrom PIL import Image\nimport cv2\n\n# Pre Processing\nfrom sklearn.preprocessing import LabelEncoder\nfrom sklearn.model_selection import train_test_split\n\n# Metric\nfrom sklearn.metrics import f1_score, accuracy_score\n\n# Deep Learning\nimport torch\nimport torchvision\nimport timm\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nfrom torch.utils.data import Dataset, DataLoader\nimport pytorch_lightning as pl\n\n# Augmentation\nimport albumentations\nfrom albumentations.pytorch.transforms import ToTensorV2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if torch.cuda.is_available():\n    device = torch.device('cuda')\nelse:\n    device = torch.device('cpu')\n    \nprint(f'Using device: {device}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"RANDOM_SEED = 42","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def seed_everything(seed=RANDOM_SEED):\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\n    torch.backends.cudnn.benchmark = True","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"seed_everything()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As you all might be aware, the original image resolutions are very high; thus require lot of time for training and iterating.  \nThanks to Ankur Singh for resizing the images which speeds up the times by upto 7x, thus we will use [his dataset](https://www.kaggle.com/ankursingh12/resized-plant2021) in this kernel. If you like this Kernel, please make sure to support him as well because it is due to his efforts we are able to train this fast here."},{"metadata":{"trusted":true},"cell_type":"code","source":"data_path = '../input/plant-pathology-2021-fgvc8'\n\nlabels_file_path = os.path.join(data_path, 'train.csv')\nsample_submission_path = os.path.join(data_path, 'sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df = pd.read_csv(labels_file_path)\nsub_df = pd.read_csv(sample_submission_path)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"le = LabelEncoder()\nle.fit(df['labels']);\ndf['labels'] = le.transform(df['labels'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"label_map = dict(zip(le.classes_, le.transform(le.classes_)))\nlabel_inv_map = {v: k for k, v in label_map.items()}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"(train_img, valid_img, train_labels, valid_labels) = train_test_split(df['image'],\n                                                                      df['labels'],\n                                                                      test_size=0.2,\n                                                                      stratify=df['labels'],\n                                                                      random_state=RANDOM_SEED)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_paths = '../input/resized-plant2021/img_sz_384/' + train_img\nvalid_paths = '../input/resized-plant2021/img_sz_384/' + valid_img\ntest_paths = '../input/plant-pathology-2021-fgvc8/test_images/' + sub_df['image']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Augmentation"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_train_transforms():\n    return albumentations.Compose(\n        [\n            albumentations.Resize(256,256),\n            albumentations.HorizontalFlip(p=0.5),\n            albumentations.VerticalFlip(p=0.5),\n            albumentations.Rotate(limit=180, p=0.7),\n            albumentations.RandomBrightness(limit=0.6, p=0.5),\n            albumentations.Cutout(\n                num_holes=8, max_h_size=8, max_w_size=8,\n                fill_value=0, always_apply=False, p=0.5\n            ),\n            albumentations.ShiftScaleRotate(\n                shift_limit=0.25, scale_limit=0.1, rotate_limit=0\n            ),\n            albumentations.Normalize(\n                [0.485, 0.456, 0.406], [0.229, 0.224, 0.225],\n                max_pixel_value=255.0, always_apply=True\n            ),\n            ToTensorV2(p=1.0),\n        ]\n    )\n\ndef get_valid_transforms():\n    return albumentations.Compose(\n        [\n            albumentations.Resize(256,256),\n            albumentations.Normalize(\n                [0.485, 0.456, 0.406], [0.229, 0.224, 0.225],\n                max_pixel_value=255.0, always_apply=True\n            ),\n            ToTensorV2(p=1.0)\n        ]\n    )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"class AppleDataset(Dataset):\n    def __init__(self, images_filepaths, labels, transform=None):\n        self.images_filepaths = images_filepaths\n        self.labels = labels\n        self.transform = transform\n\n    def __len__(self):\n        return len(self.images_filepaths)\n\n    def __getitem__(self, idx):\n        image_filepath = self.images_filepaths[idx]\n        image = cv2.imread(image_filepath)\n        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n        label = self.labels[idx]\n        if self.transform is not None:\n            image = self.transform(image=image)[\"image\"]\n        return image, label","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dataset = AppleDataset(images_filepaths=train_paths.values,\n                             labels=train_labels.values,\n                             transform=get_train_transforms())\nvalid_dataset = AppleDataset(images_filepaths=valid_paths.values,\n                             labels=valid_labels.values,\n                             transform=get_valid_transforms())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Metrics"},{"metadata":{"trusted":true},"cell_type":"code","source":"def multi_acc(output, target):\n    y_pred = torch.softmax(output, dim = 1)\n    y_pred = torch.argmax(y_pred, dim=1).cpu()\n    target = target.cpu()\n    \n    return accuracy_score(target, y_pred)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def calculate_f1_macro(output, target):\n    y_pred = torch.softmax(output, dim = 1)\n    y_pred = torch.argmax(y_pred, dim=1).cpu()\n    target = target.cpu()\n    \n    return f1_score(target, y_pred, average='macro')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class MetricMonitor:\n    def __init__(self, float_precision=3):\n        self.float_precision = float_precision\n        self.reset()\n\n    def reset(self):\n        self.metrics = defaultdict(lambda: {\"val\": 0, \"count\": 0, \"avg\": 0})\n\n    def update(self, metric_name, val):\n        metric = self.metrics[metric_name]\n\n        metric[\"val\"] += val\n        metric[\"count\"] += 1\n        metric[\"avg\"] = metric[\"val\"] / metric[\"count\"]\n\n    def __str__(self):\n        return \" | \".join(\n            [\n                \"{metric_name}: {avg:.{float_precision}f}\".format(\n                    metric_name=metric_name, avg=metric[\"avg\"],\n                    float_precision=self.float_precision\n                )\n                for (metric_name, metric) in self.metrics.items()\n            ]\n        )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"params = {\n    'model': 'efficientnet_b3',\n    'device': device,\n    'lr': 0.001,\n    'batch_size': 32,\n    'num_workers' : 0,\n    'epochs': 7,\n    'out_features': df['labels'].nunique()\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_loader = DataLoader(\n    train_dataset, batch_size=params['batch_size'], shuffle=True,\n    num_workers=params['num_workers'], pin_memory=True,\n)\n\nval_loader = DataLoader(\n    valid_dataset, batch_size=params['batch_size'], shuffle=False,\n    num_workers=params['num_workers'], pin_memory=True,\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class AppleNet(nn.Module):\n    def __init__(self, model_name=params['model'], out_features=params['out_features'],\n                 pretrained=True):\n        super().__init__()\n        self.model = timm.create_model(model_name, pretrained=pretrained)\n        n_features = self.model.classifier.in_features\n        self.model.classifier = nn.Linear(n_features, out_features)\n    \n    def forward(self, x):\n        x = self.model(x)\n        return x","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"model = AppleNet()\nmodel = model.to(params['device'])\ncriterion = nn.CrossEntropyLoss().to(params['device'])\noptimizer = torch.optim.Adam(model.parameters(), lr=params['lr'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Train and Validation"},{"metadata":{"trusted":true},"cell_type":"code","source":"def train(train_loader, model, criterion, optimizer, epoch, params):\n    metric_monitor = MetricMonitor()\n    model.train()\n    stream = tqdm(train_loader)\n    for i, (images, target) in enumerate(stream, start=1):\n        images = images.to(params['device'], non_blocking=True)\n        target = target.to(params['device'], non_blocking=True)\n        output = model(images)\n        loss = criterion(output, target)\n        f1_macro = calculate_f1_macro(output, target)\n        accuracy = multi_acc(output, target)\n        metric_monitor.update('Loss', loss.item())\n        metric_monitor.update('F1', f1_macro)\n        metric_monitor.update('Accuracy', accuracy)\n        optimizer.zero_grad()\n        loss.backward()\n        optimizer.step()\n        stream.set_description(\n            \"Epoch: {epoch}. Train.      {metric_monitor}\".format(\n                epoch=epoch,\n                metric_monitor=metric_monitor)\n        )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def validate(val_loader, model, criterion, epoch, params):\n    metric_monitor = MetricMonitor()\n    model.eval()\n    stream = tqdm(val_loader)\n    with torch.no_grad():\n        for i, (images, target) in enumerate(stream, start=1):\n            images = images.to(params['device'], non_blocking=True)\n            target = target.to(params['device'], non_blocking=True)\n            output = model(images)\n            loss = criterion(output, target)\n            f1_macro = calculate_f1_macro(output, target)\n            accuracy = multi_acc(output, target)\n            metric_monitor.update('Loss', loss.item())\n            metric_monitor.update('F1', f1_macro)\n            metric_monitor.update('Accuracy', accuracy)\n            stream.set_description(\n                \"Epoch: {epoch}. Validation. {metric_monitor}\".format(\n                    epoch=epoch,\n                    metric_monitor=metric_monitor)\n            )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for epoch in range(1, params['epochs'] + 1):\n    train(train_loader, model, criterion, optimizer, epoch, params)\n    validate(val_loader, model, criterion, epoch, params)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Save Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"torch.save(model.state_dict(), f\"{params['model']}_{params['epochs']}epochs_weights.pth\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Prediction\nI am not actively participating in this competition, thus I am not going to submit this notebook and see my LB score. But anyway this is meant to be a starter Kernel, so little does it matter.  \n\nBut I will laydown the foundation for anyone who is interested. So, let's predict and prepare the submission dataframe."},{"metadata":{"trusted":true},"cell_type":"code","source":"def multi_acc(output, target):\n    y_pred = torch.softmax(output, dim = 1)\n    y_pred = torch.argmax(y_pred, dim=1).cpu()\n    target = target.cpu()\n    \n    return accuracy_score(target, y_pred)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"labels = np.zeros(len(test_paths)) # Fake Labels\ntest_dataset = AppleDataset(images_filepaths=test_paths,\n                            labels = labels,\n                            transform=get_valid_transforms())\ntest_loader = DataLoader(\n    test_dataset, batch_size=params['batch_size'], shuffle=False,\n    num_workers=params['num_workers'], pin_memory=True\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.eval()\npredicted_labels = []\nwith torch.no_grad():\n    for (images, target) in test_loader:\n        images = images.to(params['device'], non_blocking=True)\n        output = model(images)\n        predictions = torch.softmax(output, dim = 1)\n        predictions = torch.argmax(predictions, dim=1).cpu().numpy()\n        predicted_labels += list(predictions)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub_df['labels'] = predicted_labels\nsub_df['labels'] = sub_df['labels'].map(label_inv_map)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub_df.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is a simple starter kernel on implementation of Transfer Learning using Pytorch for this problem.\nPytorch has many SOTA Image models which you can try out using the guidelines in this notebook.\n\nI hope you have learnt something from this notebook. I have created this notebook as a baseline model, which you can easily fork and paly-around with to get much better results. I might update parts of it down the line when I get more GPU hours and some interesting ideas.\n\n**If you liked this notebook and use parts of it in you code, please show some support by upvoting this kernel. It keeps me inspired to come-up with such starter kernels and share it with the community.**\n\nThanks and happy kaggling!"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}