{"cells":[{"metadata":{},"cell_type":"markdown","source":"> The visual transformer (ViT) introduced by Dosovitskiy et al is an architecture directly inherited from Natural Language Processing , but applied to image classification with raw image patches as input. Their paper\n> presents promising results with transformers trained with a large strongly supervised image dataset that is not publicly\n> available. The authors concluded that transformers ”do not generalize well when\n> trained on insufficient amounts of data”, and used extensive computing resources\n> to train their models. Both aspects limit the adoption of ViT and more generally\n> of transformers, for researchers without access to such computing resources, or\n> without a privileged access to a large private dataset.\n> In this paper, we show that none of this is required: we actually train a\n> transformer on a single 8-GPU node in two to three days (53 hours of pretraining, and optionally 20 hours of fine-tuning). This vanilla transformer is\n> competitive with convnets of a similar number of parameters and efficiencyusing Imagenet as the sole training set, and does not include a single convolution. We build upon the visual transformer architecture from Dosovitskiy et\n> al, which is very close to the original token-based transformer architecture where word embeddings are replaced with patch embeddings. With\n> our Data-efficient image Transformers (DeiT), we report large improvements\n> over previous results, see Figure 1. They mainly come from DeiT’s better training strategy for visual transformers, at both the initial training and the finetuning stage. Our ablation study details the key ingredients for a successful\n> training, and hopefully will serve as guidelines for future works.\n\n"},{"metadata":{},"cell_type":"markdown","source":"![](https://github.com/facebookresearch/deit/raw/main/.github/deit.png)","attachments":{}},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install -q timm==0.3.2","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport cv2\nimport json\nimport numpy as np\nimport pandas as pd\nfrom functools import partial\n\nimport matplotlib.pyplot as plt\n\n\nimport torch\nfrom torch import nn\nimport albumentations as A\n\nfrom timm.utils import accuracy\nfrom timm.data import Mixup\nfrom timm.models import create_model\nfrom timm.loss import LabelSmoothingCrossEntropy, SoftTargetCrossEntropy\nfrom timm.scheduler import create_scheduler\nfrom timm.optim import create_optimizer\nfrom timm.utils import NativeScaler, get_state_dict, ModelEma\nfrom timm.models.registry import register_model\nfrom timm.models.vision_transformer import VisionTransformer, _cfg\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train_df = pd.read_csv('../input/cassava-leaf-disease-classification/train.csv')\nprint('Shape of Train df', train_df.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Spliting the dataset into train and validate**"},{"metadata":{"trusted":true},"cell_type":"code","source":"val_df = pd.DataFrame()\nfor l in train_df.label.unique():\n    val_df = pd.concat([train_df[train_df.label == l].sample(400), val_df])\ntrain_df = train_df[~train_df.isin(val_df)].dropna()\nprint(val_df.shape, train_df.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is a problem of imbalanced dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.label.plot.hist(bins=25);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**SIMPLE DATA OVERSAMPLING**"},{"metadata":{"trusted":true},"cell_type":"code","source":"def return_df(df, c):\n    x = (c // df.shape[0] ) + 1\n    return pd.concat([df]*x).iloc[:c]\n\n\n### SIMPLE DATA OVERSAMPLING\nMAX_COUNT = 13200\nlabel = np.unique(train_df.label)\ntrain_df_modified = pd.DataFrame([], columns = ['image_id', 'label'])\nfor l in label:\n    df = return_df(train_df[train_df.label == l], MAX_COUNT)\n    train_df_modified = pd.concat([train_df_modified, df])\ntrain_df_modified.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**TRAINSET AFTER SAMPLING**"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df_modified.label.plot.hist(bins=25);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**VALIDATION SET AFTER SAMPLING**"},{"metadata":{"trusted":true},"cell_type":"code","source":"val_df.label.plot.hist(bins=25);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\n\nimport matplotlib.pyplot as plt\n\nwith open('../input/cassava-leaf-disease-classification/label_num_to_disease_map.json', 'r') as file:\n    label_map = json.load(file)\n\ndef show_examples(images):\n    _indexes = [(i, j) for i in range(4) for j in range(4)]\n    \n    f, ax = plt.subplots(4, 4, figsize=(16, 16))\n    for (img, title), (i, j) in zip(images, _indexes):\n        ax[i, j].imshow(img)\n        ax[i, j].set_title(title)\n    f.tight_layout()\n    plt.show()\n\ndef read_random_images(df):\n    data = df.sample(16)\n    Images, Label = data.values[:,0], data.values[:,1].astype(int)\n    Path = '../input/cassava-leaf-disease-classification/train_images/'\n    result = []\n    for d, label in zip(Images, Label):\n        title = f\"Label:{label_map[str(label)]}\"\n        _image = cv2.imread(Path + d)[:,:,::-1]\n        result.append((_image, title))\n    print('Showing Sample ...')\n    show_examples(result)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"label_map","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"read_random_images(train_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nclass LeafClassificationDataset(object):\n    \n    def __init__(self, df, \n                 transform = None, \n                 path = '../input/cassava-leaf-disease-classification/train_images'):\n        \n        self.path = path\n        self.df = df\n        self.transform = transform\n\n    def __len__(self):\n        return self.df.shape[0]\n    \n    def __getitem__(self, idx):\n        im = self.df.iloc[idx].image_id\n        img = self.img_loader(im)\n        label = int(self.df.iloc[idx, 1])\n        return img, torch.tensor(label)\n    \n    \n    def img_loader(self,name):\n        path = os.path.join(self.path, name)\n        image = cv2.imread(path)\n        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n        if self.transform:\n            image = self.transform(image = image)['image']\n        image = torch.as_tensor(np.ascontiguousarray(image.transpose(2, 0, 1)))\n        \n        return image\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**DATA AUGMENTATION**"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Declare an augmentation pipeline\nimage_size = 224\ntrain_transforms = A.Compose([\n        A.LongestMaxSize(max_size=image_size),\n        A.PadIfNeeded(image_size, image_size, border_mode=2),\n        A.RandomRotate90(),\n        A.Flip(),\n        A.RandomBrightnessContrast(brightness_limit=0.4, contrast_limit=0.4, p=0.7),\n        A.Transpose(),\n        A.OneOf([\n            A.IAAAdditiveGaussianNoise(),\n            A.GaussNoise(),\n        ], p=0.2),\n        A.OneOf([\n            A.MotionBlur(p=.2),\n            A.MedianBlur(blur_limit=3, p=0.6),\n            A.Blur(blur_limit=3, p=0.1),\n        ], p=0.7),\n        A.ShiftScaleRotate(shift_limit=0.0625, scale_limit=0.2, rotate_limit=45, p=0.7),\n        A.OneOf([\n            A.OpticalDistortion(p=0.3),\n            A.GridDistortion(p=.1),\n            A.IAAPiecewiseAffine(p=0.3),\n        ], p=0.5),\n        A.OneOf([\n            A.CLAHE(clip_limit=2),\n            A.IAASharpen(),\n            A.IAAEmboss(),\n            A.RandomBrightnessContrast(),            \n        ], p=0.5),\n        A.HueSaturationValue(p=0.5),\n        A.Normalize()\n    ])\nvalid_transforms = A.Compose([\n        A.LongestMaxSize(max_size=image_size),\n        A.PadIfNeeded(image_size, image_size, border_mode=2),\n        A.Normalize()\n    ])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**DATALOADERS**"},{"metadata":{"trusted":true},"cell_type":"code","source":"import collections\nfrom sklearn.model_selection import train_test_split\n\nfrom torch.utils.data import DataLoader\n\ndef get_loaders(train_df, val_df, batch_size = 32,num_workers=0,train_transforms_fn = None,valid_transforms_fn = None):\n\n    # Creates our train dataset\n    train_dataset = LeafClassificationDataset( train_df, train_transforms_fn)\n\n\n    # Creates our valid dataset\n    valid_dataset = LeafClassificationDataset( val_df, valid_transforms_fn)\n\n    train_loader = DataLoader(\n      train_dataset,\n      batch_size=batch_size,\n      shuffle=True,\n      num_workers=num_workers,\n      drop_last=True,\n    )\n\n    valid_loader = DataLoader(\n      valid_dataset,\n      batch_size=batch_size,\n      shuffle=False,\n      num_workers=num_workers,\n      drop_last=True,\n    )\n\n    loaders = collections.OrderedDict()\n    loaders[\"train\"] = train_loader\n    loaders[\"valid\"] = valid_loader\n\n    return loaders","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"batch_size = 180\n\nprint(f\"batch_size: {batch_size}\")\n\nloaders = get_loaders( train_df_modified, val_df,batch_size,0, train_transforms, valid_transforms)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**REGISTERING THE MODEL**\n\n\nPatch size is 16 and Image Size is 224"},{"metadata":{"trusted":true},"cell_type":"code","source":"@register_model\ndef deit_small_patch16_224(pretrained=False, **kwargs):\n    model = VisionTransformer(img_size=224,\n        patch_size=16, embed_dim=384, depth=12, num_heads=6, mlp_ratio=4, qkv_bias=True,\n        norm_layer=partial(nn.LayerNorm, eps=1e-6), **kwargs)\n    d_c = _cfg()\n    d_c['input_size'] = (3,224,224)\n    model.default_cfg = d_c\n    if pretrained:\n        checkpoint = torch.hub.load_state_dict_from_url(\n            url=\"https://dl.fbaipublicfiles.com/deit/deit_small_patch16_224-cd65a155.pth\",\n            map_location=\"cpu\", check_hash=True\n        )\n        checkpoint[\"model\"].pop('head.weight')\n        checkpoint[\"model\"].pop('head.bias')\n        \n        model.load_state_dict(checkpoint[\"model\"], strict=False)\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from timm.models import create_model\nmodel = create_model(\n    'deit_small_patch16_224',\n    pretrained=True,\n    num_classes=5)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_ema = ModelEma(\n            model,\n            decay=0.99,\n            device='cuda')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"n_parameters = sum(p.numel() for p in model.parameters() if p.requires_grad)\nprint('number of params:', n_parameters)\n\nlinear_scaled_lr = 0.0001 * batch_size * 1 / 512.0\nlr = linear_scaled_lr\noptimizer  = torch.optim.Adam(model.parameters(), lr=lr)\nloss_scaler = NativeScaler()\n\nlr_scheduler =torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, 0.001)\n\ncriterion = nn.CrossEntropyLoss()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mixup_fn = Mixup(\n            mixup_alpha=0.2, cutmix_alpha=0.03, \n            prob=0.8, switch_prob=0.3, mode= 'batch', num_classes=5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"@torch.no_grad()\ndef evaluate(data_loader, model, device):\n    criterion = torch.nn.CrossEntropyLoss()\n\n\n    # switch to evaluation mode\n    model.eval()\n    acc1_mean = 0.0\n    acc5_mean = 0.0\n    Batch = len(data_loader)\n    print('Total Batch:', Batch)\n    \n    for images, target in data_loader:\n        images = images.to(device, non_blocking=True)\n        target = target.to(device, non_blocking=True)\n\n        # compute output\n        with torch.cuda.amp.autocast():\n            output = model(images)\n            loss = criterion(output, target)\n\n        acc1, acc5 = accuracy(output, target, topk=(1, 5))\n        acc1_mean+=acc1.item()\n        acc5_mean+=acc5.item()\n        \n\n        batch_size = images.shape[0]\n    print('* Acc@1 {:.3f} Acc@5 {:.3f} loss {:.3f}'\n          .format(acc1_mean/Batch, acc5_mean/Batch, loss.item()))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import math\nfrom datetime import datetime\n\ndef train_one_epoch(model, criterion,\n                    data_loader, optimizer,\n                    device, epoch, loss_scaler, max_norm,\n                    model_ema, mixup_fn):\n\n    model.train()\n    model.to(device)\n    header = 'Epoch: [{}]'.format(epoch)\n    print_freq = 50\n    count = 0\n    Batch = len(data_loader)\n    print('Total Batch:', Batch)\n    loss_mean = 0.0\n    for samples, targets in data_loader:\n        count +=1\n        samples = samples.to(device, non_blocking=True)\n        targets = targets.to(device, non_blocking=True)\n\n#         if mixup_fn is not None:\n#             samples, _ = mixup_fn(samples, targets)\n\n            \n\n        with torch.cuda.amp.autocast():\n            outputs = model(samples)\n            \n            loss = criterion(outputs, targets)\n\n        loss_value = loss.item()\n        loss_mean += loss_value \n\n        if not math.isfinite(loss_value):\n            print(\"Loss is {}, stopping training\".format(loss_value))\n            sys.exit(1)\n\n        optimizer.zero_grad()\n\n        # this attribute is added by timm on one optimizer (adahessian)\n        is_second_order = hasattr(optimizer, 'is_second_order') and optimizer.is_second_order\n        loss_scaler(loss, optimizer, clip_grad=max_norm,\n                    parameters=model.parameters(), create_graph=is_second_order)\n\n        torch.cuda.synchronize()\n        if model_ema is not None:\n            model_ema.update(model)\n\n        if count % print_freq == 0:\n            current_time = datetime.now().strftime(\"%H:%M:%S\")\n            print(f'{current_time}\\tEPOCH: [{epoch}/10] STEP: [{count}/{Batch}], LOSS: {loss_mean/50}')\n            loss_mean = 0.0\n    torch.save({'model': model.state_dict(),\n                'optimizer': optimizer.state_dict(),\n                'lr_scheduler': lr_scheduler.state_dict(),\n                'epoch': epoch,\n                'model_ema': get_state_dict(model_ema),\n                }, 'checkpoint.pth')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Start training\")\nmax_accuracy = 0.0\nfor epoch in range(0, 7):\n\n    train_stats = train_one_epoch(\n        model, criterion, loaders['train'],\n        optimizer, 'cuda', epoch, loss_scaler,\n        None, model_ema, mixup_fn\n    )\n\n    lr_scheduler.step(epoch)\n    evaluate( loaders['valid'], model, 'cuda')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}