{"cells":[{"metadata":{},"cell_type":"markdown","source":"This kernel shows an example for EDA and data augmentations inspired by following two kernels.\n* https://www.kaggle.com/artgor/basic-eda-and-baseline-pytorch-model\n* https://www.kaggle.com/abhishek/pytorch-inference-kernel-lazy-tta\n* https://github.com/EthanRosenthal/spacecutter"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport torchvision\nimport torch.nn as nn\nfrom tqdm import tqdm\nfrom PIL import Image, ImageFile\nfrom torch.utils.data import Dataset\nimport torch\nfrom torchvision import transforms\nimport os\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport collections.abc\nfrom sklearn.model_selection import train_test_split\nimport re\nfrom typing import Optional\n\n\ndevice = torch.device(\"cuda:0\")\nImageFile.LOAD_TRUNCATED_IMAGES = True\nepochs = 15","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from albumentations import (\n    HorizontalFlip, IAAPerspective, ShiftScaleRotate, CLAHE, RandomRotate90,\n    Transpose, ShiftScaleRotate, Blur, OpticalDistortion, GridDistortion, HueSaturationValue,\n    IAAAdditiveGaussianNoise, GaussNoise, MotionBlur, MedianBlur, IAAPiecewiseAffine,\n    IAASharpen, IAAEmboss, RandomContrast, RandomBrightness, Flip, OneOf, Compose, RandomGamma, \n    ElasticTransform, ChannelShuffle,RGBShift, Rotate\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"First, let's try to visualize how the images look like within test/training set."},{"metadata":{"trusted":true},"cell_type":"code","source":"\nfor i,path in enumerate(os.listdir('../input/aptos2019-blindness-detection/test_images')):\n    img_path = os.path.join('../input/aptos2019-blindness-detection/test_images',path)\n    im = Image.open(img_path,'r')\n    ax = plt.subplot(3,3,i + 1)\n    ax.imshow(im)\n    if i == 8:\n        break","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nfor i,path in enumerate(os.listdir('../input/aptos2019-blindness-detection/train_images')):\n    img_path = os.path.join('../input/aptos2019-blindness-detection/train_images',path)\n    im = Image.open(img_path,'r')\n    ax = plt.subplot(3,3,i + 1)\n    ax.imshow(im)\n    if i == 8:\n        break","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Okay, it looks like there is a great variety in shape and color in both of train/test dataset.\nTherefore, we would like to investigate the effect of data augmentation. Especially, considering we have different sizes/aspect ratios, we have to crop the images no matter what.\nOne of the topics I would look into is the center crop vs random crop.\n\n  \n"},{"metadata":{},"cell_type":"markdown","source":"## Class balance\nAlso, let's see the class balance within the training set.\nIf there is a huge class imbalance, the likelihood is they are optimized to predict specific class(es), and we definetely would want to avoid that."},{"metadata":{"trusted":true},"cell_type":"code","source":"training_class = pd.read_csv('../input/aptos2019-blindness-detection/train.csv')\nsns.countplot(training_class['diagnosis'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Not that terrible. but probably should consider correcting the balance still.\nThe strategy to correct the balance here is to simply apply data augmentation more on images of less frequent classes. "},{"metadata":{"trusted":true},"cell_type":"code","source":"def data_transforms(mode = 'random',img_size = 256):\n    general_aug = Compose([\n        OneOf([\n            Transpose(),\n            HorizontalFlip(),\n            RandomRotate90()\n            ]),\n        ShiftScaleRotate(shift_limit=0.0625, scale_limit=0.2, rotate_limit=15, p=.2),\n        OneOf([\n            OpticalDistortion(p=0.2),\n            GridDistortion(distort_limit=0.2, p=.1),\n            ElasticTransform(),\n            ], p=1.)\n        ], p=1)\n    image_specific = Compose([\n        OneOf([\n            IAASharpen(),\n            RandomContrast(),\n            RandomBrightness(),\n            ], p=0.3)\n        ])\n    all_transf_pre = [\n            transforms.RandomCrop(round(1.2 * img_size))\n            ]\n\n    all_trans_after = [\n            transforms.CenterCrop(img_size)\n            ]\n    center_crop = [\n            transforms.CenterCrop(img_size)\n    ]\n    normalize = [\n            transforms.ToTensor()\n            ]\n\n    def get_augment(aug):\n        def augment(image):\n            return Image.fromarray(aug(image=np.array(image))['image'])\n        return [augment]\n\n    def normalize_to_full_image(img):\n        return img\n        #img = np.array(img).astype(np.float32)\n        #img -= img.min()\n        #img /= img.max()\n        #img *= 255\n        #return img.astype(np.uint8)\n\n    pre_crop = transforms.Compose(all_transf_pre) \n    train_img_transform = transforms.Compose(get_augment(general_aug) + get_augment(image_specific) + [normalize_to_full_image])\n    norm_transform = transforms.Compose(all_trans_after + normalize)\n    val_transform = transforms.Compose(all_trans_after) if mode == 'random' else transforms.Compose(center_crop)\n\n    return pre_crop, train_img_transform, norm_transform, val_transform","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For the augmentation, I adopted a variety of techniques, which can be categorized  as follows.\n1. Shape transformation\nThis includes affine transformation, such as rotation, shifting and scaling, flipping and nonlinear transformation.    \n2. Color transformation\nThis includes the change in brightness and contrast. \n\nIn this kernel, I first resized the image to set the aspect ratio of each examples equal and then crops twice before and after applying any data augmentation to keep the black region resulting from data augmentation as small as possible.\nBelow are some examples of augmented samples"},{"metadata":{"trusted":true},"cell_type":"code","source":"pre_crop, train_img_transform, _, center_crop = data_transforms()\nfig = plt.figure()\nfor i,path in enumerate(os.listdir('../input/aptos2019-blindness-detection/train_images')):\n    img_path = os.path.join('../input/aptos2019-blindness-detection/train_images',path)\n    im = Image.open(img_path,'r')\n    im = pre_crop(im.resize((320, 320), resample=Image.BILINEAR))\n    ax = fig.add_subplot(4,2,2 * i + 1)\n    ax.imshow(center_crop(im))\n    ax.set_title('original')\n    ax = fig.add_subplot(4,2,2 * i + 2)\n    ax.set_title('augmented')\n    im = train_img_transform(im)\n    ax.imshow(center_crop(im))\n    \n    if i == 3:\n        break\nfig.suptitle('original vs augmented')\nfig.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"class RetinopathyDatasetTest(Dataset):\n    def __init__(self, data,mode = 'test'):\n        #self.data = pd.read_csv(csv_file)\n        self.mode = mode\n        self.data = data.reset_index()\n        self.img_dir = '../input/aptos2019-blindness-detection/test_images' if mode == 'test' else '../input/aptos2019-blindness-detection/train_images' \n        _,_,_,self.transform = data_transforms('center') \n        \n    def __len__(self):\n        return len(self.data)\n\n    def __getitem__(self, idx):\n        img_name = os.path.join(self.img_dir, self.data.loc[idx, 'id_code'] + '.png')\n        image = Image.open(img_name)\n        image = image.resize((320, 320), resample=Image.BILINEAR)\n        image = self.transform(image)\n        if self.mode == 'test':\n            return {'image': transforms.ToTensor()(image)}\n        else:\n            return {'image': transforms.ToTensor()(image),'label': self.data.loc[idx,'diagnosis']}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#trainds = RetinopathyDatasetTrain('../input/aptos2019-blindness-detection/train.csv')\n#valds = RetinopathyDatasetTest('../input/aptos2019-blindness-detection/train.csv')\ntrain_df = pd.read_csv('../input/aptos2019-blindness-detection/train.csv')\ntr, val = train_test_split(train_df, stratify=train_df.diagnosis, test_size=0.05)\n#train_sampler = SubsetRandomSampler(list(tr.index))\n#val_sampler = SubsetRandomSampler(list(val.index))\n# prepare data loaders (combine dataset and sampler)\n#train_loader = torch.utils.data.DataLoader(trainds, batch_size=32, sampler=train_sampler, num_workers=4)\n#val_loader = torch.utils.data.DataLoader(valds, batch_size=32, sampler=val_sampler, num_workers=4)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def this_collate_fn(batch):\n    elem = batch[0]\n    return {key:torch.cat([d[key] for d in batch],dim = 0) for key in elem} ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class RetinopathyDatasetTrain(Dataset):\n    def __init__(self, data ,img_size = 224):\n        #self.data = pd.read_csv(csv_file)\n        self.data = data.reset_index()\n        most_freq_class_num = len(self.data.query('diagnosis == 0'))\n        self.aug_times = {str(diagnosis):np.round(most_freq_class_num / len(self.data.query('diagnosis == ' + str(diagnosis)))) for diagnosis in self.data['diagnosis'].unique()}\n        \n        #self.images = [Image.open(os.path.join('../input/aptos2019-blindness-detection/train_images',path),'r') for i,path in enumerate(os.listdir('../input/aptos2019-blindness-detection/train_images'))] \n    def __len__(self):\n        return len(self.data)\n\n    def __getitem__(self, idx):\n        self.pre_crop, self.train_img_transform, self.norm_transform, _ = data_transforms()\n        img_name = os.path.join('../input/aptos2019-blindness-detection/train_images', self.data.loc[idx, 'id_code'] + '.png')\n        image = Image.open(img_name)\n        image = self.pre_crop(image.resize((320, 320), resample=Image.BILINEAR))\n        key = str(self.data.loc[idx,'diagnosis'])\n        aug_time = int(self.aug_times[key])\n        img_list = [self.norm_transform(image)] + [self.norm_transform(self.train_img_transform(image)) for i in range(aug_time)]\n        labels = [torch.tensor(self.data.loc[idx,'diagnosis'])] * len(img_list)\n        return {'image': torch.stack(img_list),'label':torch.stack(labels,dim = 0)}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Since internet connection is disabled, I just copied the classes and libraries of  Ethan Rosenthal here https://github.com/EthanRosenthal/spacecutter"},{"metadata":{"trusted":true},"cell_type":"code","source":"def _reduction(loss: torch.Tensor, reduction: str) -> torch.Tensor:\n    \"\"\"\n    Reduce loss\n    Parameters\n    ----------\n    loss : torch.Tensor, [batch_size, num_classes]\n        Batch losses.\n    reduction : str\n        Method for reducing the loss. Options include 'elementwise_mean',\n        'none', and 'sum'.\n    Returns\n    -------\n    loss : torch.Tensor\n        Reduced loss.\n    \"\"\"\n    if reduction == 'elementwise_mean':\n        return loss.mean()\n    elif reduction == 'none':\n        return loss\n    elif reduction == 'sum':\n        return loss.sum()\n    else:\n        raise ValueError(f'{reduction} is not a valid reduction')\n\n\ndef cumulative_link_loss(y_pred: torch.Tensor, y_true: torch.Tensor,\n                         reduction: str = 'elementwise_mean',\n                         class_weights: Optional[np.ndarray] = None\n                         ) -> torch.Tensor:\n    \"\"\"\n    Calculates the negative log likelihood using the logistic cumulative link\n    function.\n    See \"On the consistency of ordinal regression methods\", Pedregosa et. al.\n    for more details. While this paper is not the first to introduce this, it\n    is the only one that I could find that was easily readable outside of\n    paywalls.\n    Parameters\n    ----------\n    y_pred : torch.Tensor, [batch_size, num_classes]\n        Predicted target class probabilities. float dtype.\n    y_true : torch.Tensor, [batch_size, 1]\n        True target classes. long dtype.\n    reduction : str\n        Method for reducing the loss. Options include 'elementwise_mean',\n        'none', and 'sum'.\n    class_weights : np.ndarray, [num_classes] optional (default=None)\n        An array of weights for each class. If included, then for each sample,\n        look up the true class and multiply that sample's loss by the weight in\n        this array.\n    Returns\n    -------\n    loss: torch.Tensor\n    \"\"\"\n    eps = 1e-15\n    likelihoods = torch.clamp(torch.gather(y_pred, 1, y_true), eps, 1 - eps)\n    neg_log_likelihood = -torch.log(likelihoods)\n\n    if class_weights is not None:\n        # Make sure it's on the same device as neg_log_likelihood\n        class_weights = torch.as_tensor(class_weights,\n                                        dtype=neg_log_likelihood.dtype,\n                                        device=neg_log_likelihood.device)\n        neg_log_likelihood *= class_weights[y_true]\n\n    loss = _reduction(neg_log_likelihood, reduction)\n    return loss\n\n\nclass CumulativeLinkLoss(nn.Module):\n    \"\"\"\n    Module form of cumulative_link_loss() loss function\n    Parameters\n    ----------\n    reduction : str\n        Method for reducing the loss. Options include 'elementwise_mean',\n        'none', and 'sum'.\n    class_weights : np.ndarray, [num_classes] optional (default=None)\n        An array of weights for each class. If included, then for each sample,\n        look up the true class and multiply that sample's loss by the weight in\n        this array.\n    \"\"\"\n\n    def __init__(self, reduction: str = 'elementwise_mean',\n                 class_weights: Optional[torch.Tensor] = None) -> None:\n        super().__init__()\n        self.class_weights = class_weights\n        self.reduction = reduction\n\n    def forward(self, y_pred: torch.Tensor,\n                y_true: torch.Tensor) -> torch.Tensor:\n        return cumulative_link_loss(y_pred, y_true,\n                                    reduction=self.reduction,\n                                    class_weights=self.class_weights)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from copy import deepcopy\n\nimport torch\nfrom torch import nn\n\n\nclass LogisticCumulativeLink(nn.Module):\n    \"\"\"\n    Converts a single number to the proportional odds of belonging to a class.\n    Parameters\n    ----------\n    num_classes : int\n        Number of ordered classes to partition the odds into.\n    init_cutpoints : str (default='ordered')\n        How to initialize the cutpoints of the model. Valid values are\n        - ordered : cutpoints are initialized to halfway between each class.\n        - random : cutpoints are initialized with random values.\n    \"\"\"\n\n    def __init__(self, num_classes: int,\n                 init_cutpoints: str = 'ordered') -> None:\n        assert num_classes > 2, (\n            'Only use this model if you have 3 or more classes'\n        )\n        super().__init__()\n        self.num_classes = num_classes\n        self.init_cutpoints = init_cutpoints\n        if init_cutpoints == 'ordered':\n            num_cutpoints = self.num_classes - 1\n            cutpoints = torch.arange(num_cutpoints).float() - num_cutpoints / 2\n            self.cutpoints = nn.Parameter(cutpoints)\n        elif init_cutpoints == 'random':\n            cutpoints = torch.rand(self.num_classes - 1).sort()[0]\n            self.cutpoints = nn.Parameter(cutpoints)\n        else:\n            raise ValueError(f'{init_cutpoints} is not a valid init_cutpoints '\n                             f'type')\n\n    def forward(self, X: torch.Tensor) -> torch.Tensor:\n        \"\"\"\n        Equation (11) from\n        \"On the consistency of ordinal regression methods\", Pedregosa et. al.\n        \"\"\"\n        sigmoids = torch.sigmoid(self.cutpoints - X)\n        link_mat = sigmoids[:, 1:] - sigmoids[:, :-1]\n        link_mat = torch.cat((\n                sigmoids[:, [0]],\n                link_mat,\n                (1 - sigmoids[:, [-1]])\n            ),\n            dim=1\n        )\n        return link_mat\n\n\nclass OrdinalLogisticModel(nn.Module):\n    \"\"\"\n    \"Wrapper\" model for outputting proportional odds of ordinal classes.\n    Pass in any model that outputs a single prediction value, and this module\n    will then pass that model through the LogisticCumulativeLink module.\n    Parameters\n    ----------\n    predictor : nn.Module\n        When called, must return a torch.FloatTensor with shape [batch_size, 1]\n    init_cutpoints : str (default='ordered')\n        How to initialize the cutpoints of the model. Valid values are\n        - ordered : cutpoints are initialized to halfway between each class.\n        - random : cutpoints are initialized with random values.\n    \"\"\"\n\n    def __init__(self, predictor: nn.Module, num_classes: int,\n                 init_cutpoints: str = 'ordered') -> None:\n        super().__init__()\n        self.num_classes = num_classes\n        self.predictor = deepcopy(predictor)\n        self.link = LogisticCumulativeLink(self.num_classes,\n                                           init_cutpoints=init_cutpoints)\n\n    def forward(self, X: torch.Tensor) -> torch.Tensor:\n        return self.link(self.predictor(X))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"res_model = torchvision.models.resnet34(pretrained=False)\nres_model.load_state_dict(torch.load(\"../input/resnet34/resnet34.pth\"))\nres_model.fc = nn.Linear(512, 1)\nmodel =  OrdinalLogisticModel(res_model, 5)\nmodel = model.to(device)\noptimizer = torch.optim.Adam(model.parameters(),lr = 1e-3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_ds = RetinopathyDatasetTrain(tr)\nval_ds = RetinopathyDatasetTest(val,mode = 'val')\ntrain_loader = torch.utils.data.DataLoader(train_ds, batch_size=32,collate_fn=this_collate_fn, shuffle = True, num_workers=4)\nval_loader = torch.utils.data.DataLoader(val_ds, batch_size=32, shuffle = False, num_workers=4)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#test_preds = np.zeros((len(test_dataset), 1))\ncriterion = CumulativeLinkLoss()\nfor epoch in range(epochs):\n    train_loss = []\n    val_loss = []\n    model.train()\n    for i, x_batch in enumerate(train_loader):\n        \n        model.zero_grad()\n        img = x_batch[\"image\"]\n        img = img.to(device).float()\n        label = x_batch['label'].to(device).long().reshape(-1,1)\n        output = model(img)\n        loss = criterion(output,label)\n        train_loss.append(loss)\n        loss.backward()\n        optimizer.step()   \n    model.eval()\n    with torch.no_grad():\n        for j,x_batch in enumerate(val_loader):\n            img = x_batch[\"image\"]\n            img = img.to(device).float()\n            label = x_batch['label'].to(device).long().reshape(-1,1)\n            output = model(img)\n            loss = criterion(output,label)\n            val_loss.append(loss)\n    train_mean_loss = torch.mean(torch.stack(train_loss)).data.cpu().numpy()\n    val_mean_loss = torch.mean(torch.stack(val_loss)).data.cpu().numpy()\n        \n    print(f'Epoch {epoch}, train loss: {train_mean_loss:.4f}, valid loss: {val_mean_loss:.4f}.')\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for param in model.parameters():\n    param.requires_grad = False\n\nmodel.eval()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df = pd.read_csv('../input/aptos2019-blindness-detection/test.csv')\ntest_dataset = RetinopathyDatasetTest(test_df,mode = 'test')\ntest_data_loader = torch.utils.data.DataLoader(test_dataset, batch_size=32, shuffle=False, num_workers=4)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_preds = np.zeros((len(test_dataset), 1))\n\nfor i, x_batch in enumerate(test_data_loader):\n    x_batch = x_batch[\"image\"]\n    _,pred = torch.max(model(x_batch.to(device)),1)\n    test_preds[i * 32:(i + 1) * 32] = pred.cpu().squeeze().numpy().ravel().reshape(-1, 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample = pd.read_csv(\"../input/aptos2019-blindness-detection/sample_submission.csv\")\nsample.diagnosis = test_preds.astype(int)\nsample.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* To do; improve some data augmentations to avoid obviously wrong ones\n"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}