{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Removing unwanted correlations in training data\n\nThe training data come from different sources, with different image sizes, and characteristics like brightness and hue. As noted in https://www.kaggle.com/taindow/be-careful-what-you-train-on, some of this image metadata - like the height and width - are correlated with the label. The most extreme case is that all 351 train images with dimensions 2048x1536 are class 0. Those correlations don't transfer to the test data (or the real world), so they should be removed so the final model doesn't learn them.\n\nHow to remove the spurious correlations? One could do heavy random augmentation, especially cropping, and hope to \"cover\" the correlations under the randomness of augmentation. Or one could do more targeted, deterministic preprocessing to remove the correlations, which is what we'll do here. \n\nIn either case, one would like to test whether one has successfully removed them. To test whether a preprocessing method removes spurious correlations, we'll take the preprocessed images and apply with a transformation that obscures the ground truth - such as turning all foreground pixels white and all background pixels black, or doing a Gaussian blur with large radius. \n\nThen, we'll train a simple model to predict which class the image is. If that model performs much better than random, there are still some superficial correlations and we need to do more preprocessing. If we don't manage to train a good model, it's good. We can never have a guarantee that we've removed all correlations which are there on train and are not there on test, but weakening unwanted correlations is already better than nothing.\n\nTo cut down runtime, work with data that's rescaled to 1/4 of the height and 1/4 of the width (keeping the aspect ratio).\n\n---\n\nThis is my first public kernel, feedback would be much appreciated! If there is interest, I can publish a version of the train images with this transformation applied. Editing this notebook takes a lot of RAM on my local machine, if anybody know why please let me know..."},{"metadata":{},"cell_type":"markdown","source":"# Load and utils"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom PIL import Image\nimport matplotlib.pyplot as plt\n\n\ntrain_path='../input/aptos-quarter/aptos_quarter/aptos_quarter/'\ntest_path = '../input/aptos2019-blindness-detection/test_images/'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Get train ids+labels as well as test ids"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train_df = pd.read_csv('../input/aptos2019-blindness-detection/train.csv').values\ntest_df = pd.read_csv('../input/aptos2019-blindness-detection/test.csv').values\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def show_me_some_imgs(n_rows=2, n_columns=2, train=True, preprocess = lambda x:x, idx_subset = None, figsize = (20, 10)):\n    \"\"\"\n    show some random images, preprocessed by the parameter preprocess. \n    train: whether to show train or test images.\n    idx_subset: if passed, choose at random from subset of train / test set. Else, choose random images from all of train/test.\n    \"\"\"\n    if idx_subset is None:\n        idx_subset = range(len(train_df)) if train else range(len(test_df))\n    fig, axs = plt.subplots(n_rows, n_columns, figsize = figsize)\n    if n_rows == 1:\n        axs = [axs]\n    if n_columns == 1:\n        axs = [ [ax] for ax in axs ]\n    for row in range(n_rows):\n        for column in range(n_columns):\n            if train:\n                idx = np.random.choice(idx_subset)\n                img = preprocess(Image.open(train_path + train_df[idx][0] + \".png\"))\n                axs[row][column].imshow(img)\n                axs[row][column].set_title(f'train img #{idx}. class {train_df[idx][1]}', fontsize=8)\n            else:\n                idx = np.random.choice(idx_subset)\n                img = preprocess(Image.open(test_path + test_df[idx][0] + \".png\"))\n                axs[row][column].imshow(img)\n                axs[row][column].set_title(f'test img #{idx}', fontsize=8)\n                \n#Test:\nshow_me_some_imgs()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training Dataset\n\nLet's figure out and visualize the different sources that the training dataset comes from."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dim_per_sample = len(train_df) * [None] #list of len(train_df) that has a string denoting the image's size for every image\nfor i in range(len(train_df)):\n    img = Image.open(train_path + train_df[i][0] + '.png')\n    train_dim_per_sample[i] = img.size\ntrain_unique_dims = sorted(list(set(train_dim_per_sample)))\n#two-step to get sorting right\ntrain_dim_per_sample = [f'{d[0]}x{d[1]}' for d in train_dim_per_sample]\ntrain_unique_dims = [f'{d[0]}x{d[1]}' for d in train_unique_dims]\nprint(f\"List of sizes that occur in training set: \\n {train_unique_dims} \\nThat's {len(train_unique_dims)} different sizes\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_histogram_per_dim = {}\nfor train_unique_dim in train_unique_dims:\n    train_histogram_per_dim[train_unique_dim] = [0,0,0,0,0]\n    for i in range(len(train_df)):\n        if train_dim_per_sample[i] == train_unique_dim:\n            train_histogram_per_dim[train_unique_dim][train_df[i][1]] += 1\nprint('The counts of each class, per image size, is:')\nprint(train_histogram_per_dim)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We see that some of the per-size class histograms are highly imbalanced. In the real world of course, there is no correlation between image size and degree of retinopathy.\n\nFor later, we'll need a dictionary that maps each size to a list of the images with that size:"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_samples_per_dim = {}\nfor train_unique_dim in train_unique_dims:\n    train_samples_per_dim[train_unique_dim] = []\n    for i in range(len(train_df)):\n        if train_dim_per_sample[i] == train_unique_dim:\n            train_samples_per_dim[train_unique_dim].append(i)\nprint(f\"For example, train_samples_per_dim[train_unique_dims[0]] = {train_samples_per_dim[train_unique_dims[0]]}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's plot some examples of every image size:"},{"metadata":{"trusted":true},"cell_type":"code","source":"for i, train_unique_dim in enumerate(train_unique_dims):\n    print(f'Size #{i}. dim: {train_unique_dim}. histogram {train_histogram_per_dim[train_unique_dim]}. ' +\n         f'{ 100 * sum(train_histogram_per_dim[train_unique_dim])/ len(train_df):.3f}% of the training data')\n    show_me_some_imgs(idx_subset = train_samples_per_dim[train_unique_dim], \n                      n_rows = 1, n_columns = 10, figsize = (20, 5))\n    plt.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Size #2 seems to come from two different sources, since there are images with two different crops. The rest of the sizes seems more homogenous in characteristics like how tight the crop is, whether there is a rectangle or triangle marker on the right side, the hue, whether there is glare, etc.\n\nFor easier comparison, plot one image per size, with relative histograms, and how much of the data is of that size:"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, axs = plt.subplots(4, 5, figsize = (20, 15))\n\nfor i, train_unique_dim in enumerate(train_unique_dims):\n    id_code = train_df[np.random.choice(train_samples_per_dim[train_unique_dim])][0]\n    relative_histogram = [f\"{n / len(train_samples_per_dim[train_unique_dim]):.2f}\" for n in train_histogram_per_dim[train_unique_dim]]\n    img = Image.open(train_path + id_code + '.png')\n    axs[i//5][i%5].imshow(img)\n    axs[i//5][i%5].set_title(f'#{i}. {train_unique_dim}. {relative_histogram}. ' +\n         f'{ 100 * sum(train_histogram_per_dim[train_unique_dim])/ len(train_df):.2f}%.', fontsize=8)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"keep in mind that size #2 comes from two sources, here we just see one. The most common train size is #3. \n\nNote that almost all images that show a full circle (some of size #2, size #3, size #8) are class 0 on train. If we don't remove that information, the final network could learn to always label images that show a full circle as 0, we don't want that.\n\nAlmost all train images show the full width of the circle, with the exception of size #1. However, size #1 is by far the most common on test..."},{"metadata":{},"cell_type":"markdown","source":"# Test data\nLet's do the same steps for test, and visualize what sizes and crops there are. Of course, we can't look at class histograms."},{"metadata":{"trusted":true},"cell_type":"code","source":"test_dim_per_sample = len(test_df) * [None]\nfor i in range(len(test_df)):\n    img = Image.open(test_path + test_df[i][0] + '.png')\n    test_dim_per_sample[i] = img.size\ntest_unique_dims = sorted(list(set(test_dim_per_sample)))\n#two-step to get sorting right\ntest_dim_per_sample = [f'{d[0]}x{d[1]}' for d in test_dim_per_sample]\ntest_unique_dims = [f'{d[0]}x{d[1]}' for d in test_unique_dims]\nprint(f\"List of sizes that occur in test set: \\n {test_unique_dims} \\nThat's {len(test_unique_dims)} different sizes\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_samples_per_dim = {}\nfor test_unique_dim in test_unique_dims:\n    test_samples_per_dim[test_unique_dim] = []\n    for i in range(len(test_df)):\n        if test_dim_per_sample[i] == test_unique_dim:\n            test_samples_per_dim[test_unique_dim].append(i)\nprint(f\"For example, test_samples_per_dim[test_unique_dims[1]] = {test_samples_per_dim[test_unique_dims[1]]}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for i, unique_dim in enumerate(test_unique_dims):\n    print(f'Size #{i}. dim: {unique_dim}. ' +\n         f'{ 100 * len(test_samples_per_dim[unique_dim]) / len(test_df):.3f}% of the test data')\n    show_me_some_imgs(idx_subset = test_samples_per_dim[unique_dim], train=False,\n                      n_rows = 1, n_columns = 10, figsize = (20, 5))\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"By far the most common size is size #0, corresponding to training size #1 (recall that train images are rescaled to 1/4). It is the same crop as training size #1 as well.\n\nHere is the side by side comparison:"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, axs = plt.subplots(3, 5, figsize = (20, 15))\n\nfor i, test_unique_dim in enumerate(test_unique_dims):\n    id_code = test_df[np.random.choice(test_samples_per_dim[test_unique_dim])][0]\n    img = Image.open(test_path + id_code + '.png')\n    axs[i//5][i%5].imshow(img)\n    axs[i//5][i%5].set_title(f'#{i}. {test_unique_dim}. ' +\n         f'{ 100 * len(test_samples_per_dim[test_unique_dim])/ len(test_df):.2f}%')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visualize transformations"},{"metadata":{"trusted":true},"cell_type":"code","source":"def visualize_transform(transform, compare=False, train=True):\n    \"\"\"\n    Visualizes a transformation, with one example per original size.\n    Parameters:\n    - transform: the transformation. Should take a pytorch tensor to a pytorch tensor.\n    - compare: if true, show the unmodified and transformed images next to each other\n    - train:\n    \"\"\"\n    if train:\n        unique_dims = train_unique_dims\n        samples_per_dim = train_samples_per_dim\n        df = train_df\n        path = train_path\n    else:\n        unique_dims = test_unique_dims\n        samples_per_dim = test_samples_per_dim\n        df = test_df\n        path = test_path  \n    if compare:\n        fig, axs = plt.subplots(8, 5, figsize = (20, 30))\n    else:\n        fig, axs = plt.subplots(4, 5, figsize = (20, 15))\n\n    for i, unique_dim in enumerate(unique_dims):\n        row = 2*(i//5) if compare else i//5\n        img_idx = np.random.choice(samples_per_dim[unique_dim])\n        id_code = df[img_idx][0]\n        img = transforms.ToTensor()(Image.open(path + id_code + '.png'))\n        axs[row][i%5].imshow(transforms.ToPILImage()(transform(img)))\n        if train:\n            axs[row][i%5].set_title(f'#{i}. {unique_dim}. {train_histogram_per_dim[unique_dim]}. ' +\n                 f'{ 100 * sum(train_histogram_per_dim[unique_dim])/ len(train_df):.2f}%', fontsize=8)\n        else:\n            axs[row][i%5].set_title(f'#{i}. {unique_dim}. ' +\n                 f'{ 100 * len(samples_per_dim[unique_dim])/ len(df):.2f}%', fontsize=8)\n        if compare:\n            axs[row+1][i%5].imshow(transforms.ToPILImage()(img))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Define Network\n\nAs mentioned above, we'll define a simple neural network (from a pretrained resnet18) that we'll use to test whether we have successfully removed correlations we don't want in the training set.\n\nWe'll define things in two variants: one with target='class', where the task of the NN is to predict the class of each training example. One with target='size', where the task of the NN is to predict the original size.\n\nThe reason for using target='size' is the following: The most common class on train is zero, hence if we use target='class' the network is going to guess zero if in doubt. This is going to lead to high accuracy for example for the images of original size #8, and we won't know if that's because there are still unwanted correlations or if it's just a guess. Using target='size' is more finegrained."},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch\nfrom torch.utils.data import Dataset, DataLoader, SubsetRandomSampler\nfrom torchvision import models, transforms\nfrom tqdm import tqdm_notebook\nimport cv2\ndevice = torch.device('cuda')\n\nclass Aptos_train_ds(Dataset):\n    def __init__(self, \n                 train_path, \n                 transform = lambda x:x,\n                 target = 'size'):\n        self.train_path = train_path\n        self.train_df = pd.read_csv('../input/aptos2019-blindness-detection/train.csv').values\n        self.to_tensor = transforms.ToTensor() \n        self.transform=transform\n        self.target = target\n    def __len__(self):\n        return len(self.train_df)\n    def __getitem__(self, idx):\n        id_code = self.train_df[idx][0]\n        img = self.to_tensor(Image.open(train_path + id_code + '.png'))\n        x = self.transform(img)\n        if self.target == 'size':\n            y = train_unique_dims.index(train_dim_per_sample[idx])\n        else:\n            y = self.train_df[idx][1]\n        return x, y\n\ndef get_model(target = 'size'):\n    model = models.resnet18(pretrained=True)\n    if target == 'size':\n        print('predicting size')\n        model.fc = torch.nn.Linear(512, 17)\n    else:\n        print('predicting class')\n        model.fc = torch.nn.Linear(512, 5)\n    model.train()\n    return model.to(device)\n\ndef get_trainvalloader(train_ds):\n    idcs = torch.randperm(len(train_ds))\n    train_idcs = idcs[:int(.8*len(train_ds))]\n    val_idcs = idcs[int(.8*len(train_ds)):]\n    \n    train_loader = DataLoader(train_ds, batch_size = 32, num_workers=9, \n                                   sampler = SubsetRandomSampler(train_idcs), pin_memory=True)\n    val_loader = DataLoader(train_ds, batch_size = 32, num_workers=9, \n                                   sampler = SubsetRandomSampler(val_idcs), pin_memory=True)\n    return train_loader, val_loader, val_idcs.tolist()\n\ndef acc_from_data_loader(model, data_loader):\n    \"\"\"\n    given a model and a data loader, return the accuracy of the model on the samples in the dataloader\n    \"\"\"\n    training = model.training\n    model.eval()\n    val_preds = []\n    val_gts = []\n    with torch.no_grad():\n        for x, y in data_loader:\n            x, y = x.to(device), y.to(device)\n            output = model(x)\n            pred = torch.argmax(output, dim=1)\n            val_preds.append(pred)\n            val_gts.append(y)\n        val_preds, val_gts = torch.cat(val_preds), torch.cat(val_gts)\n        acc = (val_preds == val_gts).to(torch.float).mean()\n    model.train(training)\n    return acc\n\ndef acc_from_subset(model, train_ds, subset):\n    \"\"\"\n    Given a model, a dataset and a list of indices, return the accuracy of the model on the elements of the dataset which are contained \n    in the list of indices\n    \"\"\"\n    if len(subset)==0:\n        return -1\n    data_loader = DataLoader(train_ds, batch_size = 32, num_workers=5, \n                                   sampler = SubsetRandomSampler(subset))\n    return acc_from_data_loader(model, data_loader)\n\ndef train(model, train_loader, val_loader, num_epochs = 1):\n    optimizer = torch.optim.Adam(model.parameters())\n    criterion = torch.nn.CrossEntropyLoss()\n    model.train()\n    for epoch in range(num_epochs):\n        for (x, y) in tqdm_notebook(train_loader):\n        #for (x, y) in train_loader:\n            x, y = x.to(device), y.to(device)\n            output = model(x)\n            loss = criterion(output, y)\n            loss.backward()\n            optimizer.step()\n            optimizer.zero_grad()\n        print(f'val acc: {acc_from_data_loader(model, val_loader)}')\n    \ndef get_acc_per_dim(transform, num_epochs = 1, target='size'):\n    \"\"\"\n    This is the function we'll use. Train a resnet18 to predict the target.\n    Then show the accuracy on val, separated by the original size.\n    \"\"\"\n    torch.manual_seed(2019)\n    torch.backends.cudnn.deterministic=True\n    np.random.seed(2019)\n    train_ds = Aptos_train_ds(train_path, transform = transform, target=target)\n    plt.figure(figsize = (2,2))\n    plt.imshow(transforms.ToPILImage()(train_ds[np.random.randint(len(train_ds))][0]))\n    plt.show()\n    train_dl, val_dl, val_idcs = get_trainvalloader(train_ds)\n    model = get_model(target=target)\n    train(model, train_dl, val_dl, num_epochs = num_epochs)\n    print('Accuracies per dimension on val:')\n    for i, dim in enumerate(train_unique_dims):\n        num = sum(train_histogram_per_dim[dim])\n        rel_hist = [ f\"{h/num:.3f}\" for h in train_histogram_per_dim[dim] ]\n        val_samples_per_dim = list(set(train_samples_per_dim[dim]) & set(val_idcs)) \n            #intersection between val indices and indices of images with current size\n        print(f'#{i: <2}. size: {dim: <8}, relative histogram: {rel_hist}, ' + \n              f'fraction of data { 100* num / len(train_df):.1f}%, ' +\n              f'acc: {acc_from_subset(model, train_ds, val_samples_per_dim):.3f}')\n        \nto_128_torch = lambda img: torch.nn.functional.interpolate(img[None, ...], size = (128, 128), mode='bilinear', align_corners = False)[0, ...]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As a test, see if our model can figure out the original size of resized train images: Acc of -1 means that there are no images of that original size in val. This can happen if there are very few train images of a given size, but those rare sizes we don't care about so much anyway."},{"metadata":{"trusted":true},"cell_type":"code","source":"transform = to_128_torch\nget_acc_per_dim(transform, target='size', num_epochs=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It does pretty well, i.e. resizing does not hide the original size of the image. Since the original size is in some cases highly correlated with the label, there is an unwanted correlation here.\n# Removing shape information\nOne might guess that the network figures out the original size primarily by looking at what kind of crop we have, i.e., the shape of the foreground. To test that, let's come up with a transformation that makes all of the foreground white and all of the background black."},{"metadata":{"trusted":true},"cell_type":"code","source":"def shapify_torch(img_data):\n    height, width = img_data.shape[1:]\n    black = img_data[:, :int(height/20), :int(width/20)].mean(dim=(1, 2)) # get average r, g, b of top-left corner as an estimate for black value\n    mask = ((img_data - black[:, None, None]).max(dim = 0)[0] > .02).to(img_data.dtype) #note torch's max with argument dim returns (max, argmax)\n    return torch.stack((mask, mask, mask))\ndef shapify_pil(img):\n    img_data = transforms.ToTensor()(img) #rgb, height, width. range [0, 1]\n    img_data = shapify_torch(img_data)\n    return transforms.ToPILImage()(img_data)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_transform(shapify_torch, train=True, compare=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Alright, now we can check whether the shape is enough to say which source the image comes from:"},{"metadata":{"trusted":true},"cell_type":"code","source":"get_acc_per_dim(transforms.Compose([to_128_torch, shapify_torch]), target='size', num_epochs=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Yes, the shape of the foreground is enough: 94% of validation data are assigned the correct original size after 1 epoch. Just for completeness, let's see if we can predict the class from the shape:"},{"metadata":{"trusted":true},"cell_type":"code","source":"get_acc_per_dim(transforms.Compose([to_128_torch, shapify_torch]), target='class', num_epochs=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Yes, to an accuracy that's much better than chance. It doesn't always guess zero either, for size #9 it guesses mostly 2. That's bad. Let's set out to remove the correlation between shape and class from the data.\n\nFirst remove black space everywhere:"},{"metadata":{"trusted":true},"cell_type":"code","source":"def crop_out_black(img):\n    height, width = img.shape[1:]\n    black = img[:, :int(height/20), :int(width/20)].mean(dim=(1, 2))\n    rowmeans = img.mean(dim=1)\n    linemeans = img.mean(dim=2)\n    nonblack_rows = ((rowmeans - black[:, None]).max(dim=0)[0] > .02).nonzero()\n    nonblack_lines = ((linemeans - black[:, None]).max(dim=0)[0] > .02).nonzero()\n    try:\n        left, right = nonblack_rows[0].item(), nonblack_rows[-1].item()\n        upper, lower = nonblack_lines[0].item(), nonblack_lines[-1].item()\n        img = img[:, upper:lower, left:right]\n    except:\n        print('crop out black didnt work')\n    return img\n\n\n#visualize_transform(crop_out_black, compare=True, train=False)\nvisualize_transform(crop_out_black, compare=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"That seems to work alright. Let's see if that removes the information on the original size from the shapified image:"},{"metadata":{"trusted":true},"cell_type":"code","source":"get_acc_per_dim(transforms.Compose([crop_out_black, to_128_torch, shapify_torch]), target='size', num_epochs=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It does not, the network is still very good at figuring out the original size of the train image. We'll also need to crop. \n\nIt turns out that after removing the black, not all images are quite centered, so to make life easier let's put them back in the center."},{"metadata":{"trusted":true},"cell_type":"code","source":"def center(img):\n    # crop such that the center of mass of non-black pixels is roughly in the center\n    _, height, width = img.shape\n    shapified = shapify_torch(img)[0, ...] #just take one of 3 channels\n    nonzero = (shapified).nonzero().to(torch.float)\n    center = nonzero.mean(dim = 0).to(torch.int)\n    if center[0] > height/2: #center too low, crop from top\n        new_height = 2 * (height - center[0])\n        img = img[:, -new_height:, :]\n    else: #center too high, crop from bottom\n        new_height = 2 * center[0]\n        img = img[:, :new_height, :]\n    if center[1] > width/2: #center too far right, crop from left\n        new_width = 2*(width- center[1])\n        img = img[:, :, -new_width:]\n    else: #center too far left, crop from right\n        new_width = 2*center[1]\n        img = img[:, :, :new_width]\n    return img\n\ntransform = transforms.Compose([crop_out_black, center, to_128_torch])\nvisualize_transform(transform, compare=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's think about how to crop. We should crop away some from the top and bottom of the image, since as we saw on train most images that show the full height of the circle are class 0, which is an unwanted correlation.\n\nShould we crop from the right or left? Most of the test data just shows the center part of the retina, so it makes sense to focus on that and crop train images the same. That's what we'll do here. We're not cropping out too much area, 6% on either side.\n\n(To get a little better with the part of the test data that shows the full width, it may be worth it to have some train images that do show the full width, for example crop train images to the center part with prob 3/4 and show the full width with prob 1/4.)\n\nHere is a function that crops a centered image to look like train size #1 / test size #0:"},{"metadata":{"trusted":true},"cell_type":"code","source":"\ndef tight_crop(img): #assumes black cropped out and centered\n    shapified = shapify_torch(img)[0, ...]\n    if shapified.to(torch.float).mean() > .95: #already tight crop\n        #print('already tight crop, passing')\n        return img\n    width = img.shape[2]\n    width_margin = int(.06 * width)\n    img = img[:, :, width_margin:-width_margin]\n    shapified = shapified[ :, width_margin:-width_margin]\n    num_white_per_line = shapified.sum(dim=1) / shapified.shape[1]\n    white_above_threshold = (num_white_per_line > .9).nonzero()\n    upper, lower = white_above_threshold[0], white_above_threshold[-1]\n    img = img[:, upper:lower, :]\n    return img\n\n\ntransform = transforms.Compose([crop_out_black, center, tight_crop, to_128_torch])\nvisualize_transform(transform, compare=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"That seems to work alright, let's see if our test network can still figure out the size after we \"shapify\":"},{"metadata":{"trusted":true},"cell_type":"code","source":"transform = transforms.Compose([crop_out_black, center, tight_crop, to_128_torch, shapify_torch])\nvisualize_transform(transform)\nget_acc_per_dim(transform, target='size', num_epochs=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Surprisingly, it still can in some cases. My guess is that it's using the shape of the black areas in the corners, such as whether there is a triangle or a square marker on the right, and maybe artefacts from the resizing. To make extra sure, let's turn the four corners of each image black."},{"metadata":{"trusted":true},"cell_type":"code","source":"\ndef remove_corners(img): #blacken a triangle of 1/6 at each corner. assumes square input\n    corner_size = img.shape[1]//6\n    mask = torch.ones( (corner_size, corner_size )).triu()\n    img[:, :corner_size, :corner_size] *= mask.flip(dims=(0,))[None, :, :]\n    img[:, :corner_size, -corner_size:] *= mask.flip(dims=(0,1))[None, :, :]\n    img[:, -corner_size:, :corner_size] *= mask[None, :, :]\n    img[:, -corner_size:, -corner_size:] *= mask.flip(dims=(1,))[None, :, :]\n    return(img)\n\ntransform = transforms.Compose([crop_out_black, center, tight_crop, to_128_torch, remove_corners])\nvisualize_transform(transform)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Since the foreground has exactly the same shape for every train image now, we can be sure that any unwanted correlation between the shape of the foreground and the label is gone. Standardizing the foreground shape is also done in https://www.kaggle.com/taindow/pre-processing-train-and-test-images . There the active area is cropped to a circle, here by just taking of the corners we can keep a bit more of the retina, and the result is closer to the most common crop on the test set."},{"metadata":{},"cell_type":"markdown","source":"# Removing color correlations\nSo, we removed the spurious shape correlation. There's a possibility that unwanted correlations are still hiding in the colors though... Let us remove the true label information by a Gaussian blur."},{"metadata":{"trusted":true},"cell_type":"code","source":"def gaussian_blur(img, radius=None, rel_size = None):\n    if radius is None:\n        radius = int(rel_size * img.shape[1])\n    if radius % 2 == 0:\n        radius = radius + 1\n    img_numpy = img.permute(1, 2, 0).numpy()\n    img_numpy = cv2.GaussianBlur(img_numpy,(radius,radius),0)\n    img = torch.Tensor(img_numpy).permute(2, 0, 1)\n    return img\ntransform = transforms.Compose([crop_out_black, center, tight_crop, to_128_torch, remove_corners, lambda img: gaussian_blur(img, rel_size=.5)])\nvisualize_transform(transform)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's see if our model can still tell the original size / class from the blurred images:"},{"metadata":{"trusted":true},"cell_type":"code","source":"get_acc_per_dim(transform, target='size')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"get_acc_per_dim(transform, target='class')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It can, with accuracies above chance. For example, for original size #3 it mostly guessed 0, and for original size 6 it mostly guessed 3, it's not supposed to be able to do that. Let's remove some background coloring by subtracting the Gaussian blur, as in https://www.kaggle.com/ratthachat/aptos-updatedv14-preprocessing-ben-s-cropping (originally Ben Graham)"},{"metadata":{"trusted":true},"cell_type":"code","source":"def subtract_gaussian_blur(img, rel_size = .2, color_scale = 1):\n    img_blurred = gaussian_blur(img, rel_size = rel_size)\n    img = (4*color_scale*(img - img_blurred)).sigmoid() #sigmoid to squish to [0, 1]. Factor 4 because the slope of sigmoid at 0 is 4.\n    return img\n\nvisualize_transform(subtract_gaussian_blur)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The original color is still visible as the color of the rim. Let's combine it with the crop - by subtracting the blur first and then cropping we can get rid of the colorful rim."},{"metadata":{"trusted":true},"cell_type":"code","source":"transform = transforms.Compose([\n    crop_out_black, \n    center, \n    tight_crop, \n    to_128_torch, \n    lambda img: subtract_gaussian_blur(img, rel_size=.2, color_scale=1), \n    remove_corners])\nvisualize_transform(transform)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Those start looking really similar to each other. Let's test again whether the model can predict the original size from the blurred version:"},{"metadata":{"trusted":true},"cell_type":"code","source":"transform = transforms.Compose([\n    crop_out_black, \n    center, \n    tight_crop, \n    to_128_torch, \n    lambda img: subtract_gaussian_blur(img, rel_size=.2, color_scale=1), \n    remove_corners,\n    lambda img: gaussian_blur(img, rel_size = .3)])\nvisualize_transform(transform)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# get_acc_per_dim(transform, target='size', num_epochs=2) #overfits\nget_acc_per_dim(transform, target='size', num_epochs=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It still somewhat can, in particular it's learned to keep apart original sizes 3 and 11 (the two most frequent ones) by their blurred versions. My guess is that the information is in the hue of the bright spot. But the situation is better than when we started out, let's stop here."},{"metadata":{},"cell_type":"markdown","source":"# Making sure the transformation works on the original train images and on test\n\nI've tried to keep things 'relative', but let's check:"},{"metadata":{"trusted":true},"cell_type":"code","source":"size = 512\ntrain_path = '../input/aptos2019-blindness-detection/train_images/'\n\ntransform = transforms.Compose([\n    crop_out_black, \n    center, \n    tight_crop, \n    lambda img: torch.nn.functional.interpolate(img[None, ...], size = (size, size), mode='bilinear', align_corners = False)[0, ...] ,\n    lambda img: subtract_gaussian_blur(img, rel_size=.2, color_scale=2), \n    remove_corners])\n\nvisualize_transform(transform, compare=True)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Seems to work. Make sure it works on test:"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_path = '../input/aptos2019-blindness-detection/test_images/'\nvisualize_transform(transform, compare=True, train=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Apply."},{"metadata":{"trusted":true},"cell_type":"code","source":"#!mkdir 2_preprocessed_512\n#target_dir = '2_preprocessed_512/'\n#import os\n#train_image_files = os.listdir('../input/aptos2019-blindness-detection/train_images')\n\n#for i,train_image_file in enumerate( train_image_files):\n#    img = transforms.ToTensor()(Image.open('../input/aptos2019-blindness-detection/train_images/' + train_image_file))\n#    img = transforms.ToPILImage()(transform(img))\n#    #torch.save(img, target_dir + train_image_file + '.pth')\n#    img.save(target_dir + train_image_file)\n#    print(i)\n\n#import zipfile\n#zf = zipfile.ZipFile('2_preprocessed_512.zip', mode='w')\n\n#for i,train_image_file in enumerate( train_image_files):\n#    zf.write(target_dir + train_image_file)\n#    #zf.write('../input/train_images/' + train_image_file)\n#zf.close()\n\n#!rm -r 2_preprocessed_512","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}