{"cells":[{"metadata":{},"cell_type":"markdown","source":"One of the things worth considering for this competition is whether convolutions are really reasonable operations to use for self driving cars. We can test with various different toy problems to see if the models are able to capture the information we think they should, like the distance between cars.\n\nThis analysis was triggered by trying to train models at 128x128 and then fine-tune on 300x300 and finding that the performance took a significant hit right away when switching resolutions."},{"metadata":{},"cell_type":"markdown","source":"Goals: \nDone\n* Can a CNN learn to measure distance between two points?\n* Does a pretrained model already have some of this information?\n* Is it resilient to new resolutions?\n* What do the intermediate layers look like out of the CNN solving this problem?\n* Does pooling put a limit on the performance of the model?\n\n\nin progress:\n* Can a CNN learn to compute area of a rectangle?\n\n* Can a CNN simulatenously locate objects to measure between\n* Is it resilient to additional objects in validation time\n\n* Can a CNN count?\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Simple model definition, just a mix of some convolutions and max-pooling. Global max pooling configured so the model is agnostic to resolution. In theory can be applied to images of any resolutions beyond a certain threshold."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torchvision.models.resnet import resnet18\n\nclass Net(nn.Module):\n    def __init__(self):\n        super(Net, self).__init__()\n        self.preconv1 = nn.Conv2d(3, 2, 1)\n        self.fc1 = nn.Linear(4, 128)\n        self.fc2 = nn.Linear(128, 128)\n        self.fc3 = nn.Linear(128, 128)\n        self.fc4 = nn.Linear(128, 128)\n        self.fc5 = nn.Linear(128, 128)\n        self.fc6 = nn.Linear(128, 128)\n        self.fc7 = nn.Linear(128, 128)\n        self.fc8 = nn.Linear(128, 1)\n\n    def forward(self, x):\n        x = x.to(torch.float32)\n        x = self.preconv1(x)\n        x = x.view(x.shape[0], x.shape[1], -1)\n        x_max = x.max(axis = 2)[0]\n        x_min = (-x).max(axis = 2)[0]\n        \n        x_max = (x_max - 1)*128\n        x_min = (-x_min+1)*128\n        \n        x = torch.cat((x_max, x_min), axis = -1)\n        x = F.elu(self.fc1(x))\n        x = F.elu(self.fc2(x))\n        x = F.elu(self.fc3(x))\n        x = F.elu(self.fc4(x))\n        x = F.elu(self.fc5(x))\n        x = F.elu(self.fc6(x))\n        x = F.elu(self.fc7(x))\n        x = self.fc8(x)\n        \n        return torch.squeeze(x)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"net = Net()\nnet = net.to(\"cuda\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Resnet model we will explore in parallel. Similar principles. We can look at pretrained and randomly initialized and see what information is captured in the pretrained weights"},{"metadata":{"trusted":true},"cell_type":"code","source":"class resnet_model(nn.Module):\n    def __init__(self):\n        super(resnet_model, self).__init__()\n        self.backbone = resnet18(pretrained=True)\n        self.backbone.fc = nn.Linear(512, 1)\n\n    def forward(self, x):\n        x = self.backbone(x)\n        return x.squeeze()\n\n\nresnet = resnet_model().to(\"cuda\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch.optim as optim\nfrom torch.utils.data import Dataset, DataLoader","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Creating a simple pytorch dataset we can generate images of an arbitrary size and then plot two random pixels on them. The model will then try to learn the euclidean distance between these points. "},{"metadata":{"trusted":true},"cell_type":"code","source":"sample = np.zeros((3, 128, 128))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class measure_dataset(Dataset):\n    def __init__(self, size = (128, 128), data_size = 10000):\n        self.size = size\n        self.data_size = data_size\n        \n    def __len__(self):\n        return self.data_size\n    \n    def __getitem__(self, index):\n        sample = np.zeros((3, self.size[0], self.size[1]))\n        sample[1, :, :] = np.arange(0, self.size[0])[None, :]/(self.size[0] + 1)\n        sample[2, :, :] = np.arange(0, self.size[1])[:, None]/(self.size[1] + 1)\n        \n        first_x = np.random.randint(0, self.size[0])\n        second_x = np.random.randint(0, self.size[0])\n        first_y = np.random.randint(0, self.size[1])\n        second_y = np.random.randint(0, self.size[1])\n        \n        sample[0, first_y, first_x] = 1\n        sample[0, second_y, second_x] = -1\n        return sample, np.sqrt((first_x - second_x)**2 + (first_y - second_y)**2)\n        \n        ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"trn_dataset = measure_dataset()\ntrn_dataloader = DataLoader(trn_dataset, batch_size=512, drop_last = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for x,y in trn_dataloader:\n    break","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is what the training samples and labels will look like"},{"metadata":{"trusted":true},"cell_type":"code","source":"dual_channel = np.zeros((128, 128, 3))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dual_channel[:, :, 0] = x[0, 0] * x[0, 1]\ndual_channel[:, :, 1] = x[0, 0] * x[0, 2]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"np.unique(dual_channel[:, :, 0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"np.unique(dual_channel[:, :, 1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(dual_channel[:, :, 0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(x[0].permute(1, 2, 0))\nprint(y[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(x[0, 2])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We will now create a simple training loop that tries to train the models on this generated data"},{"metadata":{"trusted":true},"cell_type":"code","source":"def train_model(trn_dataloader, model, epochs = 10):\n    criterion = nn.L1Loss()\n    model.train()\n    optimizer = optim.Adam(model.parameters(), lr=0.0001)\n    running_loss = []\n    for epoch in range(epochs):  # loop over the dataset multiple times\n        for i, data in enumerate(trn_dataloader, 0):\n            # get the inputs; data is a list of [inputs, labels]\n            inputs, labels = data\n            inputs = inputs.to(torch.float32).to(\"cuda\")\n            labels = labels.to(torch.float32).to(\"cuda\")\n            # zero the parameter gradients\n            optimizer.zero_grad()\n\n            # forward + backward + optimize\n            outputs = model(inputs)\n            loss = criterion(outputs, labels)\n            loss.backward()\n            optimizer.step()\n\n            # print statistics\n            running_loss.append(loss.cpu().detach().numpy())\n        print(\"epoch\", str(epoch) + \":\",  np.mean(np.array(running_loss[-100:]).reshape(-1,)))\n    return running_loss","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We will start with the custom model we defined"},{"metadata":{"trusted":true},"cell_type":"code","source":"net.preconv1.weight.data[0, 0] = 1\nnet.preconv1.weight.data[0, 1] = 1\nnet.preconv1.weight.data[0, 2] = 0\n\nnet.preconv1.weight.data[1, 0] = 1\nnet.preconv1.weight.data[1, 1] = 0\nnet.preconv1.weight.data[1, 2] = 1\n\nnet.preconv1.bias.data[:] = 0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"net.preconv1.requires_grad_ = False","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"net.preconv1.weight.data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"net.preconv1.bias.data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(x[0, 0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"net(x[0:1].to(\"cuda\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"running_loss = train_model(trn_dataloader, net, 100)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next we will train our resnet18 pretrained model and see how it compares"},{"metadata":{"trusted":true},"cell_type":"code","source":"running_loss = train_model(trn_dataloader, resnet, 100)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see right away that the pretrained resnet converges way faster and to a lower minimum than the custom convnet. We can test after if it is the pretrained weights that are special here or the better crafted NN architecture"},{"metadata":{},"cell_type":"markdown","source":"Now we can validate the models. This isn't true validation since the data is randomly generated and can have overlaps with the training set, but I assume these will be fairly rare given the low probability of hitting the same two pixels out of 128x128 images. "},{"metadata":{"trusted":true},"cell_type":"code","source":"def validate_model(trn_dataloader, model):\n    val_criterion = nn.L1Loss(reduction = 'none')\n    running_loss = []\n    running_labels = []\n    model.eval()\n    with torch.no_grad():\n        for i, data in enumerate(trn_dataloader, 0):\n            inputs, labels = data\n            inputs = inputs.to(torch.float32).to(\"cuda\")\n            labels = labels.to(torch.float32).to(\"cuda\")\n            preds = model(inputs)\n            loss = val_criterion(preds, labels)\n            running_loss.append(loss.cpu().detach().numpy())\n            running_labels.append(labels.cpu().detach().numpy())\n    return np.array(running_loss).reshape(-1,), np.array(running_labels).reshape(-1,)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The validate function will return the loss of each sample as well as the label for each sample. In this way we can plot the loss against the labels and see if the model is particularly worse at larger distances between points or any other pattern like that. "},{"metadata":{"trusted":true},"cell_type":"code","source":"running_loss, running_labels = validate_model(trn_dataloader, net)\nplt.scatter(running_labels, running_loss)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is quite an interesting loss vs label pattern. My understanding here is that the model quickly converges to the average case, on a 128x128 image the average euclidean distance between any two random points must be around 75. It probably converges to that point and then begins improving at the other labels predictions, but we can see it is still doing a poor job"},{"metadata":{"trusted":true},"cell_type":"code","source":"running_loss, running_labels = validate_model(trn_dataloader, resnet)\nplt.scatter(running_labels, running_loss)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that the resnet pattern looks significantly different. For starters all loss values are below ~4.0 and there is not the V shape present from the other model. "},{"metadata":{},"cell_type":"markdown","source":"One thing to consider is the effect that the pooling layers have on our model. In theory max pooling is reducing the resolution of our image, but maybe the convolutions can reformat the data to still capture accurate positions regardless. For example putting a bit in a different channel to denote if it originated from the  top-left pixel or top-right pixel in a pooling layer."},{"metadata":{},"cell_type":"markdown","source":"Now I will train the simple model a bit further and then we can inspect the activations from the various layers to see what it is learning"},{"metadata":{"trusted":true},"cell_type":"code","source":"running_loss = train_model(trn_dataloader, net, 100)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x = x.to(torch.float32).to('cuda')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we can look at the activations of a specific sample across the layers of our simple CNN we trained and see what our model is learning"},{"metadata":{"trusted":true},"cell_type":"code","source":"layers = []\nwith torch.no_grad():\n    for i, module in enumerate(net.modules()):\n        try:\n            if i == 0:\n                continue\n            print(module)\n            if len(layers) == 0:\n                layers.append(module(x))\n            else:\n                layers.append(module(layers[-1]))\n        except:\n            print(\"failed layer\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Viewing across the layers we can see how information is propogating through to the final predictions. I will just look at the first sample from a batch of data and trace it through to the final dense layer of the network. As showing many images all at once is difficult I will limit it the visualizations to be a max of 32 channels per layer. Hopefully we can still find something amongst the incomplete set of images"},{"metadata":{"trusted":true},"cell_type":"code","source":"from mpl_toolkits.axes_grid1 import ImageGrid\nwith torch.no_grad():\n    for i, layer in enumerate(layers):\n        layer = layer[0]\n        layer = layer[:32]\n        print(\"layer\" + str(i))\n        fig = plt.figure(figsize=(len(layer)*4, len(layer)*4))\n        grid = ImageGrid(fig, 111,  # similar to subplot(111)\n                         nrows_ncols=(int(np.ceil(len(layer)/8)), 8),  # creates 2x2 grid of axes\n                         axes_pad=0.1,  # pad between axes in inch.\n                         )\n\n        for ax, im in zip(grid, layer.cpu().detach().numpy()):\n            # Iterating over the grid returns the Axes.\n            ax.imshow(im)\n        print(im.shape)\n        plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From this we can see... not a whole lot. I am not sure what to infer from these images. It looks like the pixels are just being blurred. I don't entirely know what I expected to see here, but maybe something that looked more like: "},{"metadata":{},"cell_type":"markdown","source":"| ![frame 1](https://i.imgur.com/MmZWBWW.png) \t| ![frame 2](https://i.imgur.com/5IgnXRn.png) \t| ![frame 3](https://i.imgur.com/yW3DlMk.png) \t| ![frame 4](https://i.imgur.com/WizGDZz.png) \t| ![frame 5](https://i.imgur.com/mVhb7ME.png) \t| ![frame 6](https://i.imgur.com/u8y0FBH.png) \t| ![frame 7](https://i.imgur.com/WvyaRim.png) \t|    \n|-\t|-\t|-\t|-\t|-\t|-\t|-\t|"},{"metadata":{},"cell_type":"markdown","source":"In theory then the model could sum up the pixels and infer distance with this information. It's unclear to me how the problem is solved in the activations we can visualize, but it seems to have learned sufficiently for the task of measuring distance"},{"metadata":{},"cell_type":"markdown","source":"One hypothesis I have is that the model isnt trying to find the line between the two points and rather inferring x, y coordinates by measuring distance to the borders of the image. It would be interesting to see if the model has learned generalizable features for measuring distance or if it is specific to the resolution it was trained on. In practice I have seen models that work perfectly fine on alternate resolutions detecting cat vs dog and other tasks like that so I would assume there is still some possibility of performance. I would think beyond a certain resolution we would have problems though because the model has never learned to make predictions so high. \n\nWe can start with 256 x 256 and see what happens with our two different models"},{"metadata":{"trusted":true},"cell_type":"code","source":"big_trn_dataset = measure_dataset((256, 256))\nbig_trn_dataloader = DataLoader(big_trn_dataset, batch_size=64, drop_last = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"running_loss, running_labels = validate_model(big_trn_dataloader, net)\nplt.scatter(running_labels, running_loss)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"running_loss, running_labels = validate_model(big_trn_dataloader, resnet)\nplt.scatter(running_labels, running_loss)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Interestingly we get a bit of a snail pattern on the custom CNN. Beyond 128 we start seeing a linear growth between label and loss. Kind of expected given that our model likely only learned to predict within the possibilities of distances for 128x128 images. \n\nWith the resnet model we see linear growth almost the whole time. very puzzling. Maybe we can inspect predictions more closely and understand what is causing that"},{"metadata":{"trusted":true},"cell_type":"code","source":"def validate_model(trn_dataloader, model):\n    val_criterion = nn.L1Loss(reduction = 'none')\n    running_loss = []\n    running_labels = []\n    predictions = []\n    model.eval()\n    with torch.no_grad():\n        for i, data in enumerate(trn_dataloader, 0):\n            inputs, labels = data\n            inputs = inputs.to(torch.float32).to(\"cuda\")\n            labels = labels.to(torch.float32).to(\"cuda\")\n            preds = model(inputs)\n            loss = val_criterion(preds, labels)\n            running_loss.append(loss.cpu().detach().numpy())\n            running_labels.append(labels.cpu().detach().numpy())\n            predictions.append(preds.cpu().detach().numpy())\n    return np.array(running_loss).reshape(-1,), np.array(running_labels).reshape(-1,), np.array(predictions).reshape(-1,)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"_, _, predictions = validate_model(big_trn_dataloader, resnet)\npredictions_df = pd.DataFrame(predictions)\nprint(predictions_df.describe())\npredictions_df.hist(bins = 20)\nplt.title(\"distribution of predictions of resnet\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"_, _, predictions = validate_model(big_trn_dataloader, net)\npredictions_df = pd.DataFrame(predictions)\nprint(predictions_df.describe())\npredictions_df.hist(bins = 20)\nplt.title(\"distribution of predictions of custom CNN\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Interestingly the models have very different modes of failure. The resnet seems to have predictions centered around the totally wrong region, while the custom CNN is skewing its predictions to the top of the range it is capable of but cant predict beyond that range. \n\nNow on the flip side we can try predicting on smaller images. Maybe performance will work when we arent trying to extapolate beyond a known range"},{"metadata":{"trusted":true},"cell_type":"code","source":"sml_trn_dataset = measure_dataset((110, 110))\nsml_trn_dataloader = DataLoader(sml_trn_dataset, batch_size=64, drop_last = True)\nrunning_loss, running_labels, predictions = validate_model(sml_trn_dataloader, resnet)\nplt.scatter(running_labels, running_loss)\n\npredictions_df = pd.DataFrame(predictions)\nprint(predictions_df.describe())\npredictions_df.hist(bins = 20)\nplt.title(\"distribution of predictions of resnet\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"running_loss, running_labels, predictions = validate_model(sml_trn_dataloader, net)\nplt.scatter(running_labels, running_loss)\n\npredictions_df = pd.DataFrame(predictions)\nprint(predictions_df.describe())\npredictions_df.hist(bins = 20)\nplt.title(\"distribution of predictions of custom CNN\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Interestingly on the slightly smaller images the resnet does signficantly better and the custom CNN seems to have decayed rapidly. It would be interesting to see an ablation done over varying resolutions to see what range of resolutions are acceptable for the models."},{"metadata":{},"cell_type":"markdown","source":"to be continued..."},{"metadata":{"trusted":true},"cell_type":"code","source":"losses = []\nfor i in range(64, 256, 10):\n    sml_trn_dataset = measure_dataset((i, i))\n    sml_trn_dataloader = DataLoader(sml_trn_dataset, batch_size=64, drop_last = True)\n    running_loss, running_labels, predictions = validate_model(sml_trn_dataloader, resnet)\n    plt.scatter(running_labels, running_loss)\n\n    predictions_df = pd.DataFrame(predictions)\n    print(\"image size:\", str(i))\n    print(predictions_df.describe())\n    predictions_df.hist(bins = 20)\n    plt.title(\"distribution of predictions of resnet\")\n    plt.show()\n    losses.append(np.mean(running_loss))\n    \nplt.plot(list(range(64, 256, 10)), losses)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"losses = []\nfor i in range(110, 256, 10):\n    sml_trn_dataset = measure_dataset((i, i))\n    sml_trn_dataloader = DataLoader(sml_trn_dataset, batch_size=64, drop_last = True)\n    running_loss, running_labels, predictions = validate_model(sml_trn_dataloader, net)\n    plt.scatter(running_labels, running_loss)\n\n    predictions_df = pd.DataFrame(predictions)\n    print(\"image size:\", str(i))\n    print(predictions_df.describe())\n    predictions_df.hist(bins = 20)\n    plt.title(\"distribution of predictions of custom NN\")\n    plt.show()\n    losses.append(np.mean(running_loss))\n    \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.plot(list(range(110, 256, 10)), losses)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}