{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Visualising CNN layers\nThis notebook is about visualising aspects of CNNs that I find interesting and was curious to learn more about. Let's begin! ☠ ☠"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"from fastai import *\nfrom fastai.vision.all import *\nimport pandas as pd\nimport numpy as np\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom matplotlib import animation, rc\nfrom IPython.display import HTML","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"path = Path('../input/ranzcr-clip-catheter-line-classification')\n# All labels train\ntrain_all_lab = pd.read_csv('../input/ranzcr-clip-catheter-line-classification/train.csv').drop(columns = \"PatientID\")\n# Condensed labels train\nann = pd.read_csv(\"../input/ranzcr-clip-catheter-line-classification/train_annotations.csv\")[['StudyInstanceUID', 'label']]\nann['StudyInstanceUID'] = ann['StudyInstanceUID']+ \".jpg\"\nann.columns = ['name', 'label']\n# Converting the input to multilabel\nann = ann.sort_values(['name', 'label']).groupby('name')['label'].unique().agg({'label': ';'.join}).reset_index().drop(columns = \"level_0\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# EDA\n\nThe dataset contains images about the position and presence of catheters and tubes in medical images. The following categories are present:\n* ETT - Abnormal - endotracheal tube placement abnormal\n* ETT - Borderline - endotracheal tube placement borderline abnormal\n* ETT - Normal - endotracheal tube placement normal\n* NGT - Abnormal - nasogastric tube placement abnormal\n* NGT - Borderline - nasogastric tube placement borderline abnormal\n* NGT - Incompletely Imaged - nasogastric tube placement inconclusive due to imaging\n* NGT - Normal - nasogastric tube placement borderline normal\n* CVC - Abnormal - central venous catheter placement abnormal\n* CVC - Borderline - central venous catheter placement borderline abnormal\n* CVC - Normal - central venous catheter placement normal\n* Swan Ganz Catheter Present"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"corr_inp = train_all_lab.drop(columns = \"StudyInstanceUID\").corr()\nmask = np.triu(np.ones_like(corr_inp, dtype = bool))\ncmap = sns.diverging_palette(230, 20, as_cmap=True)\nfig, ax = plt.subplots(1, 2, figsize = (12, 6))\ntrain_all_lab.drop(columns = 'StudyInstanceUID').sum().plot(kind = 'barh', ax = ax[0])\nax.ravel()[0].invert_yaxis()\nax.ravel()[0].set(ylabel = '', title = 'Category occurrence')\nsns.heatmap(corr_inp, mask=mask, cmap=cmap, center=0, square=True, linewidths=1, cbar_kws={\"shrink\": .5}, ax = ax[1]).set_title('Correlation between categories')\nfig.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the left chart it becomes clear that the most common category is CVC - Normal. The abnormal categories for ETT and NGT are very rare followed by the presence of a Swan Ganz Catheter.  \n\nThere are correlations between the categories as well: The ETT (endotracheal tube) is correlated with the NGT (nasogastric tube) but without medical knowledge is hard to tell exactly why (anyone with medical knowledge please share!). \n\n## What are the combinations of categories present in the images?"},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":false},"cell_type":"code","source":"ax = ann.groupby('label').size().sort_values(ascending = False).head(15).plot(kind = 'barh', figsize = (8,8))\nax.invert_yaxis()\n_ = ax.set(ylabel = '', title = 'Most frequent multi-label categories')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are multiple category combinations in our dataset. Some multi-label categories include different assessments for the same type of tube (for example in category 6 \"CVC-Borderline;CVC-Normal\" we have two CVC categories). The reason for this could be that the patient has multiple CV catheters or that there were different assessments of the placement."},{"metadata":{},"cell_type":"markdown","source":"# Dataset preparation and model training\n\nSince this notebook is about visualising the layers of the model, our batch preparation will not have any fancy augmentations.  \nWe will stick to resizing, simple normalisation and a very plain resnet50 model for our Learner."},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_x(r): return path/'train'/r['name']\ndef get_y(r): return r['label'].split(';')\ndef get_dls(bs, size, do_flip = True, xtra_tfms = None, min_scale = 0.8):\n    dblock = DataBlock(blocks = (ImageBlock, MultiCategoryBlock),\n                          get_x = get_x,\n                          get_y = get_y,\n                          item_tfms = Resize(size),\n                          batch_tfms = Normalize.from_stats(*imagenet_stats),\n                          splitter = RandomSplitter(seed = 42))\n    return dblock.dataloaders(ann, bs = bs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"set_seed(42,True)\ndls = get_dls(bs = 32, size = 350)\nlearn = cnn_learner(dls, \n                resnet50,\n                loss_func=BCEWithLogitsLossFlat(),\n                metrics=[accuracy_multi]).to_fp16()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In order for our visualisations to be meaningful, our Learner needs to be trained for some epochs in order to get a better grasp of the data and create the necessary patterns.  \nFor this purpose we are going to use the lr_find method of fastai to find a suitable learning rate and then fine_tune the model for 2 epochs (one only for the very last layer and one for the whole model)."},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.lr_find()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Based on the figure, a learning rate around 0.03 seems to be appropriate."},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.fine_tune(1, 3e-2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visualising filters and feature maps\n\nThis part of the notebook focuses on the feature maps of the convolutional layers in the model.  \nWe are going to start out with the entry layer of the network, visualise the filters and the feature maps there.  \nThe next step is getting the feature map of the last convolutional layer in order to compare the two.  \nLastly, we are going to visualise how the feature maps change as we progress through the layers of the network."},{"metadata":{},"cell_type":"markdown","source":"## Obtaining a test patient\n\nFor our visualisations we are going to need an image to work with. I picked one from the test set and it's this one here:"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"temp = PILImage.create('../input/ranzcr-clip-catheter-line-classification/test/1.2.826.0.1.3680043.8.498.10010309624955962953138679469906703941.jpg')\nimg, = first(dls.test_dl([temp]))\nx_dec = TensorImage(dls.train.decode((img,))[0][0])\nimg = img[0]\n_ = img.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The image is originally grayscale, but our model expects 3 channels as input and therefore all 3 channels get the same value (before normalisation).  \nWe can visualise this better here: \n(Notice how all three channels have the same intensity)"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"image_rgb = x_dec.numpy()\nfig, ax = plt.subplots(1, 4, figsize = (18, 30))\nax[0].imshow(image_rgb.transpose((1,2,0))) \nax[0].axis('off')\nax[0].set_title('Original', size = 15)\nfor i, cmap in enumerate(['Reds','Greens','Blues']):\n    ax[i+1].imshow(image_rgb[i,:,:], cmap=cmap) \n    ax[i+1].axis('off')\n    ax[i+1].set_title(cmap[:-1] + ' channel', size = 15)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Analysing the first layer\n\nWe start with the specifications of the first layer."},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.model[0][0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"A convolutional layer with the following specifications:\n* Input channels: 3 - these are the 3 colors of the image we use as input \n* Output channels: 64 - the channels that will be output after the convolution\n* Kernel size: 7x7 - the size of the kernel that will be applied to our images\n* Stride: 2 - the step at which the kernel will be applied on the images during the convolution\n* Padding: 3 - additional pixels that will be added to the sides of our image in order to not lose edge information"},{"metadata":{},"cell_type":"markdown","source":"### Filters weights and filters\n\nGiven that we have a kernel of 7x7 and the output has 64 channels, we are going to need a matrix of size at least 64x7x7.  \nBut this output has to be produced for each of our input channels (we have 3) and therefore we are going to need a matrix of 64x3x7x7.\nJust to double-check:"},{"metadata":{"trusted":true},"cell_type":"code","source":"weights = learn.model[0][0].weight\nweights.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"These weights are combined into filters: Every 64x7x7 filter is comprised of the 3 filter weights that we saw before.\nThese filters are the patterns that the model layer is trying to find in the image.  "},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def normalise(tens):\n    temp = tens.detach().cpu().numpy()\n    return (255*(temp - np.min(temp))/np.ptp(temp)).astype(int)   ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig,axes = plt.subplots(8, 8, figsize=(15,15))\nfor i in range(64):\n    axes.ravel()[i].axis('off')\n    axes.ravel()[i].imshow(normalise(weights[i].permute(1,2,0)), interpolation = 'bilinear')\n    axes.ravel()[i].set_title(i)\nfig.suptitle('Filters', fontsize = 20)\nfig.tight_layout()\nfig.subplots_adjust(top=0.92)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that the first layer is mainly looking for finding edges (diagonal, horizontal or vertical) as well as areas where one or two colors are more prevalent in the data.  \nWhat do the colors mean in the plot? Remember the 3 channels that we had (R, G, B). The filters that are green are more focused on the green channel of our input, the filters that are red on the red channel etc. The yellow images for example are more focused on the red and green channels compared to the blue one.\nLet us have a look at one of the yellow images to see if the underlying weight tensor justifies our beliefs."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"vmax_weights, vmin_weights = weights[60].max(), weights[60].min()\nfig, ax = plt.subplots(1, 4, figsize = (20, 20))\nax[0].imshow(normalise(weights[60].permute(1,2,0)), interpolation = 'bilinear') \nax[0].axis('off')\nax[0].set_title('Filter', size = 15)\nfor i, cmap in enumerate(['Reds','Greens','Blues']):\n    ax[i+1].imshow(weights[60].detach().cpu().numpy()[i,:,:], cmap=cmap, vmin = vmin_weights, vmax = vmax_weights, interpolation = 'bilinear') \n    ax[i+1].axis('off')\n    ax[i+1].set_title(cmap[:-1] + ' Channel', size = 15)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The images show that the red and green channels are more intense in the bottom right quadrant than the blue one and that is how the yellow filter gets produced.  \nOn the other hand the bottom left quadrant is more intense for the blue channel and therefore the blue color is more pronounced in that area of the filter.\n\nAn important question I had when writing this kernel is the following: The input of the model is using the same values in each of the 3 input channels, how does it manage to have different filters for each channel? My expectation would be that they would be the same. There are 3 parameters in play:\n1. Pretraining. The model we are using (resnet50) has been trained on the Imagenet dataset (which has colored pictures) and while the weights do get updated by our training, there is always residual\n2. Image normalization with Imagenet stats. The starting image has the same values in all 3 channels but the normalisation changes this to some extent"},{"metadata":{},"cell_type":"markdown","source":"### Time to get some hooks in!\n\nPytorch hooks are great in order to get a better understanding of the internal operations of a model.  \nThere are 2 types of hooks available:\n* Forward hooks (also a forward -pre hook but we are not going to touch on that)\n* Backward hooks  \n\nThe main properties of hooks are:\n1. They can be attached to a specific layer of the model \n2. They have access to the input and output of that particular layer\n3. They are triggered/executed when either the forward pass of that layer takes place (forward hook) or when the backward pass takes place (backward hook)\n4. You can attach to them custom functions that can use the data of the hook layer"},{"metadata":{},"cell_type":"markdown","source":"### Baby steps - Hooking the first convolutional layer"},{"metadata":{"trusted":true},"cell_type":"code","source":"class fHook():\n    def hook_f(self, module, inp, out): \n        self.inp_stored = inp\n        self.out_stored = out.detach().clone()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"hook_data = fHook() # Creating an instance of the hook\nhook_new = learn.model[0][0].register_forward_hook(hook_data.hook_f) # Attaching it to the first layer of the model\nwith torch.no_grad(): output = learn.model.eval()(img.unsqueeze(0)) # Propagating an image through the network to get all the layers working\nhook_new.remove() # Removing the hook","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Having hooked the first layer, we have access now to the input and output of the layer.  \n**Important point: the input of the layer is stored as a tensor inside a tuple while the output is a tensor!**"},{"metadata":{"trusted":true},"cell_type":"code","source":"hook_data.inp_stored[0].shape, hook_data.out_stored.shape, ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\nDid we expect these inputs and outputs?  \n**Yes!**    \n* The input of the first layer in our network is the image we passed to our model converted to a tensor and resized to 350 pixels (1x3x350x350).  \n\nThe output (1x64x175x175) can be explained as follows:\n* The 1 at the first dimension of the tensor is the number of inputs (1 image --> 1 input) \n* 175x175 is the size of the output image and it can be derived from the original input size (350) via the following equation: $ \\text{output size} = (\\text{input size} + 2*\\text{padding} - \\text{kernel size}) // \\text{stride} + 1 $. Substituting our figures we end up with: $(350 + 2*3 - 7) // 2 + 1 = 349 // 2 + 1 = 174 + 1 = 175$  \n* The 64 is a bit trickier than it looks:  \n  We started out with 3 channels and we used filter weights of 64x3x7x7, therefore one would expect the output not to be 64 but to be 64x3.  \n  The hidden calculation here is that after we derive the 64 outputs for each of the 3 channels we had as input, we add them together!\n\n\n| ![Final addition](https://raw.githubusercontent.com/fastai/fastbook/8be580737ee0cc17746a5ed68283150d489b3dc4/images/chapter9_rgb_conv_stack.svg) | \n|:--:| \n| *Image taken from fastbook https://github.com/fastai/fastbook/blob/master/13_convolutions.ipynb* |\n\n**Sanity checks for our hook results:**\n1. The input from the hook is exactly equal to the input tensor\n2. The output is the result of doing a convolution between the input image and the filter weights"},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Sanity check 1: ' + str(torch.equal(img.unsqueeze(0), hook_data.inp_stored[0])))\nprint('Sanity check 2: ' + str(torch.equal(F.conv2d(img.unsqueeze(0), weights, padding = 3, stride = 2), hook_data.out_stored)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Feature maps\n\nFeature maps highlight the patterns that the filters managed to find in the input. "},{"metadata":{"trusted":true},"cell_type":"code","source":"fm = hook_data.out_stored","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig,axes = plt.subplots(8, 8, figsize=(15,15))\nfor i in range(64):\n    axes.ravel()[i].imshow(fm[0][i].detach().cpu(), interpolation='bilinear')\n    axes.ravel()[i].axis('off')\n    axes.ravel()[i].set_title(i)\nfig.suptitle('Feature maps', fontsize = 20)\nfig.tight_layout()\nfig.subplots_adjust(top=0.92)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"These are the activations of the first layer. The output of convolving our original image with the weights we visualized earlier.  \n**Important note: The color of the image reflects how high (or low) a pixel value is in comparison to the other pixels in the same picture. As an example in picture 1, the really light pixels have a high value in comparison to the dark pixels.**\n\nThere are some interesting patterns in the pictures:\n* The original image is still clearly visible in the result of the convolution\n* If a filter pattern gets discovered the particular areas where it gets discovered will show up lighter in the images. This does not mean that the filters which are uniform as useless, it simply means that for this **particular image** their patterns are not present!"},{"metadata":{},"cell_type":"markdown","source":"### One step further - Hooking the last convolutional layer"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"hook_data = fHook() # Creating an instance of the hook\nhook_new = learn.model[0][-1].register_forward_hook(hook_data.hook_f) # Attaching it to the last layer of the model\nwith torch.no_grad(): output = learn.model.eval()(img.unsqueeze(0)) # Propagating an image through the network to get all the layers working\nhook_new.remove() # Removing the hook","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The last convolutional layer is comprised of 3 Bottleneck constructions but the final output is the result of a convolutions(512, 2048) followed by a BatchNorm2d(2048) and then a ReLU (all belonging to the last Bottleneck)."},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.model[0][-1][-1]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It is always good to double-check our expectations. The output size should have 2048 channels."},{"metadata":{"trusted":true},"cell_type":"code","source":"fm_last = hook_data.out_stored\nfm_last.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now that we have checked the output size, let's have a look at some feature maps. Given that there are 2048 of them it would be pretty tough to view all of them at once so we will stick to a random sample of 64:"},{"metadata":{"trusted":true},"cell_type":"code","source":"sample64 = random.sample(range(2048), 64)\nfig,axes = plt.subplots(8, 8, figsize=(15,15))\nfor i,ind in enumerate(sample64):\n    axes.ravel()[i].imshow(fm_last[0][ind].detach().cpu(), interpolation='bilinear')\n    axes.ravel()[i].axis('off')\n    axes.ravel()[i].set_title(ind)\nfig.suptitle('Feature maps', fontsize = 20)\nfig.tight_layout()\nfig.subplots_adjust(top=0.92)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I think it is pretty conclusive that after all the layers and transformations the image has gone through it is pretty hard to distinguish any resemblance to our original lovely skeleton."},{"metadata":{},"cell_type":"markdown","source":"## How do the convolutions change as we progress through the network?\nSo far we have seen how the feature maps at the first convolutional layer look like as well as the output (after a batchnorm and ReLu) of the last convolutional layer. \nBut how do the outputs change from layer to layer?  \nWe can only answer this question by hooking every convolutional layer in the model!"},{"metadata":{"trusted":true},"cell_type":"code","source":"conv_out = []\nhook_data = fHook()\nfor layer in learn.model.modules():\n    if isinstance(layer, torch.nn.modules.conv.Conv2d):\n        hook_handler = layer.register_forward_hook(hook_data.hook_f)\n        with torch.no_grad(): output = learn.model.eval()(img.unsqueeze(0))\n        conv_out.append(hook_data.out_stored[0])\n        hook_handler.remove()\n        ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Before we delve deeper into the feature maps, it's good to get a basic understanding of what types of layers are present in the model.  "},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"cv_sizes = pd.DataFrame([(x.shape[0], x.shape[1]) for x in conv_out])\ncv_sizes.columns = ['Feature maps', 'Pixel size']\ncv_sizes = cv_sizes.groupby(['Feature maps', 'Pixel size']).size().to_frame('Layers').reset_index()\n_ = sns.scatterplot(x = cv_sizes['Feature maps'], y = cv_sizes['Pixel size'], size = cv_sizes['Layers'])\n_ = plt.title('Breakdown of convolutional layer size')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The majority of the layers have 250 activations with a pixel size of 25, but there are plenty of combinations available in the model.  \n\nIn order to better visualise how the feature maps change as we go deeper into the network we are going to use an animation showcasing a sample of 15 feature maps at each convolutional layer of the model. This will give a better representation of how the evolution from the clear feature maps of the first layer to the unrecognizeable ones of the last one happens."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, ax = plt.subplots(3, 5, figsize=(5,3), dpi = 150)\ntitle = plt.suptitle(t='', fontsize = 12, family = 'fantasy')\nfig.subplots_adjust(top=0.92)\n\ndef animate_tensors(i, pic_num = 15):\n    tens = conv_out[i]\n    \n    tens_positions = random.sample(range(tens.shape[0]), pic_num)\n    to_plot_maps = tens[tens_positions]\n    \n    for j in range(pic_num):\n        ax.ravel()[j].imshow(to_plot_maps[j].detach().cpu(), interpolation='bilinear')\n        ax.ravel()[j].axis('off')\n        title.set_text('Feature map {}'.format(i))\n        plt.tight_layout()\n\nanim = animation.FuncAnimation(fig, animate_tensors, frames= len(conv_out), interval=1500, repeat=True)\n\nplt.close()\nHTML(anim.to_html5_video())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I stop being able to see anything relatable to the original picture at around Feature map 25. What about you? 🤔"},{"metadata":{},"cell_type":"markdown","source":"# Working backward from output classes to activations - Grad CAM\n\nAnother type of visualisation we can try is the following:\nOur model predicts category X for this image. On which areas of the image did the model focus on for obtaining that prediction?\n\nIn order to address this question, we are going to implement the Grad-CAM that was first suggested in this paper: [Grad-CAM paper](https://arxiv.org/abs/1610.02391)\n\nMore specifically, we are going to: \n1. Use a forward hook in order to get the activations of the last layer in the convolutional part of the resnet\n2. Use a backward hook in order to get the gradients of the layer"},{"metadata":{},"cell_type":"markdown","source":"## Simple approach"},{"metadata":{"trusted":true},"cell_type":"code","source":"class fHook():\n    def hook_fwd_f(self, module, inp, out): \n        self.out_stored = out.detach().clone()\n        \nclass bHook():\n    def hook_bwd_f(self, module, inp, out):\n        self.out_stored = out[0].detach().clone()    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## Forward hook\nfwd_hook = fHook()\nfwd_handler = learn.model[0].register_forward_hook(fwd_hook.hook_fwd_f)\nlearn.model.eval()(img.unsqueeze(0))\nact = fwd_hook.out_stored #Storing the activation\nfwd_handler.remove()\n\n## Backward hook\ngrads = []\nbwd_hook = bHook()\nfor cls in range(11):\n    bwd_handler = learn.model[0].register_backward_hook(bwd_hook.hook_bwd_f)\n    output = learn.model.eval()(img.unsqueeze(0))\n    output[0,cls].backward()\n    grads.append(bwd_hook.out_stored) #Storing the gradients\n    bwd_handler.remove()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The forward part of the code is exactly the same as before. We simply store the activation.  \n\nThe backward part is different. We are using a for loop in order to calculate the gradients for each of the output classes that we have in the data (the 11 categories of our images).  \nThis is needed because the grad can only be created for **scalar outputs**. In simple terms this means that PyTorch can calculate the backwards gradient only for each individual class.  \nTherefore, we do the same process for all 11 of our categories and store them."},{"metadata":{},"cell_type":"markdown","source":"## More complicated but better approach\n\nThe below implementation is taken from this chapter in the fastai course: [Chapter 18 fastai](https://github.com/fastai/fastbook/blob/master/18_CAM.ipynb). The logic is to turn the hooks into a context manager.  \nEssentially, this approach allows the Hook classes to be used in a with statement without having to remove them manually afterwards. Simply hook the layer, get the data and after you are done the Hook will remove itself. For more details about context managers have a look [here](https://stackoverflow.com/questions/1984325/explaining-pythons-enter-and-exit)"},{"metadata":{"trusted":true},"cell_type":"code","source":"class Hook():\n    def __init__(self, m):\n        self.hook = m.register_forward_hook(self.hook_func)   \n    def hook_func(self, m, i, o): self.stored = o.detach().clone()\n    def __enter__(self, *args): return self\n    def __exit__(self, *args): self.hook.remove()\n        \nclass HookBwd():\n    def __init__(self, m):\n        self.hook = m.register_backward_hook(self.hook_func)\n    def hook_func(self, m, gi, go): self.stored = go[0].detach().clone()\n    def __enter__(self, *args): return self\n    def __exit__(self, *args): self.hook.remove()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"grads = []\nfor cls in range(11):\n    with HookBwd(learn.model[0]) as hookb:\n        with Hook(learn.model[0]) as hookf:\n            output = learn.model.eval()(img.unsqueeze(0))\n            act = hookf.stored\n        output[0,cls].backward()\n        grads.append(hookb.stored)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"After obtaining the activation and the gradients of the layer of interest, we need to multiply them together in order to derive the Grad-CAM values. \nMy understanding of the process is the following:  \n* The gradients are representing how important each activation pixel is for a particular class\n* The activation is representing how activated each individual pixel is.  \n**Multiplying them together shows which of the important class pixels are activated.**\n\nAfter the multiplication, we apply also a ReLU function. The reasoning behind this choice (from the paper on Grad-CAM):\n*We apply a ReLU to the linear combination of maps because we are only interested in the features that have a positive influence on the class of interest*"},{"metadata":{"trusted":true},"cell_type":"code","source":"cam_maps = []\nfor i in range(11):\n    w = grads[i][0].mean(dim=[1,2], keepdim = True)\n    cam_maps.append(F.relu((w*act[0]).sum(0)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"top_preds, top_ind= torch.topk(learn.model.eval()(img.unsqueeze(0)).sigmoid()[0], 4)\ntop_preds = np.round(top_preds.detach().cpu().detach().numpy(), 2)\ntop_ind = top_ind.tolist()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig,ax = plt.subplots(1,5, figsize = (15,10))\nfor i, ind in enumerate(top_ind, start = 1):\n    x_dec.show(ctx = ax.ravel()[i])\n    ax.ravel()[i].imshow(cam_maps[ind].detach().cpu(), alpha = 0.6, extent = (0,350,350,0), interpolation = 'bilinear', cmap = 'magma')\n    ax.ravel()[i].set_title(learn.dls.vocab[ind], size = 15)\n    ax.ravel()[i].text(200,30,'Pred: ' + str(top_preds[i-1]), color = 'red', size = 14)\n\nimg.show(ctx = ax.ravel()[0])\nfig.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The visualisations show exactly the areas that our model focuses on for each of the prediction classes. (Only the top 4 predicted classes are shown)  \nFor example, for the category **'CVC Normal'** the focus of the model is on the **central chest area**.  \nOn the other hand, for the **'ETT - Normal'** prediction, the focus is more narrowly on the area **around the throat**.   \nThe **'NGT-Normal'** prediction is more spread out but the intense part is on the bottom right.  \n*Note that this is a multi-label classification problem therefore the predictions do not sum to 1 as in the single-label case.*"},{"metadata":{},"cell_type":"markdown","source":"# Resources\n\n### Papers\nLearning Deep Features for Discriminative Localization: https://arxiv.org/abs/1512.04150  \nGrad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization: https://arxiv.org/abs/1610.02391\n\n### Hook related resources\nFastai notebook: https://github.com/fastai/fastbook/blob/master/18_CAM.ipynb  \nUnderstanding PyTorch hooks (kaggle kernel): https://www.kaggle.com/sironghuang/understanding-pytorch-hooks  \nPyTorch documentation: https://pytorch.org/tutorials/beginner/former_torchies/nnft_tutorial.html#forward-and-backward-function-hooks  \nPaperspace article: https://blog.paperspace.com/pytorch-hooks-gradient-clipping-debugging/  \nYoutube video: https://www.youtube.com/watch?v=syLFCVYua6Q\n\n### Matplotlib animations\nMatplotlib documentation: https://matplotlib.org/stable/api/animation_api.html\n\n### Context manager\nPython 3 documentation: https://docs.python.org/3/reference/datamodel.html#with-statement-context-managers  \nStackoverflow answer: https://stackoverflow.com/questions/1984325/explaining-pythons-enter-and-exit\n"},{"metadata":{},"cell_type":"markdown","source":"If you enjoyed this notebook, liked the visualisations, have questions/suggestions or comments please let me know!! And if you feel like it, give it an upvote 😊😊✨✨"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}