{"cells":[{"metadata":{},"cell_type":"markdown","source":"# [Pre-fit VGG Model](https://machinelearningmastery.com/how-to-visualize-filters-and-feature-maps-in-convolutional-neural-networks/)\nWe need a model to visualize.\n\nInstead of fitting a model from scratch, we can use a pre-fit prior state-of-the-art image classification model.\n\n[Keras](https://keras.io/applications/) provides many examples of well-performing image classification models developed by different research groups for the ImageNet Large Scale Visual Recognition Challenge, or ILSVRC. One example is the VGG-16 model that achieved top results in the 2014 competition.\n\nThis is a good model to use for visualization because it has a simple uniform structure of serially ordered convolutional and pooling layers, it is deep with 16 learned layers, and it performed very well, meaning that the filters and resulting feature maps will capture useful features. For more information on this model, see the 2015 paper “Very Deep Convolutional Networks for Large-Scale Image Recognition.”\n\nWe can load and summarize the VGG16 model with just a few lines of code; for example:"},{"metadata":{"trusted":true},"cell_type":"code","source":"\n\n# load vgg model\nfrom keras.applications.vgg16 import VGG16\n# load the model\nmodel = VGG16()\n# summarize the model\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# How to Visualize Filters\nPerhaps the simplest visualization to perform is to plot the learned filters directly.\n\nIn neural network terminology, the learned filters are simply weights, yet because of the specialized two-dimensional structure of the filters, the weight values have a spatial relationship to each other and plotting each filter as a two-dimensional image is meaningful (or could be).\n\nThe first step is to review the filters in the model, to see what we have to work with.\n\nThe model summary printed in the previous section summarizes the output shape of each layer, e.g. the shape of the resulting feature maps. It does not give any idea of the shape of the filters (weights) in the network, only the total number of weights per layer.\n\nWe can access all of the layers of the model via the model.layers property.\n\nEach layer has a layer.name property, where the convolutional layers have a naming convolution like block#_conv#, where the ‘#‘ is an integer. Therefore, we can check the name of each layer and skip any that don’t contain the string ‘conv‘."},{"metadata":{"trusted":true},"cell_type":"code","source":"\n# summarize filter shapes\nfor layer in model.layers:\n    # check for convolutional layer\n    print(layer.name)\n    if 'conv' not in layer.name:\n        continue\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Each convolutional layer has two sets of weights.\n\nOne is the block of filters and the other is the block of bias values. These are accessible via the layer.get_weights() function. We can retrieve these weights and then summarize their shape."},{"metadata":{"trusted":true},"cell_type":"code","source":"\n# summarize filters in each convolutional layer\nfrom keras.applications.vgg16 import VGG16\nfrom matplotlib import pyplot\n# load the model\nmodel = VGG16()\n# summarize filter shapes\nfor layer in model.layers:\n\t# check for convolutional layer\n\tif 'conv' not in layer.name:\n\t\tcontinue\n\t# get filter weights\n\tfilters, biases = layer.get_weights()\n\tprint(layer.name, filters.shape)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\nWe can see that all convolutional layers use 3×3 filters, which are small and perhaps easy to interpret.\n\nAn architectural concern with a convolutional neural network is that the depth of a filter must match the depth of the input for the filter (e.g. the number of channels).\n\nWe can see that for the input image with three channels for red, green and blue, that each filter has a depth of three (here we are working with a channel-last format). We could visualize one filter as a plot with three images, one for each channel, or compress all three down to a single color image, or even just look at the first channel and assume the other channels will look the same. The problem is, we then have 63 other filters that we might like to visualize.\n\nWe can retrieve the filters from the first layer as follows:"},{"metadata":{"trusted":true},"cell_type":"code","source":"\n# normalize filter values to 0-1 so we can visualize them\nf_min, f_max = filters.min(), filters.max()\nfilters = (filters - f_min) / (f_max - f_min)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we can enumerate the first six filters out of the 64 in the block and plot each of the three channels of each filter.\n\nWe use the matplotlib library and plot each filter as a new row of subplots, and each filter channel or depth as a new column."},{"metadata":{"trusted":true},"cell_type":"code","source":"f = pyplot.figure(figsize=(16,16))\n# plot first few filters\nn_filters, ix = 6, 1\nfor i in range(n_filters):\n\t# get the filter\n\tf = filters[:, :, :, i]\n\t# plot each channel separately\n\tfor j in range(3):\n\t\t# specify subplot and turn of axis\n\t\tax = pyplot.subplot(n_filters, 3, ix)\n\t\tax.set_xticks([])\n\t\tax.set_yticks([])\n\t\t# plot filter channel in grayscale\n\t\tpyplot.imshow(f[:, :, j], cmap='gray')\n\t\tix += 1\n# show the figure\npyplot.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"ying this together, the complete example of plotting the first six filters from the first hidden convolutional layer in the VGG16 model is listed below."},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.applications.vgg16 import VGG16\nfrom matplotlib import pyplot\nf = pyplot.figure(figsize=(16,16))\n\n# load the model\nmodel = VGG16()\n# retrieve weights from the second hidden layer\nfilters, biases = model.layers[1].get_weights()\n# normalize filter values to 0-1 so we can visualize them\nf_min, f_max = filters.min(), filters.max()\nfilters = (filters - f_min) / (f_max - f_min)\n# plot first few filters\nn_filters, ix = 6, 1\nfor i in range(n_filters):\n\t# get the filter\n\tf = filters[:, :, :, i]\n\t# plot each channel separately\n\tfor j in range(3):\n\t\t# specify subplot and turn of axis\n\t\tax = pyplot.subplot(n_filters, 3, ix)\n\t\tax.set_xticks([])\n\t\tax.set_yticks([])\n\t\t# plot filter channel in grayscale\n\t\tpyplot.imshow(f[:, :, j], cmap='gray')\n\t\tix += 1\n# show the figure\npyplot.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Running the last example creates a figure with six rows of three images, or 18 images, one row for each filter and one column for each channel\n\nWe can see that in some cases, the filter is the same across the channels (the first row), and in others, the filters differ (the last row).\n\nThe dark squares indicate small or inhibitory weights and the light squares represent large or excitatory weights. Using this intuition, we can see that the filters on the first row detect a gradient from light in the top left to dark in the bottom right.\n\n\nAlthough we have a visualization, we only see the first six of the 64 filters in the first convolutional layer. Visualizing all 64 filters in one image is feasible.\n\nSadly, this does not scale; if we wish to start looking at filters in the second convolutional layer, we can see that again we have 64 filters, but each has 64 channels to match the input feature maps. To see all 64 channels in a row for all 64 filters would require (64×64) 4,096 subplots in which it may be challenging to see any detail."},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport skimage.io\nfrom skimage.transform import resize\nfrom imgaug import augmenters as iaa\nfrom tqdm import tqdm\nimport PIL\nfrom PIL import Image, ImageOps\nimport cv2\nfrom sklearn.utils import class_weight, shuffle\nfrom keras.losses import binary_crossentropy, categorical_crossentropy\n#from keras.applications.resnet50 import preprocess_input\nfrom keras.applications.densenet import DenseNet121,DenseNet169\nimport keras.backend as K\nimport tensorflow as tf\nfrom sklearn.metrics import f1_score, fbeta_score, cohen_kappa_score\nfrom keras.utils import Sequence\nfrom keras.utils import to_categorical\nfrom sklearn.model_selection import train_test_split\nimport imgaug as ia\n\nWORKERS = 2\nCHANNEL = 3\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")\nSIZE = 300\nNUM_CLASSES = 5","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train = pd.read_csv('../input/train.csv')\ndf_test = pd.read_csv('../input/test.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def display_samples(df, columns=4, rows=3):\n    fig=plt.figure(figsize=(5*columns, 4*rows))\n\n    for i in range(columns*rows):\n        image_path = df.loc[i,'id_code']\n        image_id = df.loc[i,'diagnosis']\n        img = cv2.imread(f'../input/train_images/{image_path}.png')\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n        \n        fig.add_subplot(rows, columns, i+1)\n        plt.title(image_id)\n        plt.imshow(img)\n    \n    plt.tight_layout()\n\ndisplay_samples(df_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x = df_train['id_code']\ny = df_train['diagnosis']\n\nx, y = shuffle(x, y, random_state=8)\ny.hist()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y = to_categorical(y, num_classes=NUM_CLASSES)\ntrain_x, valid_x, train_y, valid_y = train_test_split(x, y, test_size=0.15,\n                                                      stratify=y, random_state=8)\nprint(train_x.shape)\nprint(train_y.shape)\nprint(valid_x.shape)\nprint(valid_y.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We know the result will be a feature map with 224x224x64. We can plot all 64 two-dimensional images as an 8×8 square of images.\n\n"},{"metadata":{},"cell_type":"markdown","source":"Tying all of this together, the complete code example of visualizing the feature map for the first convolutional layer in the VGG16 model for a bird input image is listed below."},{"metadata":{"trusted":true},"cell_type":"code","source":"# plot feature map of first conv layer for given image\nfrom keras.applications.vgg16 import VGG16\nfrom keras.applications.vgg16 import preprocess_input\nfrom keras.preprocessing.image import load_img\nfrom keras.preprocessing.image import img_to_array\nfrom keras.models import Model\nfrom matplotlib import pyplot \nfrom numpy import expand_dims\n\n\nf = plt.figure(figsize=(16,16))\n# load the modelf = plt.figure(figsize=(10,3))\nmodel = VGG16()\n# redefine model to output right after the first hidden layer\nmodel = Model(inputs=model.inputs, outputs=model.layers[1].output)\nmodel.summary()\n# load the image with the required shape\nimg = load_img(f'../input/test_images/270a532df702.png', target_size=(224, 224))\n# convert the image to an array\nimg = img_to_array(img)\n# expand dimensions so that it represents a single 'sample'\nimg = expand_dims(img, axis=0)\n# prepare the image (e.g. scale pixel values for the vgg)\nimg = preprocess_input(img)\n# get feature map for first hidden layer\nfeature_maps = model.predict(img)\n# plot all 64 maps in an 8x8 squares\nsquare = 8\nix = 1\nfor _ in range(square):\n\tfor _ in range(square):\n\t\t# specify subplot and turn of axis\n\t\tax = pyplot.subplot(square, square, ix)\n\t\tax.set_xticks([])\n\t\tax.set_yticks([])\n\t\t# plot filter channel in grayscale\n\t\tpyplot.imshow(feature_maps[0, :, :, ix-1], cmap='viridis')\n\t\tix += 1\n# show the figure\npyplot.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is an interesting result and generally matches our expectation. We could update the example to plot the feature maps from the output of other specific convolutional layers.\n\nAnother approach would be to collect feature maps output from each block of the model in a single pass, then create an image of each.\n\nThere are five main blocks in the image (e.g. block1, block2, etc.) that end in a pooling layer. The layer indexes of the last convolutional layer in each block are [2, 5, 9, 13, 17].\n\nWe can define a new model that has multiple outputs, one feature map output for each of the last convolutional layer in each block; for example:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# visualize feature maps output from each block in the vgg model\nfrom keras.applications.vgg16 import VGG16\nfrom keras.applications.vgg16 import preprocess_input\nfrom keras.preprocessing.image import load_img\nfrom keras.preprocessing.image import img_to_array\nfrom keras.models import Model\nimport matplotlib.pyplot as plt\nfrom numpy import expand_dims\n\n\n\n\n# load the model\nmodel = VGG16()\n# redefine model to output right after the first hidden layer\nixs = [2, 5, 9, 13, 17]\noutputs = [model.layers[i].output for i in ixs]\nmodel = Model(inputs=model.inputs, outputs=outputs)\n# load the image with the required shape\n# convert the image to an array\nimg = load_img(f'../input/test_images/270a532df702.png', target_size=(224, 224))\n# convert the image to an array\nimg = img_to_array(img)\n# expand dimensions so that it represents a single 'sample'\nimg = expand_dims(img, axis=0)\n# prepare the image (e.g. scale pixel values for the vgg)\nimg = preprocess_input(img)\n# get feature map for first hidden layer\nfeature_maps = model.predict(img)\n# plot the output from each block\nsquare = 8\nfor fmap in feature_maps:\n\t# plot all 64 maps in an 8x8 squares\n\tix = 1\n\tfor _ in range(square):\n\t\tplt.figure(figsize=(64,64))\n\t\tfor _ in range(square):\n           \n\n\t\t\t# specify subplot and turn of axis\n\t\t\tax = pyplot.subplot(square, square, ix)\n\t\t\tax.set_xticks([])\n\t\t\tax.set_yticks([])\n\t\t\t\n\t\t\t# plot filter channel in grayscale\n\t\t\tplt.imshow(fmap[0, :, :, ix-1], cmap='viridis')\n\t\t\tix += 1\n\t# show the figure\n\n        \n\tplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Running the example results in five plots showing the feature maps from the five main blocks of the VGG16 model.\n\nWe can see that the feature maps closer to the input of the model capture a lot of fine detail in the image and that as we progress deeper into the model, the feature maps show less and less detail.\n\nThis pattern was to be expected, as the model abstracts the features from the image into more general concepts that can be used to make a classification. Although it is not clear from the final image that the model saw a bird, we generally lose the ability to interpret these deeper feature maps."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}