{"cells":[{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input/sample-images'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow import keras\nimport cv2\nimport json\nimport imageio\nimport glob\n\nfrom IPython.display import Image\nimport matplotlib.pyplot as plt\nimport matplotlib.cm as cm\n\nFILE_PATH = \"/kaggle/input/sample-images/\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# GradCAM, Model Interpretability - VGG16 & Xception Networks\n\n### Objective\n\nThe objective of this post is to understand the importance of \"Visual Explanations\" for CNN based large scale Deep Neural Network Models.  \nI will be proud if one feels, this post is nothing but a commentary on **R.R Selvaraju et al** paper in **IEEE ICCV, 2017** title [Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization](https://ieeexplore.ieee.org/document/8237336)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Goal \nDecipher these Formulas and Program them using Keras and Numpy\n$$y_c = \\frac{\\partial{y^c}}{\\partial{A^k}}$$\n\n$$Weights_{NeuronImportance} = \\alpha^c_k = \\frac{1}{Z} \\sum_{i=1}^u \\sum_{i=1}^{v} y_c$$\n$$i.e.$$\n$$\\alpha^c_k = \\frac{1}{Z} \\sum_{i=1}^u \\sum_{i=1}^{v} \\frac{\\partial{y^c}}{\\partial{A^k}}$$","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Top 6 Predictions and GradCAM Heatmaps of VGG16 for a Sample Image**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![Heatmap](https://raw.githubusercontent.com/gowrishankarin/data_science/master/topics/dl/model_explanation/anime.gif)","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"display(Image(FILE_PATH+\"vgg16_dock.png\"))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Is this topic relevant to SIIM-ISIC Melanoma Detection?  \nYes, Its is. If you feel the same, please make me aware - We are in same page.\n\n### Notebook Organization  \n**PART 1: GradCAM Theory**    \n- Problem Statement\n- Seeing is Believing\n- Grad-CAM Rationale\n- The Approach\n- The Algorithm - Math Involved\n    - Gradient calculation via Backpropagation\n    - Global Average Pooled Gradients\n    - ReLU activation on Weights\n\n**PART 2: Demo using Keras & Tensorflow**    \n**Section 1**.  \n- VGG16 and Xception Network Properties\n- Image Preprocessing\n- Step by Step Implementation of Heatmap Calculation\n- Superimposition of Heatmap on Original Image\n\n**Section 2**.  \n- Exploration of Top - 9 Predictions and their Heatmap\n\n**Section 3**.  \n- Compute GradCAM heatmaps for predefined classes on an image.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# GradCAM Theory\n\n## The Problem.  \nSignificant breakthroughs are achieved in computer vision during the past decade or so using Deep Neural Networks based on Convolutional Neural Network architecture. Following are some of the key areas where CNN based models drew extraordinary results,\n1. Image Classification\n2. Object Detection\n3. Semantic Segmentation\n4. Image Captioning \n5. Visual Question Answering \n6. Visual Dialog and \n7. Embodied Question Answering  \n\nThough the performance of them are great and intuitive, they fail when it comes to model explanation and interpretability. \n- There is always a trade-off between accuracy and interpretability\n- Lack of decomposability result in lack of explainable/interpretable models\n- High complexity make interpretability hard. For e.g 200+ layer depth models like ResNets \n- Explainable models are usually lack accuracy scores \n\nHowever by making a model explainable and interpretable trust and faith on the model naturally increases exponentially.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### GradCAM Architecture\nImage Ref: [http://gradcam.cloudcv.org/](http://gradcam.cloudcv.org/)","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":false},"cell_type":"code","source":"display(Image('http://gradcam.cloudcv.org/static/images/network.png'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Seeing is Believing.    \nBuilding a model that have the ability to explain why they predict what they predict results in transparency and trustworthiness. This is achieved by\n- Highlighting important pixels of predictios \n- Visualizing partial derivatives of predicted scores wrt pixel intensities\n- Deconvolution by making modifications to raw gradients \n- Methods that sythesize images to maximally activate a network unit\n- Invert a latent representation\n\nThe above all has good visual explainability. However they are not class discriminative. They tend to result in visualizations wrt different classes are nearly identical and lack finesse\n\nThen what are the criteria for a good visual explanation.  \n- Class Discriminative - i.e. localize the category in the image\n- High Resolution - i.e. Capture fine-grained detail","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Why Grad-CAM?  \nGradient-weighted Class Activation Mapping(GradCAM) techniques comes with comprehensive answers for all above challenges. \n- They are class-discriminative using localization techniques\n- They generate comprehendable visual explantaion for most of the CNN models\n- They work without changing the architecture of the model\n- They do not seek retraining the model\n- They can be applied to widely acclaimed models lik ResNets, Xception, Inception etc with minimum effort\n- They perform well with CNN+LSTM or Text Explanationable networks","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## The Approach  \n- Single shot localization through single forward and a partial backward pass per image\n- Convolutional layers naturally retail spatial information that is lost in fully connected layers\n- Last Conv layers to have the best compromize between high level semantics and detailed spatial info\n- Neurons here look for semantic class specific info in the image\n- GradCAM uses gradient information flow into the last conv layer to assign importance values to each neurons for a class of interest\n- This can be applied for any layer of a DNN","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## The Algorithm.  \nThe neurons in the last convolution layers concentrate on semantic class specific info of the input image. That is object/class specific parts of the whole image through segmentation. GradCAM fetches the gradient information flowing and assign importance values to each neurons.\n\n### Gradient via Backpropagation\nLet us call,  \nThe class discriminative localization map is $L_{GradCAM}^c \\in \\ R ^{u \\times v}$\n\nWhere,  \n$L$ is the GradCAM for a particular class $c$  \n$u$ and $v$ are the width and height of the class $c$  \n\nFurther,  \n$y_c$ is the gradient score for the class $c$ before softmax classification  \nGradient score is wrt $A^k$ the feature map activations of the last convolution layer. ie\n\n$$y_c = \\frac{\\partial{y^c}}{\\partial{A^k}}\\tag{1. Gradients via Backpropagation}$$","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Global Average Pooling on Activation Maps\nUsually the gradients $y_c$ flown back are Globally Average pooled over width and height but vary wrt architecture. Ref profperties of VGG16 and Xception. \n\n$$Weights_{NeuronImportance} = \\alpha^c_k = \\frac{1}{Z} \\sum_{i=1}^u \\sum_{i=1}^{v} y_c \\tag{Global Average Pooled Gradients}$$\n$$i.e.$$\n$$\\alpha^c_k = \\frac{1}{Z} \\sum_{i=1}^u \\sum_{i=1}^{v} \\frac{\\partial{y^c}}{\\partial{A^k}} \\tag{2}$$","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### GradCAM Activation Maps\n- The weights calculated represents the partial linearization of the DNN downstream\n- It captures the importance feature map $k$ for the target class $c$\n\nPerform ReLU on the weighted combination of forward activation maps\n\n$$L_{GradCAM}^c = ReLU(\\,\\sum_{i=1}^{k}\\alpha^c_k) \\,$$\n- This operation results in coarse heatmap of the same size as the convolution feature maps (VGG16 $(14 \\times 14)$, Xception $(10 \\times 10)$)\n- a ReLU is applied because, only the positive influence on the class of interest is considered\n- i.e Negative influence are likedly to belong to other categories in the image\n- Note: $y_c$ need not be the class score produced by an image classification, it could be any differentiable activation including words from a caption or answer to a question, in machine translation problems.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Demo using Keras & Tensorflow","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### VGG16 and Xception Properties\n\nWe shall demonstrate GradCAM approach on 2 widely accepted CNN Networks VGG16 and Xception. Following are the properties and one could extend this to other networks...\n\n**VGG16**  \n- Input Image Size is (224, 224)\n- Last Convolution Layer Name:  block5_conv3\n- Last Classifier Layers after Conv Layers: 5\n- Heatmap Dimension: (14, 14)  \n\nRef Image: [Research Gate, Max Fergusan](https://www.researchgate.net/figure/Fig-A1-The-standard-VGG-16-network-architecture-as-proposed-in-32-Note-that-only_fig3_322512435)","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":false},"cell_type":"code","source":"display(Image(\"https://www.researchgate.net/profile/Max_Ferguson/publication/322512435/figure/fig3/AS:697390994567179@1543282378794/Fig-A1-The-standard-VGG-16-network-architecture-as-proposed-in-32-Note-that-only.png\"))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Xception**  \n- Input Image Size is (299, 299)\n- Last Convolution Layer Name:  block14_sepconv2_act\n- Last Classifier Layers after Conv Layers: 2\n- Heatmap Dimension: (10, 10)\n\nRef Image: [Review of Xception by Sik-Ho Tsang](https://towardsdatascience.com/review-xception-with-depthwise-separable-convolution-better-than-inception-v3-image-dc967dd42568)","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":false},"cell_type":"code","source":"display(Image(FILE_PATH+\"xception.png\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"properties = {\n    \"vgg16\": {\n        \"img_size\": (224, 224),\n        \"last_conv_layer\": \"block5_conv3\",\n        \"last_classifier_layers\": [\n            \"block5_pool\",\n            \"flatten\",\n            \"fc1\",\n            \"fc2\",\n            \"predictions\",\n        ],\n        \"model_builder\": keras.applications.vgg16.VGG16,\n        \"preprocess_input\": keras.applications.vgg16.preprocess_input,\n        \"decode_predictions\": keras.applications.vgg16.decode_predictions,\n    },\n    \"xception\": {\n        \"img_size\": (299, 299),\n        \"last_conv_layer\": \"block14_sepconv2_act\",\n        \"last_classifier_layers\": [\n            \"avg_pool\",\n            \"predictions\",\n        ],\n        \"model_builder\": keras.applications.xception.Xception,\n        \"preprocess_input\": keras.applications.xception.preprocess_input,\n        \"decode_predictions\": keras.applications.xception.decode_predictions,\n        \n    }\n}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Select the choice of your network and image, here I have taken boat_2.jpg. Also set how many top predictions we want to interospect","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"NETWORK = \"vgg16\"\nIMG_PATH = FILE_PATH + \"pier.jpg\"\nIMG_2_PATH = FILE_PATH + \"dock.jpg\"\nIMG_SIZE = properties[NETWORK][\"img_size\"]\nLAST_CONV_LAYER = properties[NETWORK][\"last_conv_layer\"]\nCLASSIFIER_LAYER_NAMES = properties[NETWORK][\"last_classifier_layers\"]\n\nTOP_N = 8","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Pick the corresponding model, image preprocessor and prediction decoder","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"model_builder = properties[NETWORK][\"model_builder\"]\npreprocess_input = properties[NETWORK][\"preprocess_input\"]\ndecode_predictions = properties[NETWORK][\"decode_predictions\"]","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":false},"cell_type":"code","source":"display(Image(IMG_PATH))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Preprocessing","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"def get_img_array(img_path, size):\n    img = keras.preprocessing.image.load_img(img_path, target_size=size)\n    array = keras.preprocessing.image.img_to_array(img)\n    array = np.expand_dims(array, axis=0)\n    return array\n\ndef load_imagenet_classes(filepath=FILE_PATH + \"imagenet_1000_idx.js\"):\n    \n    with open(filepath, 'r') as file:\n        class_dict = json.loads(file.read())\n    dict_by_name = {class_dict[key].split(\",\")[0]: int(key) for key in class_dict}\n    return dict_by_name, class_dict\n\nDICT_BY_NAME, CLASS_DICT = load_imagenet_classes()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Interospect the Predictions of the Input Image  \n- Preprocess the Image, image to array conversion\n- Get the Keras Model\n- Predict the Image\n- Decode the predictions ","execution_count":null},{"metadata":{"_kg_hide-output":true,"trusted":false},"cell_type":"code","source":"def get_predictions(image_path, image_size, top_n):\n    img_array = get_img_array(image_path, size=image_size)\n    img_array = preprocess_input(img_array)\n    model = model_builder(weights=\"imagenet\")\n    preds = model.predict(img_array)\n    preds_n = decode_predictions(preds, top=top_n)[0]\n    return preds_n\n\npreds_n = get_predictions(IMG_PATH, IMG_SIZE, TOP_N)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":false},"cell_type":"code","source":"def print_predictions(predictions):\n    print(\"Predictions\")\n    for index in np.arange(len(predictions)):\n        print(f'Id: {DICT_BY_NAME[predictions[index][1]]} Probability: {predictions[index][2]:4f} Class Name: {predictions[index][1].capitalize()}')\nprint_predictions(preds_n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Make GradCAM Heatmap  \nFollowing method works in 2 modes  \n1. Calculate GradCAM heatmaps for top N predictions, provided top_n > 0\n2. Provide Imagenet index of the classes of interest and get gradient heatmap for the input image.\n\n### STEPS\n1. Create a model that maps the input image to the activations of the last convolution layer - Get last conv layer's output dimensions\n2. Create another model, that maps from last convolution layer to the final class predictions - This is the classifier model that calculated the gradient\n3. If top N predictions are to be interospected, Get their Imagenet indices else assign the indices given\n4. Create an array to store the heatmaps\n5. Iteratively calculate heatmaps for all classes of interest using GradientTape\n6. Watch the last convolution output during the prediction process to calculate the gradients\n7. Compute the activations of last conv layer and make the tape to watch\n8. Get the class predictions and the class channel using the class index\n9. Using tape, Get the gradient for the predicted class wrt the output feature map of last conv layer\n10. Calculate the mean intensity of the gradient over its feature map channel\n11. Multiply each channel in feature map array by weight importance of the channel\n12. The channel-wise mean of the resulting feature map is our heatmap of class activation\n13. Normalize the heatmap between [0, 1] for ease of visualization","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"def get_top_predicted_indices(predictions, top_n):\n    return np.argsort(-predictions).squeeze()[:top_n]\n\ndef make_gradcam_heatmap(\n    img_array, model, \n    last_conv_layer_name, \n    classifier_layer_names,\n    top_n,\n    class_indices\n):\n    #1. Create a model that maps the input image to the activations of the last convolution layer - Get last conv layer's output dimensions\n    last_conv_layer = model.get_layer(last_conv_layer_name)\n    last_conv_layer_model = keras.Model(model.inputs, last_conv_layer.output)\n    \n    #2. Create another model, that maps from last convolution layer to the final class predictions - This is the classifier model that calculated the gradient\n    classifier_input = keras.Input(shape=last_conv_layer.output.shape[1:])\n    x = classifier_input\n    for layer_name in classifier_layer_names:\n        x = model.get_layer(layer_name)(x)\n    classifier_model = keras.Model(classifier_input, x)\n    \n    #3. If top N predictions are to be interospected, Get their Imagenet indices else assign the indices given\n    if(top_n > 0):\n        last_conv_layer_output = last_conv_layer_model(img_array)\n        preds = classifier_model(last_conv_layer_output)\n        class_indices = get_top_predicted_indices(preds, top_n)\n    else:\n        top_n = len(class_indices)\n    \n    #4. Create an array to store the heatmaps\n    heatmaps = []\n    #5. Iteratively calculate heatmaps for all classes of interest using GradientTape\n    for index in np.arange(top_n):\n    \n        #6. Watch the last convolution output during the prediction process to calculate the gradients\n        #7. Compute the activations of last conv layer and make the tape to watch\n        with tf.GradientTape() as tape:\n            # Compute activations of the last conv layer and make the tape watch it\n            last_conv_layer_output = last_conv_layer_model(img_array)\n            tape.watch(last_conv_layer_output)\n\n            #8. Get the class predictions and the class channel using the class index\n            preds = classifier_model(last_conv_layer_output)\n            class_channel = preds[:, class_indices[index]]\n            \n        #9. Using tape, Get the gradient for the predicted class wrt the output feature map of last conv layer    \n        grads = tape.gradient(\n            class_channel,\n            last_conv_layer_output\n        )\n        \n        #10. Calculate the mean intensity of the gradient over its feature map channel\n        pooled_grads = tf.reduce_mean(grads, axis=(0, 1, 2))    \n        last_conv_layer_output = last_conv_layer_output.numpy()[0]\n        pooled_grads = pooled_grads.numpy()\n        \n        #11. Multiply each channel in feature map array by weight importance of the channel\n        for i in range(pooled_grads.shape[-1]):\n            last_conv_layer_output[:, :, i] *= pooled_grads[i]\n\n        #12. The channel-wise mean of the resulting feature map is our heatmap of class activation\n        heatmap = np.mean(last_conv_layer_output, axis=-1)\n\n        #13. Normalize the heatmap between [0, 1] for ease of visualization\n        heatmap = np.maximum(heatmap, 0) / np.max(heatmap)\n\n        heatmaps.append({\n            \"class_id\": class_indices[index],\n            \"heatmap\": heatmap\n        })\n\n    return heatmaps","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Exploration of Top N predictions","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"# Calculate Heatmaps for TOP_N Predictions\nheatmaps = make_gradcam_heatmap(\n    get_img_array(IMG_PATH, IMG_SIZE), \n    model_builder(weights=\"imagenet\"), \n    LAST_CONV_LAYER, \n    CLASSIFIER_LAYER_NAMES, \n    TOP_N, \n    None\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def superimpose_heatmap(image_path, heatmap):\n    img = keras.preprocessing.image.load_img(image_path)\n    img = keras.preprocessing.image.img_to_array(img)\n    \n    # We rescale heatmap to a range 0-255\n    heatmap = np.uint8(255 * heatmap)\n    heatmap = cv2.applyColorMap(heatmap, cv2.COLORMAP_JET)\n    heatmap = keras.preprocessing.image.array_to_img(heatmap)\n    heatmap = heatmap.resize((img.shape[1], img.shape[0]))\n    \n    heatmap = keras.preprocessing.image.img_to_array(heatmap)\n    superimposed_img = cv2.addWeighted(heatmap, 0.4, img, 0.6, 0)\n    superimposed_img = np.uint8(superimposed_img)\n    \n    return superimposed_img","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":false},"cell_type":"code","source":"\ndef display_superimposed_heatmaps(heatmaps, image_path, image_id):\n    n = len(heatmaps)\n    n_rows = (n // 3) + 1 if n % 3 > 0 else n // 3\n    plt.rcParams['axes.grid'] = False\n    plt.rcParams['xtick.labelsize'] = False\n    plt.rcParams['ytick.labelsize'] = False\n    plt.rcParams['xtick.top'] = False\n    plt.rcParams['xtick.bottom'] = False\n    plt.rcParams['ytick.left'] = False\n    plt.rcParams['ytick.right'] = False\n    plt.rcParams['figure.figsize'] = [30, 15]\n    for index in np.arange(n):\n        heatmap = heatmaps[index][\"heatmap\"]\n        class_id = heatmaps[index][\"class_id\"]\n        class_name = CLASS_DICT[str(class_id)].split(\",\")[0].capitalize()\n        superimposed_image = superimpose_heatmap(image_path, heatmap)\n        plt.subplot(n_rows, 3, index+1)\n        plt.title(f\"{class_id}, {class_name}\", fontsize= 30)\n        plt.imshow(superimposed_image)\n        \n    plt.show()\ndisplay_superimposed_heatmaps(heatmaps, IMG_PATH, 1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## GradCAM for Classes of Interest  \n- For a predefined set of 6 classes we shall see the GradCAM efficacy\n- An image is carefully selected to view these predictions","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":false},"cell_type":"code","source":"\ndisplay(Image(IMG_2_PATH))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"#Classes of Interest\nclass_names = ('dock', 'pier', 'suspension_bridge', 'gondola', 'breakwater', 'dam')\nclass_indices = np.array([536, 718, 839, 576, 460, 525])\n\nclasses = [(class_indices[index], value) for index, value in enumerate(class_names)]\nclasses","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Calculate Heatmaps for TOP_N Predictions\nheatmaps = make_gradcam_heatmap(\n    get_img_array(IMG_2_PATH, IMG_SIZE), \n    model_builder(weights=\"imagenet\"), \n    LAST_CONV_LAYER, \n    CLASSIFIER_LAYER_NAMES, \n    0, \n    class_indices\n)\ndisplay_superimposed_heatmaps(heatmaps, IMG_2_PATH, 2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Reference\nArxiv Paper Link: [https://arxiv.org/abs/1610.02391](https://arxiv.org/abs/1610.02391)  \n<cite>\n@INPROCEEDINGS{8237336,  author={R. R. {Selvaraju} and M. {Cogswell} and A. {Das} and R. {Vedantam} and D. {Parikh} and D. {Batra}},  booktitle={2017 IEEE International Conference on Computer Vision (ICCV)},   title={Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization},   year={2017},  volume={},  number={},  pages={618-626},}\n</cite>","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}