{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Blindness Detection with Grad-CAM\n\nLet me introduce one visualization technique. This method is useful to understand which part of the image contributes to the final classification decision of the ConvNet. If there is a mistake in the classification, it helps to debug the decision process of ConvNet. It can also be used to locate specific objects in the image.\n\nThis kind of technique is commonly referred to as class activation map (CAM) visualization. Creates a heatmap of class activation for the input image. A class activation heatmap is a 2D point grid computed for every position in the input image for a particular output class. It tells you how important each location is to the class."},{"metadata":{},"cell_type":"markdown","source":"## Import packages"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"import os\n\nimport numpy as np\nimport pandas as pd\nimport cv2\n\nfrom PIL import Image\nfrom matplotlib import pyplot as plt\n\nfrom keras.applications.inception_resnet_v2 import preprocess_input, decode_predictions\nfrom keras.preprocessing.image import load_img, img_to_array\nfrom keras.models import Model, load_model\nfrom keras import backend as K","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Check the dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"DATA_PATH = '../input/aptos2019-blindness-detection'\nTRAIN_IMG_PATH = os.path.join(DATA_PATH, 'train_images')\nTEST_IMG_PATH = os.path.join(DATA_PATH, 'test_images')\nTRAIN_LABEL_PATH = os.path.join(DATA_PATH, 'train.csv')\nTEST_LABEL_PATH = os.path.join(DATA_PATH, 'test.csv')\n\ntrain_df = pd.read_csv(TRAIN_LABEL_PATH)\ntest_df = pd.read_csv(TEST_LABEL_PATH)\n\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load pre-trained model\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"model_path = '../input/aptos-2019-pretrained-models/'\nweight_file = 'weights-InceptionResNetV2.hdf5'\nmodel = load_model(os.path.join(model_path, weight_file))\n# model.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Visualization"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(3, 5, figsize=(20,10))\n\nimage_size = (299, 299)\nstart_index = 0\nnum_output = 5\n\nfor idx in range(num_output):\n    index = idx\n    index += start_index\n    img_path = os.path.join(TRAIN_IMG_PATH, train_df['id_code'][index]+'.png')\n    \n    # ==================================\n    #   1. Test images visualization\n    # ==================================\n    img = load_img(img_path, target_size=image_size)\n    img = np.expand_dims(img, axis=0)\n    pred_img = preprocess_input(img)\n    pred = model.predict(pred_img)\n    ax[0][idx].imshow(img[0])\n    ax[0][idx].set_title('ID: {}, Predict: {}'.format(train_df['id_code'][index], np.argmax(pred)))\n    \n    # ==============================\n    #   2. Heatmap visualization \n    # ==============================\n    # Item of prediction vector\n    pred_output = model.output[:, np.argmax(pred)]\n    \n    # Feature map of 'conv_7b_ac' layer, which is the last convolution layer\n    last_conv_layer = model.get_layer('conv_7b_ac')\n    \n    # Gradient of class for feature map output of 'conv_7b_ac'\n    grads = K.gradients(pred_output, last_conv_layer.output)[0]\n    \n    # Feature map vector with gradient average value per channel\n    pooled_grads = K.mean(grads, axis=(0, 1, 2))\n    \n    # Given a test image, get the feature map output of the previously defined 'pooled_grads' and 'conv_7b_ac'\n    iterate = K.function([model.input], [pooled_grads, last_conv_layer.output[0]])\n    \n    # Put a test image and get two numpy arrays\n    pooled_grads_value, conv_layer_output_value = iterate([pred_img])\n    \n    # Multiply the importance of a channel for a class by the channels in a feature map array\n    for i in range(int(pooled_grads.shape[0])):\n        conv_layer_output_value[:, :, i] *= pooled_grads_value[i]\n        \n    # The averaged value along the channel axis in the created feature map is the heatmap of the class activation\n    heatmap = np.mean(conv_layer_output_value, axis=-1)\n    \n    # Normalize the heatmap between 0 and 1 for visualization\n    heatmap = np.maximum(heatmap, 0)\n    heatmap /= np.max(heatmap)\n    ax[1][idx].imshow(heatmap)\n    \n    # =======================\n    #   3. Apply Grad-CAM\n    # =======================\n    ori_img = load_img(img_path, target_size=image_size)\n    \n    heatmap = cv2.resize(heatmap, image_size)\n    heatmap = np.uint8(255 * heatmap)\n    heatmap = cv2.applyColorMap(heatmap, cv2.COLORMAP_JET)\n\n    superimposed_img = heatmap * 0.5 + ori_img\n    cv2.imwrite('./grad_cam_result{}.jpg'.format(idx), superimposed_img)\n    grad_img = cv2.imread('./grad_cam_result{}.jpg'.format(idx))\n    \n    ax[2][idx].imshow(grad_img)\n    \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"! rm -rf *.jpg","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.6"}},"nbformat":4,"nbformat_minor":1}