{"cells":[{"metadata":{},"cell_type":"markdown","source":"深度学习的可解释化工具Grad-CAM相信大家都不陌生啦~本文介绍该工具的使用，找了kaggle上较好的notebook翻译后分享给大家，该notebook使用keras实现。\n原文地址：https://www.kaggle.com/ratthachat/aptos-augmentation-visualize-diabetic-retinopathy/notebook\n\n因为是可视化工具，那肯定要有问题有模型，再使用这个工具来看模型的识别情况，辅助我们决策。\n因为本文是kaggle竞赛里的一个kernel，所以大家需要了解一些背景情况：\n* 竞赛题目：通过眼球图片来识别糖尿病视网膜病变的严重程度。分为5个严重等级：0，1，2，3，4。0代表正常眼球，4代表非常严重。\n* 文中会出现一些hard exudates等专业名词，指的是医学上对眼球情况的描述。比如这里翻译为：坚硬的渗出物。还有wood spots，大家知道这是对眼球的描述就好了，请原谅我蹩脚的英语ヾ(≧O≦)〃嗷~（我是硬翻译的，有更好地翻译的同学请留言）。\n* 关于Grad-CAM的论文。Grad-CAM大家可以去找找资料，这里主要讲keras的实现方式。最早使用CAM来可视化，CAM提取最后一层卷积池化层，然后将后面的层改为GAP层，重新训练权重。从而得到GAP层对于每个类别的权重。由于GAP层还没有丢失空间信息，所以我们可以依据权重来对特征图上采样。上采样的结果和原来的图片融合在一起，就是CAM了，感兴趣的同学可以看CAM的实现：https://github.com/philipperemy/tensorflow-class-activation-mapping\n* Grad-CAM可以不需要再训练一遍，而且对没有GAP层的网络很包容。具体大家往下看哦~\n"},{"metadata":{},"cell_type":"markdown","source":"\n\n# 1. 介绍 ：这些图片是非常严重的，模型有很好地预测到吗？\n\n你想要了解，模型到底关注了图片哪些内容？和眼球图片致病区域是同一块地方吗？下面的图片显示了CNN真正关注了哪些区域，它是否真的理解了眼球致病的原因。\n\n看下面的图片：\n在第一个案例中，模型看起来运行不错。它可以指出眼球中的缺陷。\n在第二个案例中，模型完全没有指出在眼球中间的wool spots，虽然它将图片评估为等级3.这个等级说明了我们模型还没有抓取hard exudates特征。\n参考网址： https://www.eyeops.com/"},{"metadata":{},"cell_type":"markdown","source":"![grad-cam](https://i.ibb.co/6FM6VCC/gradcam-resized.png)\n\n![ref https://www.eyeops.com/](https://sa1s3optim.patientpop.com/assets/images/provider/photos/1947516.jpeg)\n\n本次使用的可视化技术名字为“Grad-CAM”，你可以点击这些链接来找到该技术的原始材料： ([Gradient-weighted Class Activation Mapping](http://gradcam.cloudcv.org/) ;在这里，我会陈述怎么通过该技术得到更多关于你模型的见解。\n\n我倾向于使用keras，所以我选择可视化他的模型，链接是：[public Keras model of @xhlulu](https://www.kaggle.com/xhlulu/aptos-2019-densenet-keras-starter)，他的模型在我写kernel的时候分数最高了。（你好棒 @xhlulu!）。对于喜欢pytorch的人，你可以看 [this kernel of @daisukelab](https://www.kaggle.com/daisukelab/verifying-cnn-models-with-cam-and-etc-fast-ai)，这篇文章也用到了这个技术。\n\nkeras版本引用了[this article](http://www.hackevolve.com/where-cnn-is-looking-grad-cam/)，该文章引用了F.Chollet的书"},{"metadata":{},"cell_type":"markdown","source":"## 1.1 Grad-CAM简介\n\n![](http://gradcam.cloudcv.org/static/images/network.png)"},{"metadata":{},"cell_type":"markdown","source":"Grad-CAM可视化可以被直观地、非正式化地解释为：\n**目标**着重强调模型做决策的像素区域（空间信息），使用热力图来可视化这些区域，如上图所示。\n**简单描述**\n* 我们相信最重要的空间信息来自于最后一层卷积层（在全局池化层前一层），也就是离全连接层最近的带有空间信息的层。\n* 对于该层的每一通道，那些激活的图片区域。也就是说，我们希望该层每个通道可以抓取不同类别的特征。比如，通道0代表了class 0的激活区域，通道1代表了class 1 的激活区域。多分类问题还有多个通道。\n* 我们通过计算结果和该层特征的梯度来强调图片中的重要信息。\n* 因此，我们通过将计算出来的梯度和该层的tensor相乘来获取热力图。\n* 最后我们将热力图的通道取平均，通过relu移除负值来获取最终热力图。\n更多信息可以阅读作者的paper。\n"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"import json\nimport math\nimport os\n\nimport cv2\nfrom PIL import Image\nimport numpy as np\nfrom keras import layers\nfrom keras.applications import DenseNet121\nfrom keras.callbacks import Callback, ModelCheckpoint\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.models import Sequential\nfrom keras.optimizers import Adam\nimport matplotlib.pyplot as plt\nimport pandas as pd\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import cohen_kappa_score, accuracy_score\nimport scipy\nfrom tqdm import tqdm\n\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2. 准备原始kernel的工具\n\n为了使用Grad-CAM来可视化模型，我们不需要再重新训练模型，我们只要直接用已经训练好的来自@xhlulu的原始kernel的模型权重。因此，在实际上，我们没有必要重新预处理训练/测试数据，我们会在线预处理。注意了，原始kernel使用了224x224的图片尺寸，我们使用图片预处理功能，如下面的代码所示。\n\n尽管如此，为了保证我们加载了正确的模型权重，我会预处理测试数据，来确保我们和原始kernel一样预测正确。"},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"train_df = pd.read_csv('../input/aptos2019-blindness-detection/train.csv')\ntest_df = pd.read_csv('../input/aptos2019-blindness-detection/test.csv')\nprint(train_df.shape)\nprint(test_df.shape)\ntest_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"def get_pad_width(im, new_shape, is_rgb=True):\n    pad_diff = new_shape - im.shape[0], new_shape - im.shape[1]\n    t, b = math.floor(pad_diff[0]/2), math.ceil(pad_diff[0]/2)\n    l, r = math.floor(pad_diff[1]/2), math.ceil(pad_diff[1]/2)\n    if is_rgb:\n        pad_width = ((t,b), (l,r), (0, 0))\n    else:\n        pad_width = ((t,b), (l,r))\n    return pad_width\n\ndef preprocess_image(image_path, desired_size=224):\n    im = Image.open(image_path)\n    im = im.resize((desired_size, )*2, resample=Image.LANCZOS)\n#     im = im.resize((desired_size, )*2)\n    \n    return im","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"N = test_df.shape[0]\nx_test = np.empty((N, 224, 224, 3), dtype=np.uint8)\n\nfor i, image_id in enumerate(tqdm(test_df['id_code'])):\n    x_test[i, :, :, :] = preprocess_image(\n        f'../input/aptos2019-blindness-detection/test_images/{image_id}.png'\n    )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 显示一些测试图片\n\n我们来可视化测试集中的前10张眼球图片。快速查看后发现，只有2/10张眼球看起来正常。下面我们应该定义Ben的预处理函数。这个预处理函数会让我们更方便地看到不正常的眼球斑点，在待会儿我们使用热力图结合眼球图片来显示的时候也会更清晰。\n\n关于Ben的预处理函数：\n指的是2015年APTOS也举办过这样的比赛，有位大牛Ben，使用了这样的预处理方式：\nimage=cv2.addWeighted( image,4, cv2.GaussianBlur( image , (0,0) ,  10) ,-4 ,128)\n加了高斯模糊，使得图片减少了光照的影响。但是在2019年今年的比赛中，很多人加了这样的处理后，结果变差。可能是2015年的眼球图片和今年的有所差异。但是这种方式在kernel中很火哦~"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# model.summary()\ndef load_image_ben_orig(path,resize=True,crop=False,norm255=True,keras=False):\n    image = cv2.imread(path)\n    \n#     if crop:\n#         image = crop_image(image)\n    \n    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n        \n#     if resize:\n#         image = cv2.resize(image,(SIZE,SIZE))\n        \n    image=cv2.addWeighted( image,4, cv2.GaussianBlur( image , (0,0) ,  10) ,-4 ,128)\n#     image=cv2.addWeighted( image,4, cv2.medianBlur( image , 10) ,-4 ,128)\n    \n    # NOTE plt.imshow can accept both int (0-255) or float (0-1), but deep net requires (0-1)\n    if norm255:\n        return image/255\n    elif keras:\n        #see https://github.com/keras-team/keras-applications/blob/master/keras_applications/imagenet_utils.py for mode\n        #see https://github.com/keras-team/keras-applications/blob/master/keras_applications/xception.py for inception,xception mode\n        #the use of tf based preprocessing (- and / by 127 respectively) will results in [-1,1] so it will not visualize correctly (directly)\n        image = np.expand_dims(image, axis=0)\n        return preprocess_input(image)[0]\n    else:\n        return image.astype(np.int16)\n    \n    return image\n\ndef transform_image_ben(img,resize=True,crop=False,norm255=True,keras=False):  \n    image=cv2.addWeighted( img,4, cv2.GaussianBlur( img , (0,0) ,  10) ,-4 ,128)\n    \n    # NOTE plt.imshow can accept both int (0-255) or float (0-1), but deep net requires (0-1)\n    if norm255:\n        return image/255\n    elif keras:\n        image = np.expand_dims(image, axis=0)\n        return preprocess_input(image)[0]\n    else:\n        return image.astype(np.int16)\n    \n    return image","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"def display_samples(df, columns=5, rows=2, Ben=True):\n    fig=plt.figure(figsize=(5*columns, 4*rows))\n\n    for i in range(columns*rows):\n        image_path = df.loc[i,'id_code']\n#         image_id = df.loc[i,'diagnosis']\n        path = f'../input/aptos2019-blindness-detection/test_images/{image_path}.png'\n        if Ben:\n            img = load_image_ben_orig(path)\n        else:\n            img = cv2.imread(path)\n            img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n        \n        fig.add_subplot(rows, columns, i+1)\n#         plt.title(image_id)\n        plt.imshow(img)\n    \n    plt.tight_layout()\n\ndisplay_samples(test_df, Ben=False)\ndisplay_samples(test_df, Ben=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. 定义模型。获取梯度的一些技巧。\n\n概念上，我们可以只是加载预训练模型，计算梯度，给出热力图就可以了！但是，原始模型使用了“Sequential method”来构造一个微调的DenseNet模型，而不是“Functional method”。不幸的是，Sequential method 我们不能直接获取最后一层卷积层。因此，我们不能计算激活和梯度值。所以我们需要一个技巧。\n\n这个技巧使用了`Sequential method`，使用共享层来构造一个模型，然后应用预训练参数。然后，我用`Functional method`和同样的层构造了另一个共享所有层的模型。因为所有层都是共享的，这两个模型其实都是一样的，有一样的参数。\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras import layers\nfrom keras.models import Model\nimport keras.backend as K","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"首先，让我们定义 `DenseNet` 的主心骨."},{"metadata":{"trusted":true},"cell_type":"code","source":"K.clear_session()\ndensenet = DenseNet121(\n    weights=None,\n    include_top=False,\n    input_shape=(None,None,3)\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"其次，我们定义和最初的kernel一样的3个共享的头部层。使用`Sequential()`模型构造（和原始模型也是一样的）。从`model.summary()`的运行结果你可以看到模型的细节你是看不到的，我们不能直接获取卷积层的最后一层。因此，我们不能从这里获取梯度。"},{"metadata":{"trusted":true},"cell_type":"code","source":"GAP_layer = layers.GlobalAveragePooling2D()\ndrop_layer = layers.Dropout(0.5)\ndense_layer = layers.Dense(5, activation='sigmoid', name='final_output')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_model_sequential():\n    model = Sequential()\n    model.add(densenet)\n    model.add(GAP_layer)\n    model.add(drop_layer)\n    model.add(dense_layer)\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"modelA = build_model_sequential()\nmodelA.load_weights('../input/aptos-data/dense_xhlulu_731.h5')\n\nmodelA.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"接下来，我们用同样的（共享）层构造另外一个模型。当我们预训练第一个模型参数后，第二个模型获得了同样的参数，因为所有层都是共享的。"},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_model_functional():\n    base_model = densenet\n    \n    x = GAP_layer(base_model.layers[-1].output)\n    x = drop_layer(x)\n    final_output = dense_layer(x)\n    model = Model(base_model.layers[0].input, final_output)\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"通过使用functional模型，运行model.summary()后，我们可以获取模型的网络结构里的所有的层。因为输出的内容太长了，如果这些内容被隐藏了，你可以点击按钮展开查看更多层。"},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"model = build_model_functional() # with pretrained weights, and layers we want\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"这里，我们可以找到最后一个卷积层。注意了，我们可以使用`conv5_block16_concat`或者`relu`模块。"},{"metadata":{},"cell_type":"markdown","source":"## 3.1 确保我们拿到了正确的参数\n\n这个模块的目的是确保我们已经加载了正确的参数。虽然在以前的版本中，已经被证明是正确的。在这里我只是说明一下。\n"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# y_test = model.predict(x_test) > 0.5\n# y_test = y_test.astype(int).sum(axis=1) - 1\n\n# test_df['diagnosis'] = y_test\n# test_df.to_csv('submission.csv',index=False)\n# y_test.shape, x_test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# import seaborn as sns\n# import cv2\n\n# SIZE=224\n# def create_pred_hist(pred_level_y,title='NoTitle'):\n#     results = pd.DataFrame({'diagnosis':pred_level_y})\n\n#     f, ax = plt.subplots(figsize=(7, 4))\n#     ax = sns.countplot(x=\"diagnosis\", data=results, palette=\"GnBu_d\")\n#     sns.despine()\n#     plt.title(title)\n#     plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# create_pred_hist(y_test,title='predicted level distribution in test set')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4. 真的还是假的特征\n\n现在我们开始最主要的模块。我们来调查一下模型的性能。首先，让我们通过calculation函数定义一个热力图。就好像介绍中说得，代码引用了[这篇文章]\n(http://www.hackevolve.com/where-cnn-is-looking-grad-cam/),这篇文章引用了 F.Chollet 的书.\n\n这个函数接收了4个参数作为输入：\n(1) 待可视化的图片，记得要放入预处理过的版本。 (2) 模型 (3) 最后一层卷积层 (4) 一个用于结合热力图来展示的辅助图片; 我使用了Ben的预处理方式处理过的图片，因为这个处理方式消除了光照条件的影响。 所以特别适合最终展示。"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def gen_heatmap_img(img, model0, layer_name='last_conv_layer',viz_img=None,orig_img=None):\n    preds_raw = model0.predict(img[np.newaxis])\n    preds = preds_raw > 0.5 # use the same threshold as @xhlulu original kernel\n    class_idx = (preds.astype(int).sum(axis=1) - 1)[0]\n#     print(class_idx, class_idx.shape)\n    class_output_tensor = model0.output[:, class_idx]\n    \n    viz_layer = model0.get_layer(layer_name)\n    grads = K.gradients(\n                        class_output_tensor ,\n                        viz_layer.output\n                        )[0] # gradients of viz_layer wrt output_tensor of predicted class\n    \n    pooled_grads=K.mean(grads,axis=(0,1,2))\n    iterate=K.function([model0.input],[pooled_grads, viz_layer.output[0]])\n    \n    pooled_grad_value, viz_layer_out_value = iterate([img[np.newaxis]])\n    \n    for i in range(pooled_grad_value.shape[0]):\n        viz_layer_out_value[:,:,i] *= pooled_grad_value[i]\n    \n    heatmap = np.mean(viz_layer_out_value, axis=-1)\n    heatmap = np.maximum(heatmap,0)\n    heatmap /= np.max(heatmap)\n\n    viz_img=cv2.resize(viz_img,(img.shape[1],img.shape[0]))\n    heatmap=cv2.resize(heatmap,(viz_img.shape[1],viz_img.shape[0]))\n    \n    heatmap_color = cv2.applyColorMap(np.uint8(heatmap*255), cv2.COLORMAP_SPRING)/255\n    heated_img = heatmap_color*0.5 + viz_img*0.5\n    \n    print('raw output from model : ')\n    print_pred(preds_raw)\n    \n    if orig_img is None:\n        show_Nimages([img,viz_img,heatmap_color,heated_img])\n    else:\n        show_Nimages([orig_img,img,viz_img,heatmap_color,heated_img])\n    \n    plt.show()\n    return heated_img","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"这里是展示图片和预测结果的工具函数"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def show_image(image,figsize=None,title=None):\n    \n    if figsize is not None:\n        fig = plt.figure(figsize=figsize)\n#     else: # crash!!\n#         fig = plt.figure()\n        \n    if image.ndim == 2:\n        plt.imshow(image,cmap='gray')\n    else:\n        plt.imshow(image)\n        \n    if title is not None:\n        plt.title(title)\n\ndef show_Nimages(imgs,scale=1):\n\n    N=len(imgs)\n    fig = plt.figure(figsize=(25/scale, 16/scale))\n    for i, img in enumerate(imgs):\n        ax = fig.add_subplot(1, N, i + 1, xticks=[], yticks=[])\n        show_image(img)\n        \ndef print_pred(array_of_classes):\n    xx = array_of_classes\n    s1,s2 = xx.shape\n    for i in range(s1):\n        for j in range(s2):\n            print('%.3f ' % xx[i,j],end='')\n        print('')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"首先，让我们测试前10个测试数据里的样本。对于每个数据，我展示了原始图片、经过Ben的方式处理过的图片、热力图、和热力图结合的图片。"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"NUM_SAMP=10\nSEED=77\nlayer_name = 'relu' #'conv5_block16_concat'\nfor i, (idx, row) in enumerate(test_df[:NUM_SAMP].iterrows()):\n    path=f\"../input/aptos2019-blindness-detection/test_images/{row['id_code']}.png\"\n    ben_img = load_image_ben_orig(path)\n    input_img = np.empty((1,224, 224, 3), dtype=np.uint8)\n    input_img[0,:,:,:] = preprocess_image(path)\n        \n    print('test pic no.%d' % (i+1))\n    _ = gen_heatmap_img(input_img[0],\n                        model, layer_name=layer_name,viz_img=ben_img)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"我们得到了很多有趣的见解。\n\n* 第1、4、5、6预测结果看起来不错\n* 第2个预测没有指出中间大大的斑点\n* 第3和第7预测结果同样错过了重要的斑点\n* 第9张图片，到处都是血斑点，模型只指出4处，并没有指出全部的斑点\n* 第8、10图片看起来比较正常，但是模型似乎获取了伪特征，将其定义为严重程度为1。\n"},{"metadata":{},"cell_type":"markdown","source":"# 5. 使用Albumentation进行鲁棒性测试\n\n在这个模块，为了看模型是否给出正确的预测，我会向大家展示5个albumentation的图片变换方式，然后使用我们的模型进行测试。第6和最终的图片增强是将这5种变换结合起来！你可以看到这些样本在下面一行行排列。注意了，第一张图片是原始图片。"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"from albumentations import *\nimport time\n\nIMG_SIZE = (224,224)\n\n'''Use case from https://www.kaggle.com/alexanderliao/image-augmentation-demo-with-albumentation/'''\ndef albaugment(aug0, img):\n    return aug0(image=img)['image']\nidx=8\nimage1=x_test[idx]\n\n'''1. Rotate or Flip'''\naug1 = OneOf([\n    Rotate(p=0.99, limit=160, border_mode=0,value=0), # value=black\n    Flip(p=0.5)\n    ],p=1)\n\n'''2. Adjust Brightness or Contrast'''\naug2 = RandomBrightnessContrast(brightness_limit=0.45, contrast_limit=0.45,p=1)\nh_min=np.round(IMG_SIZE[1]*0.72).astype(int)\nh_max= np.round(IMG_SIZE[1]*0.9).astype(int)\nprint(h_min,h_max)\n\n'''3. Random Crop and then Resize'''\n#w2h_ratio = aspect ratio of cropping\naug3 = RandomSizedCrop((h_min, h_max),IMG_SIZE[1],IMG_SIZE[0], w2h_ratio=IMG_SIZE[0]/IMG_SIZE[1],p=1)\n\n'''4. CutOut Augmentation'''\nmax_hole_size = int(IMG_SIZE[1]/10)\naug4 = Cutout(p=1,max_h_size=max_hole_size,max_w_size=max_hole_size,num_holes=8 )#default num_holes=8\n\n'''5. SunFlare Augmentation'''\naug5 = RandomSunFlare(src_radius=max_hole_size,\n                      num_flare_circles_lower=10,\n                      num_flare_circles_upper=20,\n                      p=1)#default flare_roi=(0,0,1,0.5),\n\n'''6. Ultimate Augmentation -- combine everything'''\nfinal_aug = Compose([\n    aug1,aug2,aug3,aug4,aug5\n],p=1)\n\n\nimg1 = albaugment(aug1,image1)\nimg2 = albaugment(aug1,image1)\nprint('Rotate or Flip')\nshow_Nimages([image1,img1,img2],scale=2)\n# time.sleep(1)\n\nimg1 = albaugment(aug2,image1)\nimg2 = albaugment(aug2,image1)\nimg3 = albaugment(aug2,image1)\nprint('Brightness or Contrast')\nshow_Nimages([img3,img1,img2],scale=2)\n# time.sleep(1)\n\nimg1 = albaugment(aug3,image1)\nimg2 = albaugment(aug3,image1)\nimg3 = albaugment(aug3,image1)\nprint('Rotate and Resize')\nshow_Nimages([img3,img1,img2],scale=2)\nprint(img1.shape,img2.shape)\n# time.sleep(1)\n\nimg1 = albaugment(aug4,image1)\nimg2 = albaugment(aug4,image1)\nimg3 = albaugment(aug4,image1)\nprint('CutOut')\nshow_Nimages([img3,img1,img2],scale=2)\n# time.sleep(1)\n\nimg1 = albaugment(aug5,image1)\nimg2 = albaugment(aug5,image1)\nimg3 = albaugment(aug5,image1)\nprint('Sun Flare')\nshow_Nimages([img3,img1,img2],scale=2)\n# time.sleep(1)\n\nimg1 = albaugment(final_aug,image1)\nimg2 = albaugment(final_aug,image1)\nimg3 = albaugment(final_aug,image1)\nprint('All above combined')\nshow_Nimages([img3,img1,img2],scale=2)\nprint(img1.shape,img2.shape)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"现在让我们一起看我们的模型是怎样识别不同的图片增强的。注意了，这个测试图片模型预测了没有图片增强，并预测为严重程度为3 (score `[0.998 1.000 0.999 0.953 0.068]`)\n\n因为这个增强是随机的，你可以看到不同的结果。在我的实验中，这个模型是相当鲁棒的，因为它基本上预测出一样的严重程度，除了将所有增强结合起来的那个有时候会预测为等级4，检测到的特征也都是比较正确的。\n\n主要是你可以使用所有的直觉来调整你的图片增强的架构，使得模型更加鲁棒。\n\n我们本来可以测试更多的图片，但是我会将剩下的都留给你去做。"},{"metadata":{"trusted":true},"cell_type":"code","source":"aug_list = [aug5, aug2, aug3, aug4, aug1, final_aug]\naug_name = ['SunFlare', 'brightness or contrast', 'crop and resized', 'CutOut', 'rotate or flip', 'Everything Combined']\n\nidx=8\nlayer_name = 'relu' #'conv5_block16_concat'\nfor i in range(len(aug_list)):\n    path=f\"../input/aptos2019-blindness-detection/test_images/{test_df.iloc[idx]['id_code']}.png\"\n    input_img = np.empty((1,224, 224, 3), dtype=np.uint8)\n    input_img[0,:,:,:] = preprocess_image(path)\n    aug_img = albaugment(aug_list[i],input_img[0,:,:,:])\n    ben_img = transform_image_ben(aug_img)\n    \n    print('test pic no.%d -- augmentation: %s' % (i+1, aug_name[i]))\n    _ = gen_heatmap_img(aug_img,\n                        model, layer_name=layer_name,viz_img=ben_img,orig_img=input_img[0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"热力图可视化有很多新颖的应用：\n\n* 可视化增强后的图片，来看你的模型是否足够鲁棒（预测结果一样），或者模型是否真的理解了图片（是否有保留重要信息）\n\n* 如果模型存在过拟合，可视化训练集来看一些伪特征。设计一些高效的图片增强方式来消除伪特征。\n\n* 可视化0~4严重级别的图片，去找决定每个严重等级的特征。\n\n\n结束了，希望这个notebook可以帮助到你哦~\n\n-- "}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.6"}},"nbformat":4,"nbformat_minor":1}