{"cells":[{"metadata":{},"cell_type":"markdown","source":"# 0. Spotting Blindness -- Real or Spurious?\n\nAbove is the old title I used, in this new version I also show how to analyze robustness of our model using the great **albumentation**\n\n![Albumentation meets Grad-CAM](https://i.ibb.co/H4MJWVz/gradcam-album.png)\n\nTherefore, I decide to change the title to more appropriately reflect the techniques I used here. Please see Section 5 below for the updated material.\n\n\n# 1. Introduction : This eye is in danger, I estimate severity level 4\n\nDo you want to understand, when the model saying the above statement, how does it know? Does the model look at the same bloody or cotton wool spots like us? Below are what CNN actually see. Does this make sense? \n\nLook at the picture below, in the first case, it seems that the model works great! It is able to identify important spots in the eye. In the second case, however, even though the model estimate the severity to be level 3, it almost entirely misses the big wool spots in the middle. It might infer that our model still doesn't grasp an important concept of 'hard exudates' well enough. (ref. https://www.eyeops.com/)"},{"metadata":{},"cell_type":"markdown","source":"![grad-cam](https://i.ibb.co/6FM6VCC/gradcam-resized.png)\n\n![ref https://www.eyeops.com/](https://sa1s3optim.patientpop.com/assets/images/provider/photos/1947516.jpeg)\n\nThe technique I applied to visualize two cases above is called  \"Grad-CAM\" ([Gradient-weighted Class Activation Mapping](http://gradcam.cloudcv.org/) ; please see the link and reference therein for original materials). In this kernel, I will illustrate how to use it to get more insights from your model. \n\nI prefer to write in Keras and so I choose to visualize the [public Keras model of @xhlulu](https://www.kaggle.com/xhlulu/aptos-2019-densenet-keras-starter) who achieve the highest LB score at the moment I write this kernel. (Good job @xhlulu!). For pytorch lovers, please refer to [this kernel of @daisukelab](https://www.kaggle.com/daisukelab/verifying-cnn-models-with-cam-and-etc-fast-ai) where he applied similar techniques in the recent Freesound 2019 competition.\n\nThe Keras version of Grad-CAM is adapted from [this article](http://www.hackevolve.com/where-cnn-is-looking-grad-cam/), which in turns adapted from F.Chollet's book."},{"metadata":{},"cell_type":"markdown","source":"## 1.1 Brief Introduction on Grad-CAM\n\n![](http://gradcam.cloudcv.org/static/images/network.png)"},{"metadata":{},"cell_type":"markdown","source":"The idea of Grad-CAM visualization may be intuitively and non-formally summarized like this :\n\n**Objective** Emphasize pixel regions (spatial information) which make the model make a decision on the final predicted class (here, diabetic retinophaty severity level). We visualize these regions using **heatmap** (as shown in the above figure).\n\n**Method Intuition** \n* We bellieve that most important spatial information come from the 3D-tensor of the *last convolutional layer* (just before `GlobalPooling layer`), which is the nearest spatial information flowing to the last FC layer.\n\n* For each channel of this 3D-tensor, each activated pixel region represent important features (e.g. blood vessel / scab / cotton wool) of the input image. Note that some features are important to determine class 0 (perfectly fine blood vessel), some features are important to determine class 4 (big cotton wools). Normally, we expect each channel to capture different set of features\n\n* To emphasize features which finally affected the final prediction, we calculate the **gradient of the final predicted class with respect to each feature.** If that feature is important to this class, it should have high gradient (i.e. increase the value of this feature, the prediction confidence increases)\n\n* Therefore, we multiply the activated values of this 3D-tensor and gradients together, to obtain the visualized heatmap for each channel. Note that we have multi-channels, and each channel usually have multi-features.\n\n* Finally, we combine heatmaps of all channels using simple average, and remove negative value (the `ReLu` step in the above picture) to obtain the final heatmap\n\nFor formal introduction please refer to the Author's paper. For now let us proceed to the programming part."},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"import json\nimport math\nimport os\n\nimport cv2\nfrom PIL import Image\nimport numpy as np\nfrom keras import layers\nfrom keras.applications import DenseNet121\nfrom keras.callbacks import Callback, ModelCheckpoint\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.models import Sequential\nfrom keras.optimizers import Adam\nimport matplotlib.pyplot as plt\nimport pandas as pd\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import cohen_kappa_score, accuracy_score\nimport scipy\nfrom tqdm import tqdm\n\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2. Prepare tools of the original kernel\n\nIn order to visualize the model output using Grad-CAM, we do not need to retrain model. We can just use the already-trained weights from @xhlulu original kernel directly. Therefore, in fact, we will also not need to preprocess the train/test offline here, we will just preprocess them on-the-fly. Just note that the original kernel use 224x224 image, and use the preprocess_image function below.\n\nNevertheless, to ensure that we load model weights correctly, I will preprocess the test data and ensure that we obtain exact same predictions as original kernel."},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"train_df = pd.read_csv('../input/aptos2019-blindness-detection/train.csv')\ntest_df = pd.read_csv('../input/aptos2019-blindness-detection/test.csv')\nprint(train_df.shape)\nprint(test_df.shape)\ntest_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"def get_pad_width(im, new_shape, is_rgb=True):\n    pad_diff = new_shape - im.shape[0], new_shape - im.shape[1]\n    t, b = math.floor(pad_diff[0]/2), math.ceil(pad_diff[0]/2)\n    l, r = math.floor(pad_diff[1]/2), math.ceil(pad_diff[1]/2)\n    if is_rgb:\n        pad_width = ((t,b), (l,r), (0, 0))\n    else:\n        pad_width = ((t,b), (l,r))\n    return pad_width\n\ndef preprocess_image(image_path, desired_size=224):\n    im = Image.open(image_path)\n    im = im.resize((desired_size, )*2, resample=Image.LANCZOS)\n#     im = im.resize((desired_size, )*2)\n    \n    return im","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"N = test_df.shape[0]\nx_test = np.empty((N, 224, 224, 3), dtype=np.uint8)\n\nfor i, image_id in enumerate(tqdm(test_df['id_code'])):\n    x_test[i, :, :, :] = preprocess_image(\n        f'../input/aptos2019-blindness-detection/test_images/{image_id}.png'\n    )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Displaying some original test images\n\nJust to have an idea, let us visualize first 10 eyes in the test set. Just quickly notice that only 2/10 eyes here look fine. Below we shall also define Ben's preprocessing function as it will be easier for us human to see abnormal spots, and it is cleaner when we combine the eye picture with heatmap later."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# model.summary()\ndef load_image_ben_orig(path,resize=True,crop=False,norm255=True,keras=False):\n    image = cv2.imread(path)\n    \n#     if crop:\n#         image = crop_image(image)\n    \n    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n        \n#     if resize:\n#         image = cv2.resize(image,(SIZE,SIZE))\n        \n    image=cv2.addWeighted( image,4, cv2.GaussianBlur( image , (0,0) ,  10) ,-4 ,128)\n#     image=cv2.addWeighted( image,4, cv2.medianBlur( image , 10) ,-4 ,128)\n    \n    # NOTE plt.imshow can accept both int (0-255) or float (0-1), but deep net requires (0-1)\n    if norm255:\n        return image/255\n    elif keras:\n        #see https://github.com/keras-team/keras-applications/blob/master/keras_applications/imagenet_utils.py for mode\n        #see https://github.com/keras-team/keras-applications/blob/master/keras_applications/xception.py for inception,xception mode\n        #the use of tf based preprocessing (- and / by 127 respectively) will results in [-1,1] so it will not visualize correctly (directly)\n        image = np.expand_dims(image, axis=0)\n        return preprocess_input(image)[0]\n    else:\n        return image.astype(np.int16)\n    \n    return image\n\ndef transform_image_ben(img,resize=True,crop=False,norm255=True,keras=False):  \n    image=cv2.addWeighted( img,4, cv2.GaussianBlur( img , (0,0) ,  10) ,-4 ,128)\n    \n    # NOTE plt.imshow can accept both int (0-255) or float (0-1), but deep net requires (0-1)\n    if norm255:\n        return image/255\n    elif keras:\n        image = np.expand_dims(image, axis=0)\n        return preprocess_input(image)[0]\n    else:\n        return image.astype(np.int16)\n    \n    return image","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"def display_samples(df, columns=5, rows=2, Ben=True):\n    fig=plt.figure(figsize=(5*columns, 4*rows))\n\n    for i in range(columns*rows):\n        image_path = df.loc[i,'id_code']\n#         image_id = df.loc[i,'diagnosis']\n        path = f'../input/aptos2019-blindness-detection/test_images/{image_path}.png'\n        if Ben:\n            img = load_image_ben_orig(path)\n        else:\n            img = cv2.imread(path)\n            img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n        \n        fig.add_subplot(rows, columns, i+1)\n#         plt.title(image_id)\n        plt.imshow(img)\n    \n    plt.tight_layout()\n\ndisplay_samples(test_df, Ben=False)\ndisplay_samples(test_df, Ben=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. Construct model. Some hacks to get gradients.\n\nConceptually, we can just load pretrained model and calculate the wanted gradient and that's enough to have heatmap! However, technically speaking, the original kernel use Keras' `Sequential method` to construct a fine-tuned DenseNet model instead of Keras' `Functional method`. Unfortunately, by using `Sequential method` we cannot access the last convolutional layer directly. Therefore, we cannot calulate activations and gradients, so we need some hack.\n\nThe hack I use below is to use `Sequential method` to construct a model using shared layers and then apply pretrained weights. After that, I construct another model using `Functional method` but using the same shared layers. Since all layers are shared, the two models are exactly the same, having the same weights."},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras import layers\nfrom keras.models import Model\nimport keras.backend as K","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"First, let us define the `DenseNet` backbone."},{"metadata":{"trusted":true},"cell_type":"code","source":"K.clear_session()\ndensenet = DenseNet121(\n    weights=None,\n    include_top=False,\n    input_shape=(None,None,3)\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, we define 3 shared head layers, exactly the same types as used in original kernel. Then, construct it using `Sequential()` module also same process as the original kernel. You can see from `model.summary()` below that by using `Sequential()` module, the layer details of the backbone is hidden and we cannot use it directly. Therefore, we cannot obtain gradients here."},{"metadata":{"trusted":true},"cell_type":"code","source":"GAP_layer = layers.GlobalAveragePooling2D()\ndrop_layer = layers.Dropout(0.5)\ndense_layer = layers.Dense(5, activation='sigmoid', name='final_output')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_model_sequential():\n    model = Sequential()\n    model.add(densenet)\n    model.add(GAP_layer)\n    model.add(drop_layer)\n    model.add(dense_layer)\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"modelA = build_model_sequential()\nmodelA.load_weights('../input/aptos-data/dense_xhlulu_731.h5')\n\nmodelA.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Below, we construct another model using exactly the same (shared) layers. When pretrained weights are loaded into the first model, the second model also get the same weights (since all layers are shared)"},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_model_functional():\n    base_model = densenet\n    \n    x = GAP_layer(base_model.layers[-1].output)\n    x = drop_layer(x)\n    final_output = dense_layer(x)\n    model = Model(base_model.layers[0].input, final_output)\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now by using functional module, we can access to all layers in the backbone which is evident when executing `model.summary()`. Since the output is too long, it is hidden, but you can press the output button to see all layers."},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"model = build_model_functional() # with pretrained weights, and layers we want\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Therefore, we can access the last convolutional layer here. Note that we may use either `conv5_block16_concat` or `relu` which is the rectified and batch-normalized version of `conv5_block16_concat`."},{"metadata":{},"cell_type":"markdown","source":"## 3.1 Just to make sure that we have the correct weights\n\nNote that this section purpose is to make sure that we already loaded the correct weights. Since it is proven by the LB score in the previous version, in this version I just comment out everything\n"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# y_test = model.predict(x_test) > 0.5\n# y_test = y_test.astype(int).sum(axis=1) - 1\n\n# test_df['diagnosis'] = y_test\n# test_df.to_csv('submission.csv',index=False)\n# y_test.shape, x_test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# import seaborn as sns\n# import cv2\n\n# SIZE=224\n# def create_pred_hist(pred_level_y,title='NoTitle'):\n#     results = pd.DataFrame({'diagnosis':pred_level_y})\n\n#     f, ax = plt.subplots(figsize=(7, 4))\n#     ax = sns.countplot(x=\"diagnosis\", data=results, palette=\"GnBu_d\")\n#     sns.despine()\n#     plt.title(title)\n#     plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# create_pred_hist(y_test,title='predicted level distribution in test set')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4. Real or Spurious Features?\n\nNow we finally come to the main section of this kernel. It's time to investigate the model performance. First let us define a heatmap calculation function. As said in Introduction, codes are adapted from [this article](http://www.hackevolve.com/where-cnn-is-looking-grad-cam/), which in turns adapted from F.Chollet's book.\n\nThis function will recieve 4 arguments as inputs. (1) the image to make a prediction/visualization, remember to insert the correct preprocessed version here (2) the model (3) a layer to get gradients and (4) an auxiliary image just to combine with heatmap and visualize the final result; I use Ben's preprocessed image here since it eliminate lightning conditions in the pictures, and so easy for us to visualize the final result."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def gen_heatmap_img(img, model0, layer_name='last_conv_layer',viz_img=None,orig_img=None):\n    preds_raw = model0.predict(img[np.newaxis])\n    preds = preds_raw > 0.5 # use the same threshold as @xhlulu original kernel\n    class_idx = (preds.astype(int).sum(axis=1) - 1)[0]\n#     print(class_idx, class_idx.shape)\n    class_output_tensor = model0.output[:, class_idx]\n    \n    viz_layer = model0.get_layer(layer_name)\n    grads = K.gradients(\n                        class_output_tensor ,\n                        viz_layer.output\n                        )[0] # gradients of viz_layer wrt output_tensor of predicted class\n    \n    pooled_grads=K.mean(grads,axis=(0,1,2))\n    iterate=K.function([model0.input],[pooled_grads, viz_layer.output[0]])\n    \n    pooled_grad_value, viz_layer_out_value = iterate([img[np.newaxis]])\n    \n    for i in range(pooled_grad_value.shape[0]):\n        viz_layer_out_value[:,:,i] *= pooled_grad_value[i]\n    \n    heatmap = np.mean(viz_layer_out_value, axis=-1)\n    heatmap = np.maximum(heatmap,0)\n    heatmap /= np.max(heatmap)\n\n    viz_img=cv2.resize(viz_img,(img.shape[1],img.shape[0]))\n    heatmap=cv2.resize(heatmap,(viz_img.shape[1],viz_img.shape[0]))\n    \n    heatmap_color = cv2.applyColorMap(np.uint8(heatmap*255), cv2.COLORMAP_SPRING)/255\n    heated_img = heatmap_color*0.5 + viz_img*0.5\n    \n    print('raw output from model : ')\n    print_pred(preds_raw)\n    \n    if orig_img is None:\n        show_Nimages([img,viz_img,heatmap_color,heated_img])\n    else:\n        show_Nimages([orig_img,img,viz_img,heatmap_color,heated_img])\n    \n    plt.show()\n    return heated_img","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here are simple tools to easily show images and prediction"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def show_image(image,figsize=None,title=None):\n    \n    if figsize is not None:\n        fig = plt.figure(figsize=figsize)\n#     else: # crash!!\n#         fig = plt.figure()\n        \n    if image.ndim == 2:\n        plt.imshow(image,cmap='gray')\n    else:\n        plt.imshow(image)\n        \n    if title is not None:\n        plt.title(title)\n\ndef show_Nimages(imgs,scale=1):\n\n    N=len(imgs)\n    fig = plt.figure(figsize=(25/scale, 16/scale))\n    for i, img in enumerate(imgs):\n        ax = fig.add_subplot(1, N, i + 1, xticks=[], yticks=[])\n        show_image(img)\n        \ndef print_pred(array_of_classes):\n    xx = array_of_classes\n    s1,s2 = xx.shape\n    for i in range(s1):\n        for j in range(s2):\n            print('%.3f ' % xx[i,j],end='')\n        print('')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"First let us test the first 10 test examples. For each test example, I show original input, Ben's preprocessed input, heatmap and combined heatmap respectively."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"NUM_SAMP=10\nSEED=77\nlayer_name = 'relu' #'conv5_block16_concat'\nfor i, (idx, row) in enumerate(test_df[:NUM_SAMP].iterrows()):\n    path=f\"../input/aptos2019-blindness-detection/test_images/{row['id_code']}.png\"\n    ben_img = load_image_ben_orig(path)\n    input_img = np.empty((1,224, 224, 3), dtype=np.uint8)\n    input_img[0,:,:,:] = preprocess_image(path)\n        \n    print('test pic no.%d' % (i+1))\n    _ = gen_heatmap_img(input_img[0],\n                        model, layer_name=layer_name,viz_img=ben_img)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are many interesting observations here. To name a few, \n\n* The 1st, 4th, 5th, 6th predictions look great\n* The 2nd prediction misses the whole big spots in the middle\n* The 3rd and 7th predictions also miss important spots\n* In the 9th image, bloody spots are all over the places and the model is able to capture 4 regions, not all.\n* The 8th and 10th images look normal to me, but in the 10th, it looks like the model capture a **spuriou (false)** feature and identify severity level 1."},{"metadata":{},"cell_type":"markdown","source":"# 5. [updated] Robustness Test with Albumentation\n\nIn this new section, I show how to apply five (maybe sensible?) transforms of albumentation and test with our model in order to see that the model still give the same prediction as non-augmented or not. The sixth and ultimate augmentation is to combine all five transform altogether! You can see examples of resulted transforms one-by-one for each row below. Note that the first image is the original one."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"from albumentations import *\nimport time\n\nIMG_SIZE = (224,224)\n\n'''Use case from https://www.kaggle.com/alexanderliao/image-augmentation-demo-with-albumentation/'''\ndef albaugment(aug0, img):\n    return aug0(image=img)['image']\nidx=8\nimage1=x_test[idx]\n\n'''1. Rotate or Flip'''\naug1 = OneOf([\n    Rotate(p=0.99, limit=160, border_mode=0,value=0), # value=black\n    Flip(p=0.5)\n    ],p=1)\n\n'''2. Adjust Brightness or Contrast'''\naug2 = RandomBrightnessContrast(brightness_limit=0.45, contrast_limit=0.45,p=1)\nh_min=np.round(IMG_SIZE[1]*0.72).astype(int)\nh_max= np.round(IMG_SIZE[1]*0.9).astype(int)\nprint(h_min,h_max)\n\n'''3. Random Crop and then Resize'''\n#w2h_ratio = aspect ratio of cropping\naug3 = RandomSizedCrop((h_min, h_max),IMG_SIZE[1],IMG_SIZE[0], w2h_ratio=IMG_SIZE[0]/IMG_SIZE[1],p=1)\n\n'''4. CutOut Augmentation'''\nmax_hole_size = int(IMG_SIZE[1]/10)\naug4 = Cutout(p=1,max_h_size=max_hole_size,max_w_size=max_hole_size,num_holes=8 )#default num_holes=8\n\n'''5. SunFlare Augmentation'''\naug5 = RandomSunFlare(src_radius=max_hole_size,\n                      num_flare_circles_lower=10,\n                      num_flare_circles_upper=20,\n                      p=1)#default flare_roi=(0,0,1,0.5),\n\n'''6. Ultimate Augmentation -- combine everything'''\nfinal_aug = Compose([\n    aug1,aug2,aug3,aug4,aug5\n],p=1)\n\n\nimg1 = albaugment(aug1,image1)\nimg2 = albaugment(aug1,image1)\nprint('Rotate or Flip')\nshow_Nimages([image1,img1,img2],scale=2)\n# time.sleep(1)\n\nimg1 = albaugment(aug2,image1)\nimg2 = albaugment(aug2,image1)\nimg3 = albaugment(aug2,image1)\nprint('Brightness or Contrast')\nshow_Nimages([img3,img1,img2],scale=2)\n# time.sleep(1)\n\nimg1 = albaugment(aug3,image1)\nimg2 = albaugment(aug3,image1)\nimg3 = albaugment(aug3,image1)\nprint('Rotate and Resize')\nshow_Nimages([img3,img1,img2],scale=2)\nprint(img1.shape,img2.shape)\n# time.sleep(1)\n\nimg1 = albaugment(aug4,image1)\nimg2 = albaugment(aug4,image1)\nimg3 = albaugment(aug4,image1)\nprint('CutOut')\nshow_Nimages([img3,img1,img2],scale=2)\n# time.sleep(1)\n\nimg1 = albaugment(aug5,image1)\nimg2 = albaugment(aug5,image1)\nimg3 = albaugment(aug5,image1)\nprint('Sun Flare')\nshow_Nimages([img3,img1,img2],scale=2)\n# time.sleep(1)\n\nimg1 = albaugment(final_aug,image1)\nimg2 = albaugment(final_aug,image1)\nimg3 = albaugment(final_aug,image1)\nprint('All above combined')\nshow_Nimages([img3,img1,img2],scale=2)\nprint(img1.shape,img2.shape)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now let us see how our model reacts with each type of augmentation! Note that for this particular test image the model predict non-augmentation as level 3 (score `[0.998 1.000 0.999 0.953 0.068]`)\n\nSince the augmentation is random, you will see different results from when I wrote this kernel. In my experiment, the model is pretty robust as it predict almost the same level everytime, except the ultimate (everything) augmentation where it sometimes confusingly predicts as level 4. Detected features are quite consistent as well.\n\nThe key benefit in my opinion is to use all these intuitions to adjust your augmentation scheme. Make the system more robust, and know the features it should know.\n\nIn fact, we can have fun test a lot more examples but I will leave the rest to you at the moment."},{"metadata":{"trusted":true},"cell_type":"code","source":"aug_list = [aug5, aug2, aug3, aug4, aug1, final_aug]\naug_name = ['SunFlare', 'brightness or contrast', 'crop and resized', 'CutOut', 'rotate or flip', 'Everything Combined']\n\nidx=8\nlayer_name = 'relu' #'conv5_block16_concat'\nfor i in range(len(aug_list)):\n    path=f\"../input/aptos2019-blindness-detection/test_images/{test_df.iloc[idx]['id_code']}.png\"\n    input_img = np.empty((1,224, 224, 3), dtype=np.uint8)\n    input_img[0,:,:,:] = preprocess_image(path)\n    aug_img = albaugment(aug_list[i],input_img[0,:,:,:])\n    ben_img = transform_image_ben(aug_img)\n    \n    print('test pic no.%d -- augmentation: %s' % (i+1, aug_name[i]))\n    _ = gen_heatmap_img(aug_img,\n                        model, layer_name=layer_name,viz_img=ben_img,orig_img=input_img[0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are many more possible creative uses of this heatmap visualization which I will update in the future.\n\n* Visualize augmented images and see if our model is robust enough (predict the same), or our augmentation makes sense or not (it preserve important information?) **[DONE.]**\n\n* In the case of overfitting traning data, visualize training data to see some spurious features. Then design effective augmentations to eliminate that spurious features\n\n* Visualize for each level from 0 to 4, to get an idea what features our model use to determine each severity level\n\nThat's it for now!! Hope this kernel be helpful somehow!\n\n-- "},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.6"}},"nbformat":4,"nbformat_minor":1}