{"cells":[{"metadata":{},"cell_type":"markdown","source":"In my previous [kernel](https://www.kaggle.com/meaninglesslives/alaska2-cnn-multiclass-classifier), I trained a multiclass CNN based classifier. I was curious to understand what the CNN is learning, since detecting if hidden message is present or not seems to be impossible visually. So, in this kernel, I try to do a Grad-Cam visualization to understand which parts the CNN is focussing on. Inspired by Remi's [earlier work](https://www.kaggle.com/remicogranne/inspect-impact-of-steganography-on-dct-coefs), I also plot the RGB difference and DCT difference for easier comparison. \n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Observation 1\n### You can see from gradcam visualization that the CNN seems to put very high emphasis on border areas. If you visualize all the results then you will find the hidden messages seems to occur on the border areas more often (particularly for UERD class). \n\n## Observation 2\n### The RGD difference seem to be spread out over all the channels for most images. For few pixels, some channel may be dominant but they seem to be on average preferred equally i.e. proportion of rgb is similar.\n\n## Observation 3\n### The hidden messages seem to almost always at the edges of objects. If the object has smooth texture, the hidden message is glaringly absent in the smooth part. \n\n## Observation 4\n### If you checkout the class wise accuracy we can notice two main things:\n       - As the image quality increases, classification accuracy decreases.\n       - Classification accuracy for JUNIWARD is very bad.\nI believe addressing these two limitations can drastically improve the lb score.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"! git clone https://github.com/dwgoon/jpegio\n# Once downloaded install the package\n!pip install jpegio/.\nimport jpegio as jio\n!rm -rf jpegio\n!pip install -q efficientnet_pytorch\nfrom efficientnet_pytorch import EfficientNet","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import cv2\nimport numpy as np\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.utils.data import Dataset\nfrom albumentations import ToFloat, Compose\nfrom albumentations.pytorch import ToTensor\nfrom efficientnet_pytorch import EfficientNet\nimport matplotlib.pyplot as plt\nimport os\nimport pandas as pd\nimport random\nimport jpegio as jio\nfrom tqdm.notebook import tqdm\n\n\n# # Seed everything\nseed = 42\nprint(f'setting everything to seed {seed}')\nrandom.seed(seed)\nos.environ['PYTHONHASHSEED'] = str(seed)\nnp.random.seed(seed)\ntorch.manual_seed(seed)\ntorch.cuda.manual_seed(seed)\ntorch.backends.cudnn.deterministic = True","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Class Categories","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"data_dir = '../input/alaska2-image-steganalysis'\nfolder_names = ['JMiPOD/', 'JUNIWARD/', 'UERD/']\nclass_names = ['Cover', 'JMiPOD_75', 'JMiPOD_90', 'JMiPOD_95', \n               'JUNIWARD_75', 'JUNIWARD_90', 'JUNIWARD_95',\n                'UERD_75', 'UERD_90', 'UERD_95']\nclass_labels = { name: i for i, name in enumerate(class_names) }\nprint(class_labels)\nprint('75, 90, 95 refers to JPEG image quality')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Load pretrained model weights","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class Net(nn.Module):\n    def __init__(self, num_classes):\n        super().__init__()\n        self.model = EfficientNet.from_pretrained('efficientnet-b0')\n        # 1280 is the number of neurons in last layer. is diff for diff. architecture\n        self.dense_output = nn.Linear(1280, num_classes)\n\n    def forward(self, x):\n        feat = self.model.extract_features(x)\n        feat = F.avg_pool2d(feat, feat.size()[2:]).reshape(-1, 1280)\n        return self.dense_output(feat)\n    \ndevice = 'cuda'\n# device = 'cpu'\nmodel = Net(num_classes=len(class_labels)).to(device)\nfn = '../input/alaska2trainvalsplit/val_loss_6.08_auc_0.875.pth'\n# pretrained model in my pc. now i will train on all images for 2 epochs\nmodel.load_state_dict(torch.load(fn, map_location=device))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Load the validation set.\nWe will do our visualization on validation set for an unbiased analysis.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class Alaska2Dataset(Dataset):\n\n    def __init__(self, df, augmentations=None):\n\n        self.data = df\n        self.augment = augmentations\n\n    def __len__(self):\n        return len(self.data)\n\n    def __getitem__(self, idx):\n        fn, label = self.data.loc[idx]\n        im = cv2.imread(fn)[:, :, ::-1]\n        if self.augment:\n            # Apply transformations\n            im = self.augment(image=im)\n        return im, label, fn.split('/')[-1]\n    \nval_df = pd.read_csv('../input/alaska2trainvalsplit/alaska2_val_df.csv')\nval_df.sample(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"batch_size = 16\nnum_workers = 4\nAUGMENTATIONS_TEST = Compose([\n    #     Resize(img_size, img_size, p=1), # does nothing if it's alread 512.\n    ToFloat(max_value=255),\n    ToTensor()\n], p=1)\nvalid_dataset = Alaska2Dataset(val_df.sample(1000).reset_index(drop=True),\n                               augmentations=AUGMENTATIONS_TEST)\n\nvalid_loader = torch.utils.data.DataLoader(valid_dataset,\n                                           batch_size=batch_size,\n                                           num_workers=num_workers,\n                                           shuffle=False)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# https://www.kaggle.com/anokas/weighted-auc-metric-updated\nfrom sklearn import metrics\ndef alaska_weighted_auc(y_true, y_valid):\n    tpr_thresholds = [0.0, 0.4, 1.0]\n    weights = [2,   1]\n\n    fpr, tpr, thresholds = metrics.roc_curve(y_true, y_valid, pos_label=1)\n\n    # size of subsets\n    areas = np.array(tpr_thresholds[1:]) - np.array(tpr_thresholds[:-1])\n\n    # The total area is normalized by the sum of weights such that the final weighted AUC is between 0 and 1.\n    normalization = np.dot(areas, weights)\n\n    competition_metric = 0\n    for idx, weight in enumerate(weights):\n        y_min = tpr_thresholds[idx]\n        y_max = tpr_thresholds[idx + 1]\n        mask = (y_min < tpr) & (tpr < y_max)\n        # pdb.set_trace()\n\n        x_padding = np.linspace(fpr[mask][-1], 1, 100)\n\n        x = np.concatenate([fpr[mask], x_padding])\n        y = np.concatenate([tpr[mask], [y_max] * len(x_padding)])\n        y = y - y_min  # normalize such that curve starts at y=0\n        score = metrics.auc(x, y)\n        submetric = score * weight\n        best_subscore = (y_max - y_min) * weight\n        competition_metric += submetric\n\n    return competition_metric / normalization","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tk1 = tqdm(valid_loader, total=int(len(valid_loader)))\nmodel.eval()\nrunning_loss = 0\ny, preds = [], []\nval_loss = []\ncriterion = torch.nn.CrossEntropyLoss()\nwith torch.no_grad():\n    for (im, labels, _) in tk1:\n        inputs = im[\"image\"].to(device, dtype=torch.float)\n        labels = labels.to(device, dtype=torch.long)\n        outputs = model(inputs)\n        loss = criterion(outputs, labels)\n        y.extend(labels.cpu().numpy().astype(int))\n        preds.extend(F.softmax(outputs, 1).cpu().numpy())\n        running_loss += loss.item()\n        tk1.set_postfix(loss=(loss.item()))\n\n    epoch_loss = running_loss / (len(valid_loader)/batch_size)\n    val_loss.append(epoch_loss)\n    preds = np.array(preds)\n    # convert multiclass labels to binary class\n    y = np.array(y)\n    labels = preds.argmax(1)\n\n    for class_label in np.unique(y):\n        idx = y == class_label\n        acc = (labels[idx] == y[idx]).astype(np.float).mean()*100\n        print('accuracy for class', class_names[class_label], 'is', acc)\n\n    acc = (labels == y).mean()*100\n    new_preds = np.zeros((len(preds),))\n    temp = preds[labels != 0, 1:]\n    new_preds[labels != 0] = temp.sum(1)\n    new_preds[labels == 0] = 1-preds[labels == 0, 0]\n    y = np.array(y)\n    y_all = y.copy()\n    y[y != 0] = 1\n    auc_score = alaska_weighted_auc(y, new_preds)\n    print(\n        f'Val Loss: {epoch_loss:.3}, Weighted AUC:{auc_score:.3}, Acc: {acc:.3}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Code for plotting confusion matrix","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import itertools\nimport sklearn\n# http://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html\ndef plot_confusion_matrix(cm, classes,\n                          normalize=False,\n                          title='Confusion matrix',\n                          cmap=plt.cm.Blues):\n    \"\"\"\n    This function prints and plots the confusion matrix.\n    Normalization can be applied by setting `normalize=True`.\n    \"\"\"\n    if normalize:\n        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]*100\n        print(\"Normalized confusion matrix\")\n    else:\n        print('Confusion matrix, without normalization')\n\n    print(cm)\n\n    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n    plt.title(title)\n    plt.colorbar()\n    tick_marks = np.linspace(-0.5, len(classes)-0.5, len(classes))\n    plt.xticks(tick_marks, classes, rotation=15)\n    plt.yticks(tick_marks, classes)\n\n    thresh = cm.max() / 2.\n    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n        plt.text(j+0.05, i+0.05, f'{cm[i, j]:.3}%',\n                 horizontalalignment=\"center\",\n                 color=\"white\" if cm[i, j] > thresh else \"black\")\n\n    plt.ylabel('True label')\n    plt.xlabel('Predicted label')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Compute confusion matrix\ncnf_matrix = sklearn.metrics.confusion_matrix(y_all, preds.argmax(1))\n\n# Plot non-normalized confusion matrix\nplt.figure(figsize=(30,9))\nfoo = plot_confusion_matrix(cnf_matrix, classes=class_names, normalize=True,\n                      title='Confusion matrix')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Grad-Cam Visualization\nAdaptation of [Jacob Gill's](https://github.com/jacobgil/pytorch-grad-cam) for EfficientNet. Also added batch processing.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# code inspired from https://github.com/jacobgil/pytorch-grad-cam\nclass FeatureExtractor():\n    \"\"\" Class for extracting activations and \n    registering gradients from targetted intermediate layers \"\"\"\n\n    def __init__(self, model):\n        self.model = model\n        self.gradients = []\n\n    def save_gradient(self, grad):\n        self.gradients.append(grad)\n\n    def __call__(self, x):\n        outputs = []\n        self.gradients = []\n        x = self.model(x)\n        x.register_hook(self.save_gradient)\n        outputs += [x]\n        return outputs, x\n\n\nclass ModelOutputs():\n    \"\"\" Class for making a forward pass, and getting:\n    1. The network output.\n    2. Activations from intermeddiate targetted layers.\n    3. Gradients from intermeddiate targetted layers. \"\"\"\n\n    def __init__(self, model, feature_module, final_layer):\n        self.model = model\n        self.feature_module = feature_module\n        self.feature_extractor = FeatureExtractor(\n            self.feature_module)\n        self.final_layer = final_layer\n\n    def get_gradients(self):\n        return self.feature_extractor.gradients\n\n    def __call__(self, x):\n        target_activations = []\n        for name, module in self.model._modules.items():\n            if name == '_avg_pooling':\n                break\n            if module == self.feature_module:\n                target_activations, x = self.feature_extractor(x)\n            else:\n                if '_blocks' in name:\n                    for m in module:\n                        x = m(x)\n                elif name == '_bn0':\n                    x = module(x)\n                    x = self.model._modules['_swish'](x)\n                else:\n                    x = module(x)\n\n        x = self.model._modules['_swish'](x)\n        x = F.avg_pool2d(x, x.size()[2:]).reshape(-1, 1280)\n        x = self.final_layer(x)\n        return target_activations, x\n    \nclass GradCam:\n    def __init__(self, model, feature_module, final_layer, device=device):\n        self.model = model\n        self.feature_module = feature_module\n        self.model.eval()\n        self.device = device\n        self.extractor = ModelOutputs(\n            self.model, self.feature_module, final_layer)\n\n    def forward(self, input):\n        return self.model(input)\n\n    def __call__(self, input, index=None):\n        features, output = self.extractor(input)\n\n        if index == None:\n            index = np.argmax(output.cpu().data.numpy(), 1)\n            pred_classes = output.argmax(1)\n\n        # print('predicted class', class_names[index], 'probability:', F.softmax(\n        #     output, 1)[0, index].item())\n\n        # one_hot = np.zeros((index.shape[0], output.size()[-1]), dtype=np.float32)\n        # one_hot[0][index] = 1\n        # one_hot = torch.from_numpy(one_hot).requires_grad_(True)\n        one_hot = torch.zeros(\n            (index.shape[0], output.size()[-1]), dtype=torch.float).to(self.device)\n        one_hot.scatter_(1, output.argmax(\n            1, keepdim=True), 1).requires_grad_(True)\n        one_hot = torch.sum(one_hot * output)\n\n        self.feature_module.zero_grad()\n        self.model.zero_grad()\n        one_hot.backward(retain_graph=True)\n\n        grads_val = self.extractor.get_gradients()[-1].cpu().data.numpy()\n\n        target = features[-1]\n        target = target.cpu().data.numpy()\n\n        weights = np.mean(grads_val, axis=(2, 3))\n        cam = np.zeros(\n            (target.shape[0], target.shape[2], target.shape[3]), dtype=np.float32)\n\n        for img_num, w1 in enumerate(weights):\n            for filt_num, w2 in enumerate(w1):\n                cam[img_num] += w2 * target[img_num, filt_num, :, :]\n\n        cam = np.maximum(cam, 0)\n        cam = np.array(\n            [(cv2.resize(c, input.shape[2:]) - c.min())/c.max() for c in cam])\n        # cam = np.array([cv2.resize(c, input.shape[2:]) for c in cam])\n\n        # cam = cam - np.min(cam)\n        # cam = cam / np.max(cam)\n        return cam, pred_classes","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# For plotting Images\nPlots the original image, grad-cam image, rgb difference and DCT difference.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def show_cam_on_image(img, mask, fn, p, l):\n    dct_diff = cv2.applyColorMap(np.uint8(255 * mask), cv2.COLORMAP_VIRIDIS)\n    dct_diff = np.float32(dct_diff) / 255\n    cam = dct_diff + np.float32(img)\n    cam = cam / np.max(cam)\n    cam = np.uint8(255 * cam)\n    cv2.putText(cam, 'Actual:'+class_names[l.item()], (5, img.shape[0] - 5),\n                cv2.FONT_HERSHEY_PLAIN, 1.5, (0, 255, 0), thickness=2)\n    cv2.putText(cam, 'Pred:'+class_names[p.item()], (5, img.shape[0] - 30),\n                cv2.FONT_HERSHEY_PLAIN, 1.5, (0, 255, 0), thickness=2)\n\n    orig_im = cv2.imread(f'{data_dir}/Cover/{fn}')\n    cv2.putText(orig_im, 'Orig Im.', (5, orig_im.shape[0] - 5),\n                cv2.FONT_HERSHEY_PLAIN, 1.5, (0, 255, 0), thickness=2)\n    folder_name = class_names[l.item()].split('_')[0]\n    im = cv2.imread(f\"{data_dir}/{folder_name}/{fn}\")\n#     pdb.set_trace()\n    diff_rgb = im - orig_im\n    cv2.putText(diff_rgb, 'RGB Diff.', (5, diff_rgb.shape[0] - 5),\n                cv2.FONT_HERSHEY_PLAIN, 1.5, (0, 255, 0), thickness=2)\n\n    jio_im = jio.read(f\"{data_dir}/Cover/{fn}\")\n    im = jio.read(f\"{data_dir}/{folder_name}/{fn}\")\n    dct_diff = np.zeros_like(orig_im)\n\n    if folder_name != 'Cover':\n        diff_dct = np.array(im.coef_arrays) - np.array(jio_im.coef_arrays)\n        diff_dct = (diff_dct-diff_dct.min())/(diff_dct.max()-diff_dct.min())\n        diff_dct = np.uint8(diff_dct.transpose(1, 2, 0)*255)\n        # dct coeff diff is -1\n        idx_x, idx_y, _ = np.where(diff_dct == 0)\n        dct_diff[idx_x, idx_y, 2] = 255\n        # dct coeff diff is +1\n        idx_x, idx_y, _ = np.where(diff_dct == 255)\n        dct_diff[idx_x, idx_y, 1] = 255\n\n#     pdb.set_trace()\n    cv2.putText(dct_diff, 'DCT Diff.', (5, img.shape[0] - 5),\n                cv2.FONT_HERSHEY_PLAIN, 1.5, (0, 255, 0), thickness=2)\n    im = np.concatenate([orig_im, cam, diff_rgb, dct_diff], 1)\n#     cv2.imwrite(f\"{fn}\", im)  # uncomment if you want to write images\n    return im","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"AUGMENTATIONS_TEST = Compose([\n    ToFloat(max_value=255),\n    ToTensor()\n], p=1)\n\n\ngrad_cam = GradCam(model=model.model, feature_module=model.model._conv_head,\n                   final_layer=model.dense_output, device=device)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def process_batch(loader, device=device):\n    for (im, labels, all_fn) in loader:\n        inputs = im[\"image\"].to(device, dtype=torch.float).requires_grad_(True)\n        masks, pred_classes = grad_cam(inputs, None)\n        inputs = inputs.permute(0, 2, 3, 1).detach().cpu().numpy()\n#         plt.figure(figsize=(20, 10))\n        plt.figure(figsize=(len(inputs)*5, len(inputs)*5))\n        for i, (img, mask, fn, p, l) in enumerate(zip(inputs, masks, all_fn, pred_classes, labels)):\n            viz = show_cam_on_image(img, mask, fn, p, l)\n            plt.subplot(len(inputs), 1, i+1)\n            plt.imshow(viz)\n            plt.axis('off')\n        break # remove this to process all images\n    plt.subplots_adjust(hspace=0.02, wspace=0)\n#     plt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Cover Images","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"temp_df = val_df[val_df.Label==0].sample(100).reset_index(drop=True)\nvalid_dataset = Alaska2Dataset(temp_df, augmentations=AUGMENTATIONS_TEST)\n\n\n\nvalid_loader = torch.utils.data.DataLoader(valid_dataset,\n                                           batch_size=16,\n                                           num_workers=0,\n                                           shuffle=False)\n\nprocess_batch(valid_loader)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# JMiPOD","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"temp_df = val_df[val_df.Label.isin([1,2,3])].reset_index(drop=True)\nvalid_dataset = Alaska2Dataset(temp_df, augmentations=AUGMENTATIONS_TEST)\n\n\n\nvalid_loader = torch.utils.data.DataLoader(valid_dataset,\n                                           batch_size=16,\n                                           num_workers=0,\n                                           shuffle=False)\n\nprocess_batch(valid_loader, device=device)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# JUNIWARD","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"temp_df = val_df[val_df.Label.isin([4,5,6])].reset_index(drop=True)\nvalid_dataset = Alaska2Dataset(temp_df, augmentations=AUGMENTATIONS_TEST)\n\n\n\nvalid_loader = torch.utils.data.DataLoader(valid_dataset,\n                                           batch_size=16,\n                                           num_workers=0,\n                                           shuffle=False)\n\nprocess_batch(valid_loader, device=device)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# UERD","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"temp_df = val_df[val_df.Label.isin([7,8,9])].reset_index(drop=True)\nvalid_dataset = Alaska2Dataset(temp_df, augmentations=AUGMENTATIONS_TEST)\n\n\n\nvalid_loader = torch.utils.data.DataLoader(valid_dataset,\n                                           batch_size=16,\n                                           num_workers=0,\n                                           shuffle=False)\n\nprocess_batch(valid_loader, device=device)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}