{"cells":[{"metadata":{},"cell_type":"markdown","source":"In this kernel i build a CNN based multiclass classifier that given an input image predicts which category the image belongs to namely, (No Hidden Message, JMiPOD, JUNIWARD or UERD).\nThe main highlights of the kernel:\n- Approach the problem as multiclass classification instead of binary class.\n- Proper split for train and validation set and use competition metric [proposed here](https://www.kaggle.com/anokas/weighted-auc-metric-updated)  to get good estimate of model performance on public lb.\n- Illustrate the use of proper augmentation strategy to improve model performance.\n- Using imagenet pretrained EfficientNet B0 as model backbone for the classifier.\n- One Possible way to convert multiclass probabilities to binary.\n- Use test time augmentation to improve lb score.\n\nI have added some comments at each block which further clarifies some of the above mentioned points.\n\n## Updates:\n- v3:\n    New way of converting multiclass probabilities to binary (basically take sum of probabilities of JMiPOD, JUNIWARD and UERD).\n    \n    Trained for 5 more epochs.\n- v4:\n    Based on some of the [discussion](https://www.kaggle.com/c/alaska2-image-steganalysis/discussion/148919#835400), I decided to increase the number of classes. So, now i train a CNN classifier that separates images into one of the following categories [Normal, JMiPOD_75, JMiPOD_90, JMiPOD_95, JUNIWARD_75, JUNIWARD_90, JUNIWARD_95,UERD_75, UERD_90, UERD_95].\n- v5: \n    Train for more epochs.\n- v6: \n    Train for more epochs. Reduced learning rate every 2 epochs by 0.5.\n    Added CoarseDropout and Cutout. It seems to improve performance.\n\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Load Libraries","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"!pip install -q efficientnet_pytorch\nfrom efficientnet_pytorch import EfficientNet\nfrom albumentations.pytorch import ToTensor\nfrom albumentations import (\n    Compose, HorizontalFlip, CLAHE, HueSaturationValue,\n    RandomBrightness, RandomContrast, RandomGamma, OneOf, Resize,\n    ToFloat, ShiftScaleRotate, GridDistortion, RandomRotate90, Cutout,\n    RGBShift, RandomBrightness, RandomContrast, Blur, MotionBlur, MedianBlur, GaussNoise, CoarseDropout,\n    IAAAdditiveGaussianNoise, GaussNoise, OpticalDistortion, RandomSizedCrop, VerticalFlip\n)\nimport os\nimport torch\nimport pandas as pd\nimport numpy as np\nimport random\nimport torch.nn as nn\nimport matplotlib.pyplot as plt\nfrom glob import glob\nimport torchvision\nfrom torch.utils.data import Dataset\nimport time\nfrom tqdm.notebook import tqdm\n# from tqdm import tqdm\nfrom sklearn import metrics\nimport cv2\nimport gc\nimport torch.nn.functional as F","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Seed everything","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"seed = 42\nprint(f'setting everything to seed {seed}')\nrandom.seed(seed)\nos.environ['PYTHONHASHSEED'] = str(seed)\nnp.random.seed(seed)\ntorch.manual_seed(seed)\ntorch.cuda.manual_seed(seed)\ntorch.backends.cudnn.deterministic = True","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Create dataset for training and Validation","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### I think the way we perform split is important. Just performing random split mayn't make sense for two reasons:\n    - A random split isn't representative of the actual dataset (the class distribution may be different). So, the validation score mayn't be accurate.\n    \n    - Our aim is to train a classifier that is able to tell whether image has hidden message or not. So, we must ensure that during training we have the corresponding image\n    with hidden message. Without the counter example, the discriminatve features that the classifier learns during training mayn't be good.\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"data_dir = '../input/alaska2-image-steganalysis'\nfolder_names = ['JMiPOD/', 'JUNIWARD/', 'UERD/']\nclass_names = ['Normal', 'JMiPOD_75', 'JMiPOD_90', 'JMiPOD_95', \n               'JUNIWARD_75', 'JUNIWARD_90', 'JUNIWARD_95',\n                'UERD_75', 'UERD_90', 'UERD_95']\nclass_labels = { name: i for i, name in enumerate(class_names)}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv('../input/alaska2trainvalsplit/alaska2_train_df.csv')\nval_df = pd.read_csv('../input/alaska2trainvalsplit/alaska2_val_df.csv')\n\nprint(train_df.sample(10))\ntrain_df.Label.hist()\nplt.title('Distribution of Classes')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Pytorch Dataset","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class Alaska2Dataset(Dataset):\n\n    def __init__(self, df, augmentations=None):\n\n        self.data = df\n        self.augment = augmentations\n\n    def __len__(self):\n        return len(self.data)\n\n    def __getitem__(self, idx):\n        fn, label = self.data.loc[idx]\n        im = cv2.imread(fn)[:, :, ::-1]\n        if self.augment:\n            # Apply transformations\n            im = self.augment(image=im)\n        return im, label\n\n\nimg_size = 512\nAUGMENTATIONS_TRAIN = Compose([\n    VerticalFlip(p=0.5),\n    HorizontalFlip(p=0.5),\n    RandomRotate90(p=0.5),\n    OneOf([\n        Cutout(max_h_size=50,max_w_size=50),\n        CoarseDropout(max_height=50, max_width=50)], p=0.5),\n    ToFloat(max_value=255),\n    ToTensor()\n], p=1)\n\n\nAUGMENTATIONS_TEST = Compose([\n    ToFloat(max_value=255),\n    ToTensor()\n], p=1)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Images without any augmentation","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"temp_df = train_df.sample(64).reset_index(drop=True)\ntrain_dataset = Alaska2Dataset(temp_df, augmentations=AUGMENTATIONS_TEST)\nbatch_size = 64\nnum_workers = 0\n\ntemp_loader = torch.utils.data.DataLoader(train_dataset,\n                                          batch_size=batch_size,\n                                          num_workers=num_workers, shuffle=False)\n\n\nimages, labels = next(iter(temp_loader))\nimages = images['image'].permute(0, 2, 3, 1)\nmax_images = 64\ngrid_width = 16\ngrid_height = int(max_images / grid_width)\nfig, axs = plt.subplots(grid_height, grid_width,\n                        figsize=(grid_width+1, grid_height+2))\n\nfor i, (im, label) in enumerate(zip(images, labels)):\n    ax = axs[int(i / grid_width), i % grid_width]\n    ax.imshow(im.squeeze())\n    ax.set_title(str(label.item()))\n    ax.axis('off')\n\nplt.suptitle(\"0: Cover, 1: JMiPOD_75, 2: JMiPOD_90, 3: JMiPOD_95, 4: JUNIWARD_75, 5:JUNIWARD_90,\\n 6: JUNIWARD_95, 7:UERD_75, 8:UERD_90, 9:UERD_95\")\nplt.show()\ndel images\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Images after augmentation","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dataset = Alaska2Dataset(temp_df, augmentations=AUGMENTATIONS_TRAIN)\nbatch_size = 64\nnum_workers = 0\n\ntemp_loader = torch.utils.data.DataLoader(train_dataset,\n                                          batch_size=batch_size,\n                                          num_workers=num_workers, shuffle=False)\n\n\nimages, labels = next(iter(temp_loader))\nimages = images['image'].permute(0, 2, 3, 1)\nmax_images = 64\ngrid_width = 16\ngrid_height = int(max_images / grid_width)\nfig, axs = plt.subplots(grid_height, grid_width,\n                        figsize=(grid_width+1, grid_height+2))\n\nfor i, (im, label) in enumerate(zip(images, labels)):\n    ax = axs[int(i / grid_width), i % grid_width]\n    ax.imshow(im.squeeze())\n    ax.set_title(str(label.item()))\n    ax.axis('off')\n\nplt.suptitle(\"0: Cover, 1: JMiPOD_75, 2: JMiPOD_90, 3: JMiPOD_95, 4: JUNIWARD_75, 5:JUNIWARD_90,\\n 6: JUNIWARD_95, 7:UERD_75, 8:UERD_90, 9:UERD_95\")\nplt.show()\ndel images, temp_df\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# CNN Model for multiclass classification\n\nI use pretrained Efficient BO. You can try other backbones also. EB7 seems to work slightly better but takes a long time to train.\nSo, i am sticking with EB0.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class Net(nn.Module):\n    def __init__(self, num_classes):\n        super().__init__()\n        self.model = EfficientNet.from_pretrained('efficientnet-b0')\n        # 1280 is the number of neurons in last layer. is diff for diff. architecture\n        self.dense_output = nn.Linear(1280, num_classes)\n\n    def forward(self, x):\n        feat = self.model.extract_features(x)\n        feat = F.avg_pool2d(feat, feat.size()[2:]).reshape(-1, 1280)\n        return self.dense_output(feat)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"batch_size = 8\nnum_workers = 8\n\ntrain_dataset = Alaska2Dataset(train_df, augmentations=AUGMENTATIONS_TRAIN)\nvalid_dataset = Alaska2Dataset(val_df.sample(1000).reset_index(drop=True), augmentations=AUGMENTATIONS_TEST) #for faster validation sample\n\ntrain_loader = torch.utils.data.DataLoader(train_dataset,\n                                           batch_size=batch_size,\n                                           num_workers=num_workers,\n                                           shuffle=True)\n\nvalid_loader = torch.utils.data.DataLoader(valid_dataset,\n                                           batch_size=batch_size*2,\n                                           num_workers=num_workers,\n                                           shuffle=False)\n\ndevice = 'cuda'\nmodel = Net(num_classes=len(class_labels)).to(device)\n# pretrained model in my pc. now i will train on all images for 2 epochs\nmodel.load_state_dict(torch.load('../input/alaska2trainvalsplit/val_loss_6.08_auc_0.875.pth'))\noptimizer = torch.optim.AdamW(model.parameters(), lr=1e-6)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# https://www.kaggle.com/anokas/weighted-auc-metric-updated\n\ndef alaska_weighted_auc(y_true, y_valid):\n    tpr_thresholds = [0.0, 0.4, 1.0]\n    weights = [2,   1]\n\n    fpr, tpr, thresholds = metrics.roc_curve(y_true, y_valid, pos_label=1)\n\n    # size of subsets\n    areas = np.array(tpr_thresholds[1:]) - np.array(tpr_thresholds[:-1])\n\n    # The total area is normalized by the sum of weights such that the final weighted AUC is between 0 and 1.\n    normalization = np.dot(areas, weights)\n\n    competition_metric = 0\n    for idx, weight in enumerate(weights):\n        y_min = tpr_thresholds[idx]\n        y_max = tpr_thresholds[idx + 1]\n        mask = (y_min < tpr) & (tpr < y_max)\n        # pdb.set_trace()\n\n        x_padding = np.linspace(fpr[mask][-1], 1, 100)\n\n        x = np.concatenate([fpr[mask], x_padding])\n        y = np.concatenate([tpr[mask], [y_max] * len(x_padding)])\n        y = y - y_min  # normalize such that curve starts at y=0\n        score = metrics.auc(x, y)\n        submetric = score * weight\n        best_subscore = (y_max - y_min) * weight\n        competition_metric += submetric\n\n    return competition_metric / normalization","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"criterion = torch.nn.CrossEntropyLoss()\nnum_epochs = 2\ntrain_loss, val_loss = [], []\n\nfor epoch in range(num_epochs):\n    print('Epoch {}/{}'.format(epoch, num_epochs - 1))\n    print('-' * 10)\n    model.train()\n    running_loss = 0\n    tk0 = tqdm(train_loader, total=int(len(train_loader)))\n    for im, labels in tk0:\n        inputs = im[\"image\"].to(device, dtype=torch.float)\n        labels = labels.to(device, dtype=torch.long)\n        optimizer.zero_grad()\n        outputs = model(inputs)\n        loss = criterion(outputs, labels)\n        loss.backward()\n        optimizer.step()\n        running_loss += loss.item()\n        tk0.set_postfix(loss=(loss.item()))\n\n    epoch_loss = running_loss / (len(train_loader)/batch_size)\n    train_loss.append(epoch_loss)\n    print('Training Loss: {:.8f}'.format(epoch_loss))\n\n    tk1 = tqdm(valid_loader, total=int(len(valid_loader)))\n    model.eval()\n    running_loss = 0\n    y, preds = [], []\n    with torch.no_grad():\n        for (im, labels) in tk1:\n            inputs = im[\"image\"].to(device, dtype=torch.float)\n            labels = labels.to(device, dtype=torch.long)\n            outputs = model(inputs)\n            loss = criterion(outputs, labels)\n            y.extend(labels.cpu().numpy().astype(int))\n            preds.extend(F.softmax(outputs, 1).cpu().numpy())\n            running_loss += loss.item()\n            tk1.set_postfix(loss=(loss.item()))\n\n        epoch_loss = running_loss / (len(valid_loader)/batch_size)\n        val_loss.append(epoch_loss)\n        preds = np.array(preds)\n        # convert multiclass labels to binary class\n        y = np.array(y)\n        labels = preds.argmax(1)\n        for class_label in np.unique(y):\n            idx = y == class_label\n            acc = (labels[idx] == y[idx]).astype(np.float).mean()*100\n            print('accuracy for class', class_names[class_label], 'is', acc)\n        \n        acc = (labels == y).mean()*100\n        new_preds = np.zeros((len(preds),))\n        temp = preds[labels != 0, 1:]\n        new_preds[labels != 0] = temp.sum(1)\n        new_preds[labels == 0] = 1 - preds[labels == 0, 0]\n        y = np.array(y)\n        y[y != 0] = 1\n        auc_score = alaska_weighted_auc(y, new_preds)\n        print(\n            f'Val Loss: {epoch_loss:.3}, Weighted AUC:{auc_score:.3}, Acc: {acc:.3}')\n\n    torch.save(model.state_dict(),\n               f\"epoch_{epoch}_val_loss_{epoch_loss:.3}_auc_{auc_score:.3}.pth\")\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15,7))\nplt.plot(train_loss, c='r')\nplt.plot(val_loss, c='b')\nplt.legend(['train_loss', 'val_loss'])\nplt.title('Loss Plot')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Create Inference Dataset","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# # Inference\nclass Alaska2TestDataset(Dataset):\n\n    def __init__(self, df, augmentations=None):\n\n        self.data = df\n        self.augment = augmentations\n\n    def __len__(self):\n        return len(self.data)\n\n    def __getitem__(self, idx):\n        fn = self.data.loc[idx][0]\n        im = cv2.imread(fn)[:, :, ::-1]\n\n        if self.augment:\n            # Apply transformations\n            im = self.augment(image=im)\n\n        return im\n\n\ntest_filenames = sorted(glob(f\"{data_dir}/Test/*.jpg\"))\ntest_df = pd.DataFrame({'ImageFileName': list(\n    test_filenames)}, columns=['ImageFileName'])\n\nbatch_size = 16\nnum_workers = 4\ntest_dataset = Alaska2TestDataset(test_df, augmentations=AUGMENTATIONS_TEST)\ntest_loader = torch.utils.data.DataLoader(test_dataset,\n                                          batch_size=batch_size,\n                                          num_workers=num_workers,\n                                          shuffle=False,\n                                          drop_last=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Do inference","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"model.eval()\n\npreds = []\ntk0 = tqdm(test_loader)\nwith torch.no_grad():\n    for i, im in enumerate(tk0):\n        inputs = im[\"image\"].to(device)\n        # flip vertical\n        im = inputs.flip(2)\n        outputs = model(im)\n        # fliplr\n        im = inputs.flip(3)\n        outputs = (0.25*outputs + 0.25*model(im))\n        outputs = (outputs + 0.5*model(inputs))        \n        preds.extend(F.softmax(outputs, 1).cpu().numpy())\n\npreds = np.array(preds)\nlabels = preds.argmax(1)\nnew_preds = np.zeros((len(preds),))\nnew_preds[labels != 0] = preds[labels != 0, 1:].sum(1)\nnew_preds[labels == 0] = 1 - preds[labels == 0, 0]\n\ntest_df['Id'] = test_df['ImageFileName'].apply(lambda x: x.split(os.sep)[-1])\ntest_df['Label'] = new_preds\n\ntest_df = test_df.drop('ImageFileName', axis=1)\ntest_df.to_csv('submission_eb0.csv', index=False)\nprint(test_df.head())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}