{"cells":[{"metadata":{"_uuid":"81d601b5aea88dc8ce6d1d465d5626fc85cfb376"},"cell_type":"markdown","source":"How much alike are train and test datasets? Can we rely on train cross validation to get a rough idea of the LB score? There are some hints throughout the discussions that the two sets may be different.\n\nWe will investigate this claim using adversarial validation to check this intuition. We will create a unified dataset of train + test and then train a classifier to distinguish between the two.\n\nThe results are difficult to separate? Good, train and test are not so different. \nThe results are easily separated? Well, we can expected surprises at the end of the competition.\n\n## Credits \n- @konradb's [Adversarial validation and other scary terms](https://www.kaggle.com/konradb/adversarial-validation-and-other-scary-terms)\n- roughly based on @hengck23's code for doodle challenge\n\nDisclaimer: till now I've mostly been a Keras guy. Let's see how I fare with PyTorch ;)"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport torch\nimport cv2\nimport random\nimport math\n\nfrom PIL import Image\nfrom timeit import default_timer as timer\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import roc_auc_score\nfrom torch import nn\nfrom torch.optim import SGD\nfrom torch.optim.lr_scheduler import StepLR\nfrom torchvision.models import resnet34\nfrom torch.utils.data import Dataset, DataLoader, SubsetRandomSampler\nfrom pathlib import Path\n\n\nINPUT_PATH = Path(\"../input\")\nTRAIN_PATH = INPUT_PATH / \"train\"\nTEST_PATH = INPUT_PATH / \"test\"\n\nFILTERS = [\"red\", \"green\", \"blue\", \"yellow\"]\nMEAN = [0.08069, 0.05258, 0.05487, 0.08282]\nSTD = [0.13704, 0.10145, 0.15313, 0.13814]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"01e7e6d0375bdbef14c5de2e72b3f3f9bdec0221"},"cell_type":"code","source":"SEED = 666\nrandom.seed(SEED)\nnp.random.seed(SEED)\ntorch.manual_seed(SEED)\ntorch.cuda.manual_seed_all(SEED)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6b22b2ffddb7fc870a84cedbe07d9c22513c5910"},"cell_type":"markdown","source":"Here we will use a unified PyTorch Dataset for train and test datasets. Label is `1` for train and `0` for test. We will use all 4 channels for this experiment."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"def null_collate(batch):\n    batch_size = len(batch)\n    images = np.array([x[0] for x in batch])\n    images = torch.from_numpy(images)\n    \n    labels = np.array([x[1] for x in batch])\n    labels = torch.from_numpy(labels)\n\n    assert(images.shape[0] == labels.shape[0] == batch_size)\n\n    return images, labels\n\n\nclass ProteinDataset(Dataset):\n    def __init__(self, size, augment=None):\n        super(ProteinDataset, self).__init__()\n        self.augment = augment\n\n        train_csv_filepath = INPUT_PATH / \"train.csv\"\n        self.train_df = self._load_dataframe(train_csv_filepath)\n        test_csv_filepath = INPUT_PATH / \"sample_submission.csv\"\n        self.test_df = self._load_dataframe(test_csv_filepath)\n        \n        self.train_df_len = len(self.train_df)\n        self.test_df_len = len(self.test_df)\n        \n        self.total_length = len(self.train_df) + len(self.test_df)\n        print(\"Train dataset: {}, test dataset {}, total {}\".format(self.train_df_len, self.test_df_len, len(self.train_df) + len(self.test_df)))\n        \n        self.labels = np.concatenate((\n            np.ones(self.train_df_len),\n            np.zeros(self.test_df_len)\n        ), axis=0)\n    \n    @staticmethod\n    def _load_dataframe(path):\n        print(\"Loading csv from {}\".format(path))\n        return pd.read_csv(path)\n\n    @staticmethod\n    def _load_image(path, size):\n        img = Image.open(path)\n        img = cv2.resize(np.array(img), (size, size), interpolation=cv2.INTER_AREA)\n        img = np.expand_dims(img, axis=2)\n        return img\n    \n    @staticmethod\n    def _get_row_id(df, index):\n        return df.loc[index, \"Id\"]\n\n    def __getitem__(self, index):       \n        if index > len(self.train_df) - 1:\n            # Index belongs to test dataset\n            offset_index = index - len(self.train_df)\n            image_id = self._get_row_id(self.test_df, offset_index)\n            img = np.concatenate([\n                self._load_image(TEST_PATH / (image_id + \"_\" + i + \".png\"), size) for i in FILTERS\n            ], axis=2)\n            label = 0\n        else:\n            # Index belongs to train dataset\n            image_id = self._get_row_id(self.train_df, index)\n            img = np.concatenate([\n                self._load_image(TRAIN_PATH / (image_id + \"_\" + i + \".png\"), size) for i in FILTERS\n            ], axis=2)\n            label = 1\n\n        img = np.transpose(img, axes=[2, 0, 1])\n\n        if self.augment is not None:\n            img = self.augment(img)\n\n        return img, label\n\n    def __len__(self):\n        return self.total_length","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5522ad1be3c8d2439fb12537b8e1e81e1b2971ae"},"cell_type":"markdown","source":"Let's define a simple model to test our hypotesis based on a pretrained ResNet34."},{"metadata":{"trusted":true,"_uuid":"1aac7a1a6f1034fd35ae0a98f0eff3cdb4d8f960"},"cell_type":"code","source":"class ResNet34(nn.Module):\n    def __init__(self, num_classes=1, dropout=0.5, middle_features=128):\n        super(ResNet34, self).__init__()\n        resnet = resnet34(pretrained=True)\n\n        # Support for 4-channels\n        w = resnet.conv1.weight\n        self.conv1 = nn.Conv2d(4, 64, kernel_size=7, stride=2, padding=3, bias=False)\n        self.conv1.weight = torch.nn.Parameter(\n            torch.cat((w, torch.zeros(64, 1, 7, 7)), dim=1)\n        )\n        self.bn1 = resnet.bn1\n        self.relu = resnet.relu\n        self.maxpool = resnet.maxpool\n        self.layer1 = resnet.layer1\n        self.layer2 = resnet.layer2\n        self.layer3 = resnet.layer3\n        self.layer4 = resnet.layer4\n        self.avgpool = nn.AdaptiveAvgPool2d((1, 1))#resnet.avgpool\n        \n        bottleneck_features = resnet.fc.in_features\n        self.fc = nn.Sequential(\n            nn.BatchNorm1d(bottleneck_features),\n            nn.Dropout(dropout),\n            nn.Linear(bottleneck_features, middle_features),\n            nn.ReLU(),\n            nn.BatchNorm1d(middle_features),\n            nn.Dropout(dropout),\n            nn.Linear(middle_features, num_classes),\n            nn.Sigmoid()\n        )\n        \n    def forward(self, x):\n        mean = MEAN\n        std = STD\n        x = x / 255.\n        x = torch.cat([\n            (x[:, [0]] - mean[0]) / std[0],\n            (x[:, [1]] - mean[1]) / std[1],\n            (x[:, [2]] - mean[2]) / std[2],\n            (x[:, [3]] - mean[3]) / std[3],\n        ], 1)\n        x = self.conv1(x)\n        x = self.bn1(x)\n        x = self.relu(x)\n        x = self.maxpool(x)\n\n        x = self.layer1(x)\n        x = self.layer2(x)\n        x = self.layer3(x)\n        x = self.layer4(x)\n\n        x = self.avgpool(x) \n        x = x.view(x.size(0), -1)\n        x = self.fc(x)\n        return x","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f2e50f74269a707777aa0543e990e3dce9684f99"},"cell_type":"markdown","source":"Let's setup an accuracy metric (just for vanity ;) )"},{"metadata":{"trusted":true,"_uuid":"283e9127272c615d145ef9cc0718f114144f93b0"},"cell_type":"code","source":"class Accuracy(nn.Module):\n    def __init__(self, threshold=0.5):\n        super().__init__()\n        self.threshold = threshold\n\n    def forward(self, y_true, y_pred):\n        y_pred = (y_pred > self.threshold).int()\n        y_true = y_true.int()\n        return (y_pred == y_true).float().mean()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"07433b9971981975ca778b32da6f2cbca7f4d06d"},"cell_type":"markdown","source":"Now it's time for the core of our kernel: the trainer class that contains the training/validation loop."},{"metadata":{"trusted":true,"_uuid":"ef434b5dd462a161c1acc5597070e992413590c1"},"cell_type":"code","source":"class Trainer:\n    def __init__(self, batch_size, size):\n        self.batch_size = batch_size\n        self.size = size\n\n        self.optimizer = None\n        self.scheduler = None\n\n        self.train_dataset = ProteinDataset(size)\n        self.validation_dataset = ProteinDataset(size)\n        \n        self.train_idx, self.validation_idx = train_test_split(\n            list(range(len(self.train_dataset))),\n            test_size=0.1,\n            stratify=self.train_dataset.labels\n        )\n        \n        print(\"Train len: {}, validation len: {}\".format(len(self.train_idx), len(self.validation_idx)))\n\n        loader_params = dict(\n            batch_size=batch_size,\n            num_workers=2,\n            pin_memory=True,\n            collate_fn=null_collate\n        )\n        self.train_loader = DataLoader(\n            dataset=self.train_dataset,\n            sampler=SubsetRandomSampler(self.train_idx),\n            **loader_params\n        )\n        self.validation_loader = DataLoader(\n            dataset=self.validation_dataset,\n            sampler=SubsetRandomSampler(self.validation_idx),\n            **loader_params\n        )\n        print(\"Train set: {}\".format(len(self.train_idx)))\n        print(\"Validation set: {}\".format(len(self.validation_idx)))\n\n        self.it_per_epoch = math.ceil(len(self.train_idx) / self.batch_size)\n        \n        \n    def run(self):\n        model = ResNet34()\n        model = model.cuda()\n\n        lr = 0.2\n        it = 0\n        epoch = 0\n        max_epochs = 20\n        it_save = self.it_per_epoch * 5\n        it_log = self.it_per_epoch / 5\n        it_smooth = 50\n        \n        self.optimizer = SGD(filter(lambda p: p.requires_grad, model.parameters()), lr=lr, momentum=0.9, weight_decay=0.0001)\n        self.scheduler = StepLR(self.optimizer, 5 * self.it_per_epoch, gamma=0.5)\n\n        criterion = nn.BCELoss()\n        criterion = criterion.cuda()\n        metrics = [Accuracy(), roc_auc_score]\n\n        print(\"{}'\".format(self.optimizer))\n        print(\"{}'\".format(self.scheduler))\n        print(\"{}'\".format(criterion))\n        print(\"{}'\".format(metrics))\n\n        train_loss = 0\n        train_acc = 0\n\n        print('                    |         VALID         |        TRAIN          |         ')\n        print(' lr     iter  epoch | loss    roc    acc    | loss    roc    acc    |  time   ')\n        print('------------------------------------------------------------------------------')\n\n        start = timer()\n        while epoch < max_epochs:\n            smoothed_train_loss = 0\n            smoothed_sum = 0\n\n            for inputs, labels in self.train_loader:\n                epoch = (it + 1) / self.it_per_epoch\n\n                # checkpoint\n                if it % it_save == 0 and it != 0:\n                    self.save(model, self.optimizer, it, epoch)\n\n                # training\n                self.scheduler.step()\n\n                lrs = self.scheduler.get_lr()\n                lr = lrs[-1]\n\n                model.train()\n                inputs = inputs.cuda().float()\n                labels = labels.cuda().float()\n\n                preds = model(inputs)\n                loss = criterion(preds, labels)\n                with torch.no_grad():\n                    train_acc, train_roc = [i(labels, preds).item() for i in metrics]\n\n                self.optimizer.zero_grad()\n                loss.backward()\n\n                torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)\n                self.optimizer.step()\n\n                smoothed_train_loss += loss.item()\n                smoothed_sum += 1\n                if it % it_smooth == 0:\n                    train_loss = smoothed_train_loss / smoothed_sum\n                    smoothed_train_loss = 0\n                    smoothed_sum = 0\n\n                if it % it_log == 0:\n                    batch_loss = loss.item()\n                    print(\n                        \"{:5f} {:4.1f} {:5.1f} |                    | {:0.3f}  {:0.3f}  {:0.3f}  | {:6.2f}\".format(\n                            lr, it / 1000, epoch, batch_loss, train_roc, train_acc, timer() - start\n                        ))\n\n                it += 1\n\n            # validation\n            valid_loss, valid_m = self.do_valid(model, criterion, metrics)\n            valid_acc, valid_roc = valid_m\n\n            print(\n                \"{:5f} {:4.1f} {:5.1f} | {:0.3f}* {:0.3f}  {:0.3f}  | {:0.3f}  {:0.3f}  {:0.3f}  | {:6.2f}\".format(\n                    lr, it / 1000, epoch, valid_loss, valid_roc, valid_acc, train_loss, train_roc, train_acc, timer() - start\n                ))\n\n            # Data loader end\n        # Training end\n\n        self.save(model, self.optimizer, it, epoch)\n\n    def do_valid(self, model, criterion, metrics):\n        model.eval()\n        valid_num = 0\n        losses = []\n\n        for inputs, labels in self.validation_loader:\n            inputs = inputs.cuda().float()\n            labels = labels.cuda().float()\n\n            with torch.no_grad():\n                preds = model(inputs)\n                loss = criterion(preds, labels)\n                m = [i(labels, preds).item() for i in metrics]\n\n            valid_num += len(inputs)\n            losses.append(loss.data.cpu().numpy())\n\n        assert (valid_num == len(self.validation_loader.sampler))\n        loss = np.array(losses).mean()\n        return loss, m\n    \n    def save(self, model, optimizer, iter, epoch):\n        torch.save(model.state_dict(), \"{}_model.pth\".format(iter))\n        torch.save({\n            \"optimizer\": optimizer.state_dict(),\n            \"iter\": iter,\n            \"epoch\": epoch\n        }, \"{}_optimizer.pth\".format(iter))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"956ab5a684ceef4927ea562e07e9b509936714aa"},"cell_type":"code","source":"batch_size = 64\nsize = 256","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"afbbf314ed2f066becb3ec1fac75ad881b689777","scrolled":false},"cell_type":"code","source":"trainer = Trainer(batch_size, size)\ntrainer.run()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"85fe9ca87f02c89b35cc5e8d44e570f273099b02"},"cell_type":"markdown","source":"The results are pretty clear. Even without special tuning, the classifier is able to easily distinguish between train and test set. We can conclude that finding a proper validation method is paramount fora smooth sailing towards the private LB.\n\nSuggestions and feedback are welcome!"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}