{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Forked!\nOriginal : https://www.kaggle.com/saife245/melanoma-detail-analysis-eda-ip-augmentation-model\n\n    Another great EDA kernel: https://www.kaggle.com/parulpandey/melanoma-classification-eda-starter\n    Also: https://www.kaggle.com/nroman/melanoma-pytorch-starter-efficientnet\n\nFor this kernel, I'm also studying PyTorch.\nhttps://www.kaggle.com/zzy990106/pytorch-5-fold-efficientnet-baseline\n\nThere's also apparently a library for pre-trained models / architectures: for PyTorch Timm\n\nAutoAugment\nhttps://arxiv.org/abs/1805.09501\n\n\n    Pre-processed with external images: From Roman\n    https://www.kaggle.com/nroman/melanoma-external-malignant-256","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## TODO:\n    1. Add metadata as features - age, sex, anatomy site\n    2. EDA - RandomChoice -> random brightness, contrast, gamma, color jitter\n    3. EDA - CLAHE (https://albumentations.readthedocs.io/en/latest/api/augmentations.html)\n    4. Cutout - https://albumentations.readthedocs.io/en/latest/api/augmentations.html\n    5. Saliency Maps - https://medium.com/datadriveninvestor/visualizing-neural-networks-using-saliency-maps-in-pytorch-289d8e244ab4\n\n\n## History\n    1. Baseline - efficientnet b1, 224x224 images, cross entropy loss with 1-2 weighting (0.86)\n    2. Upsample to 256 x 256 images (0.87)\n        2.1 Add\n            a. Add Color Jitter\n            b. Random Erasure\n    3. Cross Entropy Weighting 10 - 1... Plateau patience = 2 (0.85)\n    4. Label smoothing... Plateau patience = 3... Remove rotation augmentation (0.88)\n    5. Add hair transformation... Resize to 240... switch to torch toolbox, label smoothing (0.87)\n    6. Add metadata features, test time augmentation (0.897)\n    7. Efficient Net (vanilla pretrained), TTA=5, hairs up to 10\n    7. More metadata features, more test time augmentation \n    6. rollback to 256...patience to 3...epochs to 12... batch size to 128\n    2. Switch to efficient_net_pytorch\n    \n    3. Switch to 256 MB images\n\n\n## Other things:\n    1. Oversample and equalize classes","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install efficientnet_pytorch timm torchtoolbox","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Imports","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport gc\nimport json\nimport math\nimport cv2\nimport PIL\nimport re\nimport numpy as np\nimport pandas as pd\nfrom PIL import Image\nimport tensorflow as tf\nfrom tensorflow.keras import layers\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.optimizers import Adam\nimport matplotlib.pyplot as plt\n#from sklearn.metrics import cohen_kappa_score, accuracy_score\nimport scipy\nfrom tqdm import tqdm\n%matplotlib inline\n#from keras.preprocessing import image\nimport glob\nimport tensorflow.keras.applications.densenet as dense\nfrom kaggle_datasets import KaggleDatasets\nimport seaborn as sns\nsns.set_style('whitegrid')\n\nimport missingno as msno\n\nfrom plotly.offline import iplot\nimport cufflinks\ncufflinks.go_offline()\ncufflinks.set_config_file(world_readable=True, theme='pearl')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nimport os\nimport cv2\nimport torch.nn.init as init\nimport torch\nimport torch.nn as nn\nfrom PIL import Image, ImageFilter\nfrom sklearn.model_selection import train_test_split, StratifiedKFold\nfrom torch.utils.data import Dataset\n# from torchvision import transforms\nimport torchtoolbox.transform as transforms\n# from albumentations.augmentations.transforms import Cutout\n# from torchtoolbox.transform import Cutout\n\nfrom torch.optim import Adam, SGD, RMSprop\nimport time\nfrom torch.autograd import Variable\nimport torch.functional as F\nfrom tqdm import tqdm\nfrom sklearn import metrics\nimport urllib\nimport pickle\nimport cv2\nimport torch.nn.functional as F\nfrom torchvision import models\nimport seaborn as sns\nimport random\nimport timm\nfrom sklearn.metrics import roc_auc_score\nimport sys\n# sys.path.append('../input/autoaug')\n# from auto_augment import AutoAugment, Cutout","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_path = '../input/melanoma-external-malignant-256/train/train/'\ntest_path = '../input/melanoma-external-malignant-256/test/test/'\n\n# train = pd.read_csv('/kaggle/input/siim-isic-melanoma-classification/train.csv')\ntrain = pd.read_csv('/kaggle/input/melanoma-external-malignant-256/train_concat.csv')\ntest = pd.read_csv('/kaggle/input/siim-isic-melanoma-classification/test.csv')\n\nprint('Train: ', train.shape)\nprint(\"Test:\", test.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Metadata features\n(From Roman)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# from Roman\n\n# One-hot encoding of anatom_site_general_challenge feature\nall_anatom = pd.concat([train['anatom_site_general_challenge'], test['anatom_site_general_challenge']], ignore_index=True)\ndummies = pd.get_dummies(all_anatom, dummy_na=True, dtype=np.uint8, prefix='site')\ntrain = pd.concat([train, dummies.iloc[:train.shape[0]]], axis=1)\ntest = pd.concat([test, dummies.iloc[train.shape[0]:].reset_index(drop=True)], axis=1)\n\n# Sex features\ntrain['sex'] = train['sex'].map({'male': 1, 'female': 0})\ntest['sex'] = test['sex'].map({'male': 1, 'female': 0})\ntrain['sex'] = train['sex'].fillna(-1)\ntest['sex'] = test['sex'].fillna(-1)\n\n# Age features\ntrain['age_approx'] /= train['age_approx'].max()\ntest['age_approx'] /= test['age_approx'].max()\ntrain['age_approx'] = train['age_approx'].fillna(0)\ntest['age_approx'] = test['age_approx'].fillna(0)\n\n# Shall we use patient ID?\n# train['patient_id'] = train['patient_id'].fillna(0)\n\nmeta_features = ['sex', 'age_approx'] + [col for col in train.columns if 'site_' in col]\nmeta_features.remove('anatom_site_general_challenge')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# EfficientNet modeling","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def seed_everything(seed):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.benchmark = True\n    torch.backends.cudnn.deterministic = True\n    \nseed_everything(2020)\n\nmean_images = [0.485, 0.456, 0.406]\nstd_images = [0.229, 0.224, 0.225]\n\nnum_classes = 2\nbs = 64\nlr = 1e-3\n# IMG_SIZE = 224\nIMG_SIZE = 256\nn_epochs = 12\npatience = 3\n\n# positive weight\npos_weight = torch.Tensor(2).to(device)\npos_weight[0] = 1\npos_weight[1] = 1\n\n# MODEL_NAME = 'tf_efficientnet_b1_ns'\nMODEL_NAME = 'tf_efficientnet_b1'\nPRETRAINED=True\n\nsample = pd.read_csv('../input/siim-isic-melanoma-classification/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class MyDataset(Dataset):\n    \n    def __init__(self, dataframe, transform=None, test_phase=False, meta_features = None):\n        self.df = dataframe\n        self.transform = transform\n        self.test_phase = test_phase\n        self.meta_features = meta_features\n    \n    def __len__(self):\n        return len(self.df)\n    \n    def __getitem__(self, idx):\n        p = self.df.image_name.values[idx]\n        meta = np.array(self.df.iloc[idx][self.meta_features].values, dtype=np.float32)\n        \n        if self.test_phase == False:\n            p_path = train_path + p + '.jpg'\n        else:\n            p_path = test_path + p + '.jpg'\n            \n        image = cv2.imread(p_path)\n        # image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n#         image = cv2.resize(image, (IMG_SIZE, IMG_SIZE))\n#         image = transforms.ToPILImage()(image)\n        \n        if self.transform:\n            image = self.transform(image)\n        \n        \n        if not self.test_phase:\n            label = self.df.target.values[idx]\n            return image, meta, label\n        if self.test_phase:\n            return image, meta\n\n    \nclass AverageMeter:\n    \"\"\"\n    Computes and stores the average and current value\n    \"\"\"\n    def __init__(self):\n        self.reset()\n\n    def reset(self):\n        self.val = 0\n        self.avg = 0\n        self.sum = 0\n        self.count = 0\n\n    def update(self, val, n=1):\n        self.val = val\n        self.sum += val * n\n        self.count += n\n        self.avg = self.sum / self.count","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Roman's work","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class AdvancedHairAugmentation:\n    \"\"\"\n    Impose an image of a hair to the target image\n\n    Args:\n        hairs (int): maximum number of hairs to impose\n        hairs_folder (str): path to the folder with hairs images\n    \"\"\"\n\n    def __init__(self, hairs: int = 10, hairs_folder: str = \"\"):\n        self.hairs = hairs\n        self.hairs_folder = hairs_folder\n\n    def __call__(self, img):\n        \"\"\"\n        Args:\n            img (PIL Image): Image to draw hairs on.\n\n        Returns:\n            PIL Image: Image with drawn hairs.\n        \"\"\"\n        n_hairs = random.randint(0, self.hairs)\n        \n        if not n_hairs:\n            return img\n        \n        height, width, _ = img.shape  # target image width and height\n        hair_images = [im for im in os.listdir(self.hairs_folder) if 'png' in im]\n        \n        for _ in range(n_hairs):\n            hair = cv2.imread(os.path.join(self.hairs_folder, random.choice(hair_images)))\n            hair = cv2.flip(hair, random.choice([-1, 0, 1]))\n            hair = cv2.rotate(hair, random.choice([0, 1, 2]))\n\n            h_height, h_width, _ = hair.shape  # hair image width and height\n            roi_ho = random.randint(0, img.shape[0] - hair.shape[0])\n            roi_wo = random.randint(0, img.shape[1] - hair.shape[1])\n            roi = img[roi_ho:roi_ho + h_height, roi_wo:roi_wo + h_width]\n\n            # Creating a mask and inverse mask\n            img2gray = cv2.cvtColor(hair, cv2.COLOR_BGR2GRAY)\n            ret, mask = cv2.threshold(img2gray, 10, 255, cv2.THRESH_BINARY)\n            mask_inv = cv2.bitwise_not(mask)\n\n            # Now black-out the area of hair in ROI\n            img_bg = cv2.bitwise_and(roi, roi, mask=mask_inv)\n\n            # Take only region of hair from hair image.\n            hair_fg = cv2.bitwise_and(hair, hair, mask=mask)\n\n            # Put hair in ROI and modify the target image\n            dst = cv2.add(img_bg, hair_fg)\n\n            img[roi_ho:roi_ho + h_height, roi_wo:roi_wo + h_width] = dst\n                \n        return img\n\n    def __repr__(self):\n        return f'{self.__class__.__name__}(hairs={self.hairs}, hairs_folder=\"{self.hairs_folder}\")'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_transform = transforms.Compose([\n    AdvancedHairAugmentation(hairs_folder='/kaggle/input/melanoma-hairs/'),\n    transforms.RandomResizedCrop(size=IMG_SIZE, scale=(0.8, 1.1)),\n    transforms.RandomHorizontalFlip(),\n    transforms.RandomVerticalFlip(),\n#     transforms.RandomRotation(45),\n    transforms.ColorJitter(brightness=0.25,saturation=0.5, contrast=0.25,),\n#     transforms.Cutout(scale=(0.05, 0.007), value=(0, 0)),\n    \n    transforms.ToTensor(),\n    transforms.RandomErasing(scale=(0.02, 0.25)),\n    transforms.Normalize(mean=mean_images, std=std_images)\n])\n\ntest_transform = transforms.Compose([\n    transforms.ToTensor(),\n    transforms.Normalize(mean=mean_images,std=std_images)\n])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Add a wrapper in the efficient net model\n(From Roman as well)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class Net(nn.Module):\n    def __init__(self, arch, n_meta_features: int, num_classes: int):\n        super(Net, self).__init__()\n        self.arch = arch\n        self.arch.classifier = nn.Linear(in_features=1280, out_features=500, bias=True)\n        self.meta = nn.Sequential(nn.Linear(n_meta_features, 500),\n                                  nn.BatchNorm1d(500),\n                                  nn.ReLU(),\n                                  nn.Dropout(p=0.2),\n                                  nn.Linear(500, 250),\n                                  nn.BatchNorm1d(250),\n                                  nn.ReLU(),\n                                  nn.Dropout(p=0.2))\n        self.output = nn.Linear(500 + 250, num_classes)\n        \n    def forward(self, inputs):\n        x, meta = inputs\n        cnn_features = self.arch(x)\n        meta_features = self.meta(meta)\n        features = torch.cat((cnn_features, meta_features), dim=1)\n        \n        output = self.output(features)\n        return output\n        \n#     def forward(self, inputs):\n#         \"\"\"\n#         No sigmoid in forward because we are going to use BCEWithLogitsLoss\n#         Which applies sigmoid for us when calculating a loss\n#         \"\"\"\n#         x, meta = inputs\n#         cnn_features = self.arch(x)\n#         meta_features = self.meta(meta)\n#         features = torch.cat((cnn_features, meta_features), dim=1)\n#         output = self.ouput(features)\n#         return output","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def train_model(model, train_loader, epoch):\n    model.train() \n    \n    losses = AverageMeter()\n    avg_loss = 0.\n\n    optimizer.zero_grad()\n    \n    tk = tqdm(train_loader, total=len(train_loader), position=0, leave=True)\n    for idx, (imgs, meta, labels) in enumerate(tk):\n        imgs_train, meta_train, labels_train = imgs.to(device), meta.to(device), labels.to(device).long()\n        output_train = model((imgs_train, meta_train))\n#         one_hot = torch.nn.functional.one_hot(labels_valid, num_classes=2).cuda().long()\n\n        loss = criterion(output_train, labels_train)\n        loss.backward()\n\n        optimizer.step() \n        optimizer.zero_grad() \n        \n        avg_loss += loss.item() / len(train_loader)\n        \n        losses.update(loss.item(), imgs_train.size(0))\n\n        tk.set_postfix(loss=losses.avg)\n        \n    return avg_loss\n\n\ndef test_model(model, val_loader):    \n    model.eval()\n    \n    losses = AverageMeter()\n    avg_val_loss = 0.\n    \n    valid_preds, valid_targets = [], []\n    \n    with torch.no_grad():\n        tk = tqdm(val_loader, total=len(val_loader), position=0, leave=True)\n        for idx, (imgs, meta, labels) in enumerate(tk):\n            imgs_valid, meta_valid, labels_valid = imgs.to(device), meta.to(device), labels.to(device).long()\n            output_valid = model((imgs_valid, meta_valid))\n            \n            loss = criterion(output_valid, labels_valid)\n            \n            avg_val_loss += loss.item() / len(val_loader)\n\n            losses.update(loss.item(), imgs_valid.size(0))\n            \n            tk.set_postfix(loss=losses.avg)\n            \n            valid_preds.append(torch.softmax(output_valid,1)[:,1].detach().cpu().numpy())\n            valid_targets.append(labels_valid.detach().cpu().numpy())\n            \n        valid_preds = np.concatenate(valid_preds)\n        valid_targets = np.concatenate(valid_targets)\n        auc =  roc_auc_score(valid_targets, valid_preds) \n            \n    return avg_val_loss, auc","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"kf = StratifiedKFold(5, shuffle=True, random_state=0)\n#     train_df = train.loc[trn_ind][:10].append(train.loc[trn_ind][-10:])\n#     val_df = train.loc[val_ind][:10].append(train.loc[val_ind][-10:])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Label Smoothing Implementation\nhttps://medium.com/towards-artificial-intelligence/how-to-use-label-smoothing-for-regularization-aa349f7f1dbb","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch.nn.functional as F\n\ndef linear_combination(x, y, epsilon): \n    return epsilon*x + (1-epsilon)*y\n\n\ndef reduce_loss(loss, reduction='mean'):\n    return loss.mean() if reduction=='mean' else loss.sum() if reduction=='sum' else loss\n\n\nclass LabelSmoothingCrossEntropy(nn.Module):\n    def __init__(self, epsilon:float=0.1, reduction='mean'):\n        super().__init__()\n        self.epsilon = epsilon\n        self.reduction = reduction\n    \n    def forward(self, preds, target):\n        n = preds.size()[-1]\n        log_preds = F.log_softmax(preds, dim=-1)\n        loss = reduce_loss(-log_preds.sum(dim=-1), self.reduction)\n        nll = F.nll_loss(log_preds, target, reduction=self.reduction)\n        return linear_combination(loss/n, nll, self.epsilon)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fold = 0\nlist_results = []\nlist_predictions = []\n\nfor fold, (trn_ind, val_ind) in enumerate(kf.split(train.image_name, train.target), 1):\n    print('fold:', fold)\n\n    train_df = train.loc[trn_ind]\n    val_df = train.loc[val_ind]\n    train_df.reset_index(drop=True, inplace=True)\n    val_df.reset_index(drop=True, inplace=True)\n\n    trainset = MyDataset(train_df, transform=train_transform, meta_features=meta_features)\n    train_loader = torch.utils.data.DataLoader(trainset, batch_size=bs, shuffle=True, num_workers=4)\n   \n    valset = MyDataset(val_df, transform=test_transform, meta_features=meta_features)\n    val_loader = torch.utils.data.DataLoader(valset, batch_size=bs, shuffle=False, num_workers=4)\n\n    model = timm.create_model(MODEL_NAME, pretrained=PRETRAINED, num_classes=num_classes)\n    model = Net(model, len(meta_features), num_classes)\n    model.to(device)\n\n    optimizer = torch.optim.AdamW(model.parameters(), lr=lr, weight_decay=0.001)\n#     criterion = nn.CrossEntropyLoss(weight=pos_weight)\n    criterion = LabelSmoothingCrossEntropy()\n#     criterion = nn.BCEWithLogitsLoss(pos_weight=pos_weight)\n    scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, patience=2, mode='min',verbose=True)\n\n    best_loss = np.inf\n    es = 0\n\n    for epoch in range(n_epochs):\n        avg_loss = train_model(model, train_loader, epoch)\n        avg_val_loss, auc = test_model(model, val_loader)\n\n        if avg_val_loss < best_loss:\n            best_loss = avg_val_loss\n            torch.save(model.state_dict(), str(fold) + 'weight.pt')\n            es = 0\n        else:\n            es += 1\n            if es > patience:\n                break\n        dict_result = {\"fold\" : fold, \"epoch\" : epoch, \"train loss\" : avg_loss, \"val loss\": avg_val_loss, \"auc\": auc}\n        list_results.append(dict_result)\n        print(\"epoch={},train_loss={},val_loss={},best_val_loss={},val_auc={}\".format(epoch, avg_loss, avg_val_loss, best_loss, auc))\n        \n        scheduler.step(avg_val_loss)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Predictions from all validation sets","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# store indexes for testing later\n\nindexes_to_store = []\nfor fold, (trn_ind, val_ind) in enumerate(kf.split(train.image_name, train.target), 1):\n    df_indexes = pd.DataFrame(np.hstack((trn_ind, val_ind)), columns=['row_id'])\n    df_indexes.loc[:len(trn_ind), \"set\"] = 'train'\n    df_indexes[\"set\"] = df_indexes[\"set\"].fillna(\"val\")\n\n    df_indexes[\"fold\"] = fold\n    indexes_to_store.append(df_indexes)\n    \ndf_indexes = pd.concat(indexes_to_store)\ndf_indexes.to_csv(\"indexes.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"val_preds = []\n\nfor fold in range(1, 6):\n    # get indexes\n    val_ind = df_indexes.loc[(df_indexes[\"fold\"] == fold) & (df_indexes[\"set\"] == \"val\"), \"row_id\"]\n    val_df = train.loc[val_ind]\n    val_df.reset_index(drop=True, inplace=True)\n    valset = MyDataset(val_df, transform=test_transform, meta_features=meta_features)\n    val_loader = torch.utils.data.DataLoader(valset, batch_size=bs, shuffle=False, num_workers=4)\n    \n    # predict\n    model = timm.create_model(MODEL_NAME, pretrained=PRETRAINED, num_classes=num_classes)\n    model = Net(model, len(meta_features), num_classes)\n    model.load_state_dict(torch.load(\"./{}weight.pt\".format(fold), ))\n    model.to(device)\n    \n    pred = np.zeros((len(val_ind),))\n    batch_df = []\n    with torch.no_grad():\n        for i, (imgs, meta, labels) in enumerate(tqdm(val_loader, position=0, leave=True)):\n            imgs_valid, meta_valid, labels_valid = imgs.to(device), meta.to(device), labels.to(device).long()\n            \n            pred = model((imgs_valid, meta_valid))\n            \n            # accumulate predictions\n            df_val_preds_batch = pd.DataFrame(torch.softmax(pred,1).cpu().detach().numpy(), columns = [\"val0\", \"val1\"])\n            df_val_preds_batch[\"actual\"] = np.array(labels_valid.cpu())\n            df_val_preds_batch[\"fold\"] = fold\n            batch_df.append(df_val_preds_batch)\n    \n    # add indexes to predictions\n    df_val_preds = pd.concat(batch_df)\n    df_val_preds[\"index\"] = val_ind.values\n    val_preds.append(df_val_preds)\n            \npd.concat(val_preds).to_csv(\"val_predictions.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Submission","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"all_models = []\nfor i in range(1, 6):\n    model = timm.create_model(MODEL_NAME, pretrained=PRETRAINED, num_classes=num_classes)\n    model = Net(model, len(meta_features), num_classes)\n    model.to(device)\n    model.load_state_dict(torch.load(\"./{}weight.pt\".format(i), ))\n    all_models.append(model)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"testset      = MyDataset(test, transform=train_transform, test_phase=True, meta_features=meta_features)\ntest_loader  = torch.utils.data.DataLoader(testset, batch_size=bs, shuffle=False, num_workers=4)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"TTA = 5\nlist_pred = []\npreds = torch.zeros((len(test)), dtype=torch.float32, device=device)  # Predictions for test test\n\nwith torch.no_grad():\n    for model in all_models:\n        for _ in range(TTA):\n            for i, (imgs, meta) in enumerate(tqdm(test_loader, position=0, leave=True)):\n                imgs_test, meta_test = imgs.to(device), meta.to(device)\n\n                pred = model((imgs_test, meta_test))\n\n                # accumulate predictions\n                pred = torch.softmax(pred,1)[:,1]\n                preds[i*bs: (i+1)*bs] += pred\n                \n        del pred, imgs, meta\n\n        preds /= TTA\n\n    preds /= len(all_models)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# test_pred = np.zeros((len(sample),))\n\n# for i, (imgs, meta) in enumerate(tqdm(test_loader, position=0, leave=True)):\n    \n#     list_pred = []\n    \n#     for model in all_models:\n#         imgs_test, meta_test = imgs.to(device), meta.to(device)\n\n#         pred = model((imgs_test, meta_test))\n\n#         # accumulate predictions\n#         pred = torch.softmax(pred,1).cpu().detach().numpy()[:,1]\n#         list_pred.append(pred)\n        \n#     test_pred[i*bs: (i+1)*bs] = np.array(list_pred).sum(axis=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# test_pred = np.zeros((len(sample),))\n\n# with torch.no_grad():\n#     for i, data in enumerate(tqdm(test_loader, position=0, leave=True)):\n#         images, _ = data\n#         images = images.to(device)\n        \n#         pred = np.zeros((len(val_ind),))\n        \n#         pred = (model1(images) + model2(images) + model3(images) + model4(images) + model5(images)) \\\n#              + (model1(images) + model2(images) + model3(images) + model4(images) + model5(images)) \\\n#              + (model1(images) + model2(images) + model3(images) + model4(images) + model5(images)) \\\n#              + (model1(images) + model2(images) + model3(images) + model4(images) + model5(images)) \n        \n#         pred = torch.softmax(pred,1).cpu().detach().numpy()[:,1]\n    \n#         test_pred[i*bs: (i+1)*bs] = pred","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(preds[:10])\n\nsample[\"target\"] = preds.cpu().detach().numpy()\n\nsample.to_csv('submission.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.DataFrame(list_results).to_csv(\"metrics.csv\", index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}