{"cells":[{"metadata":{},"cell_type":"markdown","source":"# We will use TTA to make our prediction more accurate.\n\n# So What Exactly is TTA?\n\n**Similar to what Data Augmentation is doing to the training set, the purpose of Test Time Augmentation is to perform random modifications to the test images. Thus, instead of showing the regular, “clean” images, only once to the trained model, we will show it the augmented images several times. We will then average the predictions of each corresponding image and take that as our final guess.**\n","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Link to the efficientnet pytorch library**\n[https://github.com/lukemelas/EfficientNet-PyTorch](http://)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install efficientnet-pytorch","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**These are not the original dataset, the original ones had some duplicates so they were marked in this dataset and it also is resized to different sizes I am using the 256,256 one.**\n\n[https://www.kaggle.com/cdeotte/jpeg-melanoma-256x256](http://)","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train = pd.read_csv(\"../input/jpeg-melanoma-256x256/train.csv\")\ntest = pd.read_csv(\"../input/jpeg-melanoma-256x256/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from pathlib import Path\nimport pandas as pd\nfrom torch.utils.data import Dataset,DataLoader\nfrom PIL import Image\nfrom torchvision import transforms as T\nimport torch.nn as nn\nimport torch\nimport torch.nn.functional as F\nimport numpy as np\nfrom fastprogress.fastprogress import master_bar, progress_bar\nfrom sklearn.metrics import accuracy_score, roc_auc_score\nfrom efficientnet_pytorch import EfficientNet\nfrom torchvision import models\nimport pdb\nimport albumentations as A\nfrom albumentations.pytorch.transforms import ToTensor\nimport matplotlib.pyplot as plt\n\nimport pickle ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Different kind of augmentaion done on the images.(copied not mine)**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_augmentations(p=0.5):\n    imagenet_stats = {'mean':[0.485, 0.456, 0.406], 'std':[0.229, 0.224, 0.225]}\n    train_tfms = A.Compose([\n        A.Cutout(p=p),\n        A.RandomRotate90(p=p),\n        A.Flip(p=p),\n        A.OneOf([\n            A.RandomBrightnessContrast(brightness_limit=0.2,\n                                       contrast_limit=0.2,\n                                       ),\n            A.HueSaturationValue(\n                hue_shift_limit=20,\n                sat_shift_limit=50,\n                val_shift_limit=50)\n        ], p=p),\n        A.OneOf([\n            A.IAAAdditiveGaussianNoise(),\n            A.GaussNoise(),\n        ], p=p),\n        A.OneOf([\n            A.MotionBlur(p=0.2),\n            A.MedianBlur(blur_limit=3, p=0.1),\n            A.Blur(blur_limit=3, p=0.1),\n        ], p=p),\n        A.ShiftScaleRotate(shift_limit=0.0625, scale_limit=0.2, rotate_limit=45, p=p),\n        A.OneOf([\n            A.OpticalDistortion(p=0.3),\n            A.GridDistortion(p=0.1),\n            A.IAAPiecewiseAffine(p=0.3),\n        ], p=p), \n        ToTensor(normalize=imagenet_stats)\n        ])\n    \n    test_tfms = A.Compose([\n        ToTensor(normalize=imagenet_stats)\n        ])\n    return train_tfms, test_tfms","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**This function just gets rid of the rows with tfrecords = -1 and splits the dataset into train and val.**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_train_val_split(df):\n    df = df[df.tfrecord != -1].reset_index(drop=True)\n    train_tf_records = list(range(len(df.tfrecord.unique())))[:12]\n    split_cond = df.tfrecord.apply(lambda x: x in train_tf_records)\n    train_df = df[split_cond].reset_index()\n    valid_df = df[~split_cond].reset_index()\n    return train_df,valid_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class MelanomaDataset(Dataset):\n    def __init__(self,df,img_path,transform=None,is_test=False):\n        self.df = df\n        self.img_path = img_path\n        self.transform = transform\n        self.is_test = is_test\n    def __len__(self):\n        return len(self.df)\n    def __getitem__(self,index):\n        img_path = f\"{self.img_path}/{self.df.iloc[index]['image_name']}.jpg\"\n        img = Image.open(img_path)\n        \n        if self.transform:\n            img = self.transform(**{\"image\": np.array(img)})[\"image\"]\n        if self.is_test:\n            return img\n        target = self.df.iloc[index][\"target\"]\n        return img,torch.tensor([target],dtype=torch.float32)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class MelanomaEfficientNet(nn.Module):\n    def __init__(self,model_name=\"efficientnet-b0\",pool_type=F.adaptive_avg_pool2d):\n        super().__init__()\n        self.pool_type = pool_type\n        self.model = EfficientNet.from_pretrained(model_name)\n        in_features = 2560  #number of features obtained after image passing through different layers(check in efficientnet library)\n        self.classifier = nn.Linear(in_features,1)\n    \n    def forward(self,x):\n        features = self.pool_type(self.model.extract_features(x),1) # extract_features extract the features from the image.Vector length of 2560.\n        features = features.view(x.size(0),-1)\n        return self.classifier(features)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"path = \"../input/jpeg-melanoma-256x256\"\ndef data(train_df,valid_df,train_tfms,test_tfms,bs):\n    train_ds = MelanomaDataset(df=train_df,img_path=path+\"/train\",transform=train_tfms)\n    valid_ds = MelanomaDataset(df=valid_df,img_path=path+'/train',transform=test_tfms)\n    train_dl = DataLoader(dataset=train_ds,batch_size=bs,shuffle=True,num_workers=4)\n    valid_dl = DataLoader(dataset=valid_ds,batch_size=bs*2,shuffle=False,num_workers=4)\n    return train_dl,valid_dl","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train,X_val = get_train_val_split(train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train.shape,X_val.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_tfms,test_tfms = get_augmentations(p=0.5)\ntrain_dl,val_dl = data(X_train,X_val,train_tfms,test_tfms,bs=16)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = MelanomaEfficientNet(model_name=\"efficientnet-b7\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true},"cell_type":"code","source":"optimizer = torch.optim.AdamW(model.parameters(),lr=1e-5,weight_decay=0.01)\ndevice = torch.device(\"cuda\")\nmodel.to(device)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, len(train_dl)*10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mb = master_bar(range(10))\nmb.write(['epoch','train_loss','valid_loss','val_roc'],table=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"loss_fn=F.binary_cross_entropy_with_logits","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"val_rocs = []\n\nfor epoch in mb:\n    \n    train_loss,val_loss = 0.0,0.0\n    \n    val_preds = np.zeros((len(val_dl.dataset),1))\n    val_targs = np.zeros((len(val_dl.dataset),1))\n    \n    model.train()\n    \n    for xb,yb in progress_bar(train_dl,parent=mb):\n        xb,yb=xb.to(device), yb.to(device)\n        \n        out = model(xb)\n        optimizer.zero_grad()\n        \n        loss = loss_fn(out,yb)\n        loss.backward()\n        \n        optimizer.step()\n        scheduler.step()\n        \n        train_loss += loss.item()\n    train_loss /= mb.child.total\n    print(f\"Epoch {epoch}: Training loss: {train_loss}\")\n    \n    model.eval()\n    \n    with torch.no_grad():\n        for i,(xb,yb) in enumerate(progress_bar(val_dl,parent=mb)):\n            xb,yb= xb.to(device),yb.to(device)\n            \n            out = model(xb)\n            \n            loss =loss_fn(out,yb)\n            out = torch.sigmoid(out)\n            \n            val_loss += loss.item()\n            bs = xb.shape[0]\n            \n            val_preds[i*bs:i*bs+bs] = out.cpu().numpy()\n            val_targs[i*bs:i*bs+bs] = yb.cpu().numpy()\n            \n    val_loss /= mb.child.total\n    val_roc = roc_auc_score(val_targs.reshape(-1),val_preds.reshape(-1))\n    val_rocs.append(val_roc)\n    print(f\"Epoch {epoch}: Validation loss: {val_loss}\")\n\n    mb.write([epoch,f'{train_loss:.6f}',f'{val_loss:.6f}',f'{val_roc:.6f}'],table=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"imagenet_stats = {'mean':[0.485, 0.456, 0.406], 'std':[0.229, 0.224, 0.225]}\ntest_tfms = A.Compose([\n    A.RandomRotate90(p=0.5),\n        A.Flip(p=0.5),\n        A.OneOf([\n            A.RandomBrightnessContrast(brightness_limit=0.2,\n                                       contrast_limit=0.2,\n                                       ),\n            A.HueSaturationValue(\n                hue_shift_limit=20,\n                sat_shift_limit=50,\n                val_shift_limit=50)\n        ], p=0.5),\n        A.OneOf([\n            A.IAAAdditiveGaussianNoise(),\n            A.GaussNoise(),\n        ], p=0.5),\n    ToTensor(normalize=imagenet_stats)\n    ])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_ds = MelanomaDataset(test,img_path=\"../input/jpeg-melanoma-256x256/test\",is_test=True,transform=test_tfms)\ntest_dl = DataLoader(dataset = test_ds,batch_size=16,shuffle=False,num_workers=4)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"*As you can see we are predicting the same test set tta times but due to random augmentation of the images the prediction varies and can be very helpfull in the results.*","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_preds(model,device=None,tta=3):\n    device=torch.device(\"cuda\")\n    model.to(device)\n    preds = np.zeros(len(test_ds))\n    for tta_id in range(tta):\n        test_preds = []\n        with torch.no_grad():\n            for xb in test_dl:\n                xb = xb.to(device)\n                out = model(xb)\n                out = torch.sigmoid(out)\n                test_preds.extend(out.cpu().numpy())\n            preds += np.array(test_preds).reshape(-1)\n        print(f'TTA {tta_id}')\n    preds /= tta\n    return preds\npreds = get_preds(model,tta=25)  ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"subm = pd.read_csv(\"../input/jpeg-melanoma-256x256/sample_submission.csv\")\nsubm.target = preds\nsubm.to_csv('submission.csv',index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}