{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Update Log\n\nv14 : Added some new features , now local CV has climed to 0.87 just with Meta-Features and can climb even more , this is the first notebook exploring high end score with just the metafeatures.\n\nI have also removed the embedding layer because the model performs better without it\n\nv15 : Inference added\n\nv18: Trail with New features","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# About this competition\n\nHello everyone , In this competition , we are asked to classify whether a person has beningn or a melignant melanoma based on the images of skin lesions taken from various parts of the body over different period of times . We have been also given some metadata information to improve results. Given this let's see what we know so far\n\n# What we know so far?\n\nWe know the data is highly imbalanced with almost 98 percent of images being beningn .The discussion forum is filled with threads expressing kernels about the mystery images (image clusters that are present in test set but not in train set). This notebook does not include any EDA and discussions regarding what has already been discovered and explained thoroughly but for all those who are just getting started with this competition , I am adding all the necessary and important kernels and discussions to get up to speed quickly :-\n\n* [EDA kernel by Andrada](https://www.kaggle.com/andradaolteanu/siim-melanoma-competition-eda-augmentations)\n* [Exceptional Kernel giving all the insights about this competition by Laura](https://www.kaggle.com/allunia/don-t-turn-into-a-smoothie-after-the-shake-up)\n* [Code used to merge external Data and Data Splitting by Alex](https://www.kaggle.com/shonenkov/merge-external-data)\n* [Best Public TensorFlow Pipeline with explanantion of best CV strategy by chris deotte](https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords)\n* [Mystery Images Discussion Thread by Chris Deotte](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/168028)\n\nNow that you know what the competition is about the underlying difficulties and solutions which people have adapted so far let's understand what this notebook is all about\n\n# About this Notebook\n\nWe have been given two types of data , one is the images of skin lesions of patients , other is the tabular meta data . Now there are three ways of combining these two information together :-\n\n* Build a CNN image model and find a way to input the tabular data into the CNN image model\n* Build a Tabular data model and find a way to extract image embeddings or image features and input into the Tabular data model\n* Build 2 separate models for images and metadata and ensemble\n\n**We have tried all three and the third option works the best and gives significant amount of boost. Now another question comes to find what models can we use for modelling with tabular data , this notebook tries to answer this question only**\n\n<font color='orange'>What if I say you can use a neural network architecture based on attention and transformers especially designed for tabular data,that too with your own custom loss, custom lr and all the techniques that you might be applying with your image model , one such architecture which can give very could results IMO is Google's Tabnet. </font>\n\nPeople have already applied Tabnet for this competition using Pytorch-Tabnet implementation by Sebastien ([@optimo](https://www.kaggle.com/optimo)) . The Implementation can be found [here](https://github.com/dreamquark-ai/tabnet). However this implementations comes with following limitations : \n* We cannot use custom losses, custom LR schedulers\n* We cannot use custom samplers which we found to have improved results considerably\n* We are provided with a scikit-learn type interface which makes it really easy to use tabnet but at the same time takes away the benifits of it being a deep learning model\n\n**This Notebook also tries to address and solve these limitations**\n\n# How this  Notebook solves the Limitations\n\nHere in this notebook I show how to use Tabnet as a custom Model instead of the scikit-learn type interface provided by Pytorch-Tabnet , thanks to Sebastien and his active responses on the github repo . I show how anyone can use Tabnet just like a torchvision or any torch-hub model for any downstream task . I have tried to write a code in such a way that anyone can use this code and apply it to their tasks by just changing the dataset and dataloader. Here are the components that any deep learning models need :-\n* Dataset + DataLoader\n* Model\n* Criterion/Loss\n* Training Loop\n* Evaluation Loop\n* Engine for uniting all of them together\n\n# Things used Specific to this competition\n\n* For Data I am using the dataset and folds provided by [@alex](https://www.kaggle.com/shonenkov) [here](https://www.kaggle.com/shonenkov/melanoma-merged-external-data-512x512-jpeg) which has been generated using the notebook [here](https://www.kaggle.com/shonenkov/merge-external-data)\n* Embeddings for Categorical Variables\n* Soft Margin Focal Loss , because it has seemed to work the best as of now , you can play around with that \n* Balance Sampler for balancing classes in each batch\n* A Plotting fucntion adapted from chris Deotte's Training kernel\n* ReduceOn Pleateau Lr Scheduler \n\n**I hope you all like my efforts and find this kernel useful** ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"<font color='red'> Before diving into code , if you want to understand how tabnet works , you can watch the following talk given\nby Sebastien","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from IPython.display import IFrame, YouTubeVideo\nYouTubeVideo('ysBaZO8YmX8',width=600, height=400)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"If you want to do it the scikit learn way here is a [notebook](https://www.kaggle.com/tanulsingh077/achieving-sota-results-with-tabnet) where I explain how to that","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"#Installing Pytorch-Tabnet\n!pip install -U catalyst\n!pip install pytorch-tabnet","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport random\nimport os\nimport seaborn as sns\nfrom tqdm.autonotebook import tqdm\ntqdm.pandas()\nfrom scipy.stats import skew \nimport pickle\nimport glob\n\n#Visuals\nimport matplotlib.pyplot as plt\n\n#torch\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.utils.data import Dataset,DataLoader\nimport catalyst\n#from catalyst.data.sampler import BalanceClassSampler\n\n#CV2\nimport cv2\n\n#Importing Tabnet\nfrom pytorch_tabnet.tab_network import TabNet\n\n#error\nfrom sklearn.preprocessing import LabelEncoder\nfrom sklearn.metrics import roc_auc_score","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Utils\n\nSince we are writing our custom model , we need early stopping which is present in Pytorch-Tabnet's implementation as a built-in.\nThe following Early-Stopping Implementation can monitor both minimization and maximization of quantities","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class EarlyStopping:\n    def __init__(self, patience=7, mode=\"max\", delta=0.001,verbose=False):\n        self.patience = patience\n        self.counter = 0\n        self.mode = mode\n        self.best_score = None\n        self.early_stop = False\n        self.delta = delta\n        self.verbose = verbose\n        if self.mode == \"min\":\n            self.val_score = np.Inf\n        else:\n            self.val_score = -np.Inf\n\n    def __call__(self, epoch_score, model, model_path):\n\n        if self.mode == \"min\":\n            score = -1.0 * epoch_score\n        else:\n            score = np.copy(epoch_score)\n\n        if self.best_score is None:\n            self.best_score = score\n            self.save_checkpoint(epoch_score, model, model_path)\n        elif score < self.best_score + self.delta:\n            self.counter += 1\n            if self.verbose:\n                print('EarlyStopping counter: {} out of {}'.format(self.counter, self.patience))\n                \n            if self.counter >= self.patience:\n                self.early_stop = True\n        else:\n            self.best_score = score\n            self.save_checkpoint(epoch_score, model, model_path)\n            self.counter = 0\n\n    def save_checkpoint(self, epoch_score, model, model_path):\n        if epoch_score not in [-np.inf, np.inf, -np.nan, np.nan]:\n            if self.verbose:\n                print('Validation score improved ({} --> {}). Saving model!'.format(self.val_score, epoch_score))\n            torch.save(model.state_dict(), model_path)\n        self.val_score = epoch_score","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Configuration\n\nWe define all the configuration needed elsewhere in the notebook here","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"BATCH_SIZE = 1024 \nEPOCHS = 150\nLR = 0.02\nseed = 2020   # seed for reproducible results\npatience = 50\ndevice = torch.device('cuda')\nFOLDS = 5","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Seed","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def seed_everything(seed):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"seed_everything(seed)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data Preparation and Feature Engineering\n\nHere we input the data and prepare it for inputting to the model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Defining Categorical variables and their Indexes, embedding dimensions , number of classes each have\ndf =pd.read_csv('../input/fe-using-only-competition-data-melanoma/melanoma_folds.csv')\n\ndf.drop(['image_id','stratify_group','center','diagnosis','benign_malignant'],axis=1,inplace=True)\ntarget = 'target'\nunused_feat = ['patient_id','fold']\nfeatures = [ col for col in df.columns if col not in unused_feat+[target]] \n\ncategorical_columns = []\n\nfor col in df.columns[df.dtypes == object]:\n    \n    if col not in unused_feat:\n        print(col, df[col].nunique())\n        \n        l_enc = LabelEncoder()\n        df[col] = l_enc.fit_transform(df[col].values)\n        \n        #SAVING LABEL _ ENC\n        output = open(f'{col}_encoder.pkl', 'wb')\n        pickle.dump(l_enc, output)\n        output.close()\n        \n        categorical_columns.append(col)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class MelanomaDataset(Dataset):\n    def __init__(self,features,target):\n        self.features = features\n        self.target = target\n        \n    def __len__(self):\n        return len(self.features)\n    \n    def __getitem__(self,idx):\n        return{\n            'features': torch.tensor(self.features[idx],dtype=torch.float),\n             'target': self.one_hot(2, self.target[idx])\n        }\n    \n    def get_targets(self):\n        return list(self.target)\n    \n    @staticmethod\n    def one_hot(size, target):\n        tensor = torch.zeros(size, dtype=torch.float32)\n        tensor[target] = 1.\n        return tensor","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model\n\nHere we built our Custom Tabnet model","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"class CustomTabnet(nn.Module):\n    def __init__(self, input_dim, output_dim,n_d=8, n_a=8,n_steps=3, gamma=1.3,\n                cat_idxs=[], cat_dims=[], cat_emb_dim=1,n_independent=2, n_shared=2,\n                momentum=0.02,mask_type=\"sparsemax\"):\n        \n        super(CustomTabnet, self).__init__()\n        self.tabnet = TabNet(input_dim=input_dim,output_dim=output_dim, n_d=n_d, n_a=n_a,n_steps=n_steps, gamma=gamma,\n                             cat_idxs=cat_idxs, cat_dims=cat_dims, cat_emb_dim=cat_emb_dim,n_independent=n_independent,\n                             n_shared=n_shared, momentum=momentum,mask_type=\"sparsemax\")\n        \n        \n    def forward(self, x):\n        return self.tabnet(x)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Loss\n\nDefining SoftMarginFocal Loss which is to be used as a criterion","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class SoftMarginFocalLoss(nn.Module):\n    def __init__(self, margin=0.2, gamma=2):\n        super(SoftMarginFocalLoss, self).__init__()\n        self.gamma = gamma\n        self.margin = margin\n                \n        self.weight_pos = 2\n        self.weight_neg = 1\n    \n    def forward(self, inputs, targets):\n        em = np.exp(self.margin)\n        \n        log_pos = -F.logsigmoid(inputs)\n        log_neg = -F.logsigmoid(-inputs)\n        \n        log_prob = targets*log_pos + (1-targets)*log_neg\n        prob = torch.exp(-log_prob)\n        margin = torch.log(em + (1-em)*prob)\n        \n        weight = targets*self.weight_pos + (1-targets)*self.weight_neg\n        loss = self.margin + weight * (1 - prob) ** self.gamma * log_prob\n        \n        loss = loss.mean()\n        \n        return loss","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training\n\nOur Custom Training loop","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def train_fn(dataloader,model,criterion,optimizer,device,scheduler,epoch):\n    model.train()\n    \n    train_targets=[]\n    train_outputs=[]\n    \n    for bi,d in enumerate(dataloader):\n        features = d['features']\n        target = d['target']\n        \n        features = features.to(device, dtype=torch.float)\n        target = target.to(device, dtype=torch.float)\n        \n        optimizer.zero_grad()\n        \n        output,_ = model(features)\n        \n        loss = criterion(output,target)\n        loss.backward()\n        optimizer.step()\n        \n        if scheduler is not None:\n            scheduler.step()\n            \n        output  = 1 - F.softmax(output,dim=-1).cpu().detach().numpy()[:,0]  \n        \n        train_targets.extend(target.cpu().detach().numpy().argmax(axis=1).astype(int).tolist())\n        train_outputs.extend(output)\n            \n        \n    return loss.item(),train_outputs,train_targets","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Evaluation\n\nCustom Evaluation loop","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def eval_fn(data_loader,model,criterion,device):\n    \n    fin_targets=[]\n    fin_outputs=[]\n    \n    model.eval()\n    with torch.no_grad():\n        \n        for bi, d in enumerate(data_loader):\n            features = d[\"features\"]\n            target = d[\"target\"]\n\n            features = features.to(device, dtype=torch.float)\n            target = target.to(device, dtype=torch.float)\n\n            outputs,_ = model(features)\n            \n            loss_eval = criterion(outputs,target)\n            \n            outputs  = 1 - F.softmax(outputs,dim=-1).cpu().detach().numpy()[:,0]  \n            \n            fin_targets.extend(target.cpu().detach().numpy().argmax(axis=1).astype(int).tolist())\n            fin_outputs.extend(outputs)\n            \n    return loss_eval.item(),fin_outputs,fin_targets","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Plotter\n\nFunction for plotting the losses and auc_scores for each fold ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def print_history(fold,history,num_epochs=EPOCHS):\n        plt.figure(figsize=(15,5))\n        \n        plt.plot(\n            np.arange(num_epochs),\n            history['train_history_auc'],\n            '-o',\n            label='Train AUC',\n            color='#ff7f0e'\n        )\n        \n        plt.plot(\n            np.arange(num_epochs),\n            history['val_history_auc'],\n            '-o',\n            label='Val AUC',\n            color='#1f77b4'\n        )\n        \n        x = np.argmax(history['val_history_auc'])\n        y = np.max(history['val_history_auc'])\n        \n        xdist = plt.xlim()[1] - plt.xlim()[0]\n        ydist = plt.ylim()[1] - plt.ylim()[0]\n        \n        plt.scatter(x, y, s=200, color='#1f77b4')\n        \n        plt.text(\n            x-0.03*xdist,\n            y-0.13*ydist,\n            'max auc\\n%.2f'%y,\n            size=14\n        )\n        \n        plt.ylabel('AUC', size=14)\n        plt.xlabel('Epoch', size=14)\n        \n        plt.legend(loc=2)\n        \n        plt2 = plt.gca().twinx()\n        \n        plt2.plot(\n            np.arange(num_epochs),\n            history['train_history_loss'],\n            '-o',\n            label='Train Loss',\n            color='#2ca02c'\n        )\n        \n        plt2.plot(\n            np.arange(num_epochs),\n            history['val_history_loss'],\n            '-o',\n            label='Val Loss',\n            color='#d62728'\n        )\n        \n        x = np.argmin(history['val_history_loss'])\n        y = np.min(history['val_history_loss'])\n        \n        ydist = plt.ylim()[1] - plt.ylim()[0]\n        \n        plt.scatter(x, y, s=200, color='#d62728')\n        \n        plt.text(\n            x-0.03*xdist, \n            y+0.05*ydist, \n            'min loss', \n            size=14\n        )\n        \n        plt.ylabel('Loss', size=14)\n        \n        plt.title(f'FOLD {fold + 1}',size=18)\n        \n        plt.legend(loc=3)\n        plt.show()  ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Engine\n\nEngine where we unite everything","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def run(fold):\n    \n    df_train = df[df.fold != fold]\n    df_valid = df[df.fold == fold]\n    \n    # Defining DataSet\n    train_dataset = MelanomaDataset(\n        df_train[features].values,\n        df_train[target].values\n    )\n  \n    \n    valid_dataset = MelanomaDataset(\n        df_valid[features].values,\n        df_valid[target].values\n    )\n    \n    # Defining DataLoader with BalanceClass Sampler\n    train_loader = DataLoader(\n        train_dataset,\n        #sampler=BalanceClassSampler(\n         #   labels=train_dataset.get_targets(), \n          #  mode=\"downsampling\",\n        #),\n        batch_size=BATCH_SIZE,\n        pin_memory=True,\n        drop_last=True,\n        num_workers=4\n    )\n    \n    \n    valid_loader = torch.utils.data.DataLoader(\n        valid_dataset,\n        batch_size=BATCH_SIZE,\n        num_workers=4,\n        shuffle=False,\n        pin_memory=True,\n        drop_last=False,\n    )\n    \n    # Defining Device\n    device = torch.device(\"cuda\")\n    \n    # Defining Model for specific fold\n    model = CustomTabnet(input_dim = len(features), \n                         output_dim = 2,\n                         n_d=32, \n                         n_a=32,\n                         n_steps=4, \n                         gamma=1.6,\n                         n_independent=2,\n                         n_shared=2,\n                         momentum=0.02,\n                         mask_type=\"sparsemax\")\n    \n    model.to(device)\n    \n    #DEfining criterion\n    criterion = SoftMarginFocalLoss()\n    criterion.to(device)\n        \n    # Defining Optimizer with weight decay to params other than bias and layer norms\n    param_optimizer = list(model.named_parameters())\n    no_decay = [\"bias\", \"LayerNorm.bias\", \"LayerNorm.weight\"]\n    optimizer_parameters = [\n        {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)], 'weight_decay': 0.001},\n        {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)], 'weight_decay': 0.0},\n            ]  \n    \n    optimizer = torch.optim.Adam(optimizer_parameters, lr=LR)\n    \n    # Defining LR SCheduler\n    scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='max',\n                                                           factor=0.1, patience=10, verbose=True, \n                                                           threshold=0.0001, threshold_mode='rel',\n                                                           cooldown=0, min_lr=0, eps=1e-08)\n    #DEfining Early Stopping Object\n    es = EarlyStopping(patience=patience,verbose=False)\n    \n    # History dictionary to store everything\n    history = {\n            'train_history_loss': [],\n            'train_history_auc': [],\n            'val_history_loss': [],\n            'val_history_auc': [],\n        }\n        \n    # THE ENGINE LOOP    \n    tk0 = tqdm(range(EPOCHS), total=EPOCHS)\n    for epoch in tk0:\n        train_loss,train_out,train_targets = train_fn(train_loader, model,criterion, optimizer, device,scheduler=None,epoch=epoch)\n        \n        val_loss,outputs, targets = eval_fn(valid_loader, model, criterion,device)\n        \n        train_auc = roc_auc_score(train_targets, train_out)\n        auc_score = roc_auc_score(targets, outputs)\n        \n        scheduler.step(auc_score)\n        \n        tk0.set_postfix(Train_Loss=train_loss,Train_AUC_SCORE = train_auc,Valid_Loss = val_loss,Valid_AUC_SCORE = auc_score)\n        \n        history['train_history_loss'].append(train_loss)\n        history['train_history_auc'].append(train_auc)\n        history['val_history_loss'].append(val_loss)\n        history['val_history_auc'].append(auc_score)\n\n        es(auc_score,model,f'model_{fold}.bin')\n        \n        if es.early_stop:\n            print('Maximum Patience {} Reached , Early Stopping'.format(patience))\n            break   \n            \n    print_history(fold,history,num_epochs=epoch+1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"run(fold=0)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"run(fold=1)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"run(fold=2)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"run(fold=3)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"run(fold=4)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Inference","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test =pd.read_csv('../input/fe-using-only-competition-data-melanoma/test.csv')\ndf_test['anatom_site_general_challenge'].fillna('unknown',inplace=True)\ndf_test['target'] = 0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Defining Categorical variables and their Indexes, embedding dimensions , number of classes each have \ndf_test.drop(['image_name'],axis=1,inplace=True)\ntarget = 'target'\nunused_feat = ['patient_id','fold','center']\nfeatures = [ col for col in df_test.columns if col not in unused_feat+[target]] \n\nfor col in df_test.columns[df_test.dtypes == object]:\n    if col not in unused_feat:\n        print(col, df_test[col].nunique())\n        pkl_file = open(f'{col}_encoder.pkl', 'rb')\n        l_enc = pickle.load(pkl_file) \n        df_test[col] = l_enc.transform(df_test[col].values)\n        pkl_file.close()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def load_model():\n    \n    models = []\n    paths = glob.glob('model*')\n    \n    for path in tqdm(paths,total=len(paths)):\n                \n        model = CustomTabnet(input_dim = len(features), \n                         output_dim = 2,\n                         n_d=32, \n                         n_a=32,\n                         n_steps=4, \n                         gamma=1.6,\n                         n_independent=2,\n                         n_shared=2,\n                         momentum=0.02,\n                         mask_type=\"sparsemax\")\n        \n        model.to(device)\n        loader = torch.load(path)\n        model.load_state_dict(loader)\n        \n        models.append(model)\n        \n    return models","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"models = load_model()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def make_prediction(data_loader):\n    predictions = np.zeros((len(df_test),FOLDS))\n    for i,model in enumerate(models):\n        \n        fin_outputs=[]\n        \n        model.eval()\n        with torch.no_grad():\n            \n            for bi, d in enumerate(data_loader):\n                features = d[\"features\"]\n                target = d[\"target\"]\n\n                features = features.to(device, dtype=torch.float)\n\n                outputs,_ = model(features)\n\n                outputs  = 1 - F.softmax(outputs,dim=-1).cpu().detach().numpy()[:,0]                \n        \n                fin_outputs.extend(outputs)\n        \n        predictions[:,i] = fin_outputs\n    \n    return predictions","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_dataset = MelanomaDataset(\n        df_test[features].values,\n        df_test[target].values\n    )\n\ntest_loader = torch.utils.data.DataLoader(\n        test_dataset,\n        batch_size=BATCH_SIZE,\n        num_workers=4,\n        shuffle=False,\n        pin_memory=True,\n        drop_last=False,\n    )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred = make_prediction(test_loader)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Writing Submission File","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"pred = pred.mean(axis=-1)\npred","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ss = pd.read_csv('../input/siim-isic-melanoma-classification/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ss['target'] = pred","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ss.to_csv('submission.csv',index=False)\nss.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Thank you all for reading my Notebook","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}