{"cells":[{"metadata":{},"cell_type":"markdown","source":"This notebook is an implementation of using a MLP models to ensemble output from different folds of predictions of same model. I refer to this method as `MetaClassifier` in this notebook\n\n**Idea**\n- Generally the predictions from different folds is averaged to get the final predicition\n- Here we train a separate model (Plain Feedforward NNs) to calculate the final prediction from different fold predictions\n- Input is prediction probabilities from different folds, output is target label\n- Entire dataset is Stratified split into train, validation and test sets \n- A 3-layer NN with Batchnorm and droput is trained using SGD optimizer, CosineAnnealingWarmRestarts lr_scheduler\n- The results on test set (overall accuracy and confusion matrix) of the averaging method and the MetaClassifier approach\n\n**Notebook reference**\n- Much of notebook is inspired from this [blog post](https://towardsdatascience.com/pytorch-tabular-multiclass-classification-9f8211a123ab)\n\n***Feedback is most welcome***"},{"metadata":{},"cell_type":"markdown","source":"## Library imports"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","execution":{"iopub.execute_input":"2020-11-20T07:49:33.361273Z","iopub.status.busy":"2020-11-20T07:49:33.360454Z","iopub.status.idle":"2020-11-20T07:49:38.340877Z","shell.execute_reply":"2020-11-20T07:49:38.340239Z"},"papermill":{"duration":5.000532,"end_time":"2020-11-20T07:49:38.340989","exception":false,"start_time":"2020-11-20T07:49:33.340457","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"# basic imports\nimport os\nimport numpy as np\nimport pandas as pd\nimport random\nimport itertools\nfrom tqdm.notebook import tqdm\nimport math\nfrom scipy.special import softmax\n\n\n# DL library imports\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torchvision\nimport torch.optim as optim\nfrom torch.utils.data import Dataset, DataLoader, WeightedRandomSampler\nfrom torch.optim.lr_scheduler import ReduceLROnPlateau, CosineAnnealingWarmRestarts\n\n# metrics calculation\nfrom sklearn.metrics import accuracy_score, confusion_matrix\nfrom sklearn.model_selection import KFold, StratifiedKFold, train_test_split\n\n# basic plotting library\nimport matplotlib.pyplot as plt\nimport seaborn as sns","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Constants"},{"metadata":{"trusted":true},"cell_type":"code","source":"index_label_map = {\n                0: \"CBB\",    # Cassava Bacterial Blight \n                1: \"CBSD\",   # Cassava Brown Streak Disease \n                2: \"CGM\",    # Cassava Green Mottle\n                3: \"CMD\",    # Cassava Mosaic Disease \n                4: \"Healthy\"\n                }\n\nclass_names = [value for key,value in index_label_map.items()]\nprint(class_names)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"SEED = 42\nN_FOLDS = 5 \nN_EPOCHS = 50\nBATCH_SIZE = 256\nTRAINVAL_TEST_SPLIT = 0.2\nTRAIN_VAL_SPLIT = 0.2\nLEARNING_RATE = 1e-3\nDROPOUT_PROB = 0.4\nN_FEATURES = 25\nN_CLASSES = len(class_names)\nPRINT_EVERY = 5","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Helpter function"},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_confusion_matrix(cm, classes, normalize=False, title='Confusion matrix', cmap=plt.cm.Blues):\n    if normalize:\n        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n        print(\"Normalized confusion matrix\")\n    else:\n        print('Confusion matrix, without normalization')\n\n    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n    plt.title(title)\n    tick_marks = np.arange(len(classes))\n    plt.xticks(tick_marks, classes, rotation=45)\n    plt.yticks(tick_marks, classes)\n\n    fmt = '.2f' if normalize else 'd'\n    thresh = cm.max() / 2.\n    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n        plt.text(j, i, format(cm[i, j], fmt), horizontalalignment=\"center\", color=\"white\" if cm[i, j] > thresh else \"black\")\n\n    plt.tight_layout()\n    plt.ylabel('True label')\n    plt.xlabel('Predicted label')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def seed_reproducer(seed=2020):\n    random.seed(seed)\n    os.environ[\"PYTHONHASHSEED\"] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    if torch.cuda.is_available():\n        torch.cuda.manual_seed(seed)\n        torch.cuda.manual_seed_all(seed)\n        torch.backends.cudnn.deterministic = True\n        torch.backends.cudnn.benchmark = False\n        torch.backends.cudnn.enabled = True\nseed_reproducer(SEED)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_class_distribution(obj):\n    count_dict = {}\n    for label in class_names:\n        count_dict[label] = 0\n    \n    for i in obj:\n        if i in range(N_CLASSES): \n            count_dict[class_names[i]] += 1\n    return count_dict","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Creating data"},{"metadata":{"trusted":true},"cell_type":"code","source":"val_folder = '../input/se-resnext50-model-predictions-and-labels/'\nmodel_name = 'se_rexnext50'\n\npreds = []\nlabels = []\nfor fold in range(N_FOLDS):\n    preds.append(np.load(f'{val_folder}{model_name}_fold{fold}_preds.npy'))\n    labels.append(np.load(f'{val_folder}{model_name}_fold{fold}_labels.npy'))\n\n# labels is same for all folds as input is not shuffled    \nlabels = np.array(labels[0])\n\n# stacking the predictions\npreds = np.array(preds)\npreds = np.moveaxis(preds, 0, -1)\npreds = preds.reshape((len(preds), -1))\n\n# verify input and output shape\nX = preds\ny = np.argmax(labels, axis = 1)\nprint(f'X shape = {X.shape}, y shape = {y.shape}')\n\n# Make sure the input limits are scaled\n#print(f'X max value is {X.max(axis=0)}, X min value is {X.min(axis=0)}')\n\n# output label distribution\nsns.countplot(y)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Split train into train and val sets"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Split into train+val and test\nX_trainval, X_test, y_trainval, y_test = train_test_split(X, y, test_size=TRAINVAL_TEST_SPLIT, stratify=y, random_state=SEED)\n# Split train into train-val\nX_train, X_val, y_train, y_val = train_test_split(X_trainval, y_trainval, test_size=TRAIN_VAL_SPLIT, stratify=y_trainval, random_state=SEED)\n\nfig, axes = plt.subplots(nrows=1, ncols=3, figsize=(15, 5))\nsns.barplot(data = pd.DataFrame.from_dict([get_class_distribution(y_train)]).melt(), x = \"value\", y=\"variable\", hue=\"value\",ax=axes[0]).set_title('Class Distribution in Train Set')\nsns.barplot(data = pd.DataFrame.from_dict([get_class_distribution(y_val)]).melt(),   x = \"value\", y=\"variable\", hue=\"value\",ax=axes[1]).set_title('Class Distribution in Val Set')\nsns.barplot(data = pd.DataFrame.from_dict([get_class_distribution(y_test)]).melt(),  x = \"value\", y=\"variable\", hue=\"value\",ax=axes[2]).set_title('Class Distribution in Test Set')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"class ClassifierDataset(Dataset):\n    def __init__(self, X_data, y_data):\n        self.X_data = X_data\n        self.y_data = y_data\n        \n    def __getitem__(self, index):\n        return self.X_data[index], self.y_data[index]\n        \n    def __len__ (self):\n        return len(self.X_data)\n\ntrain_dataset = ClassifierDataset(torch.from_numpy(X_train).float(), torch.from_numpy(y_train).long())\nval_dataset = ClassifierDataset(torch.from_numpy(X_val).float(), torch.from_numpy(y_val).long())\ntest_dataset = ClassifierDataset(torch.from_numpy(X_test).float(), torch.from_numpy(y_test).long())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Weighted class samplers and Dataloaders"},{"metadata":{"trusted":true},"cell_type":"code","source":"target_list = []\nfor _, t in train_dataset:\n    target_list.append(t)\n    \ntarget_list = torch.tensor(target_list)\ntarget_list = target_list[torch.randperm(len(target_list))]\n\nclass_count = [i for i in get_class_distribution(y_train).values()]\nclass_weights = 1./torch.tensor(class_count, dtype=torch.float) \nprint(class_weights)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class_weights_all = class_weights[target_list]\nweighted_sampler = WeightedRandomSampler(\n    weights=class_weights_all,\n    num_samples=len(class_weights_all),\n    replacement=True\n)\ntrain_loader = DataLoader(dataset=train_dataset, batch_size=BATCH_SIZE) #sampler=weighted_sampler\nval_loader = DataLoader(dataset=val_dataset, batch_size=1)\ntest_loader = DataLoader(dataset=test_dataset, batch_size=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Metaclassifier"},{"metadata":{"trusted":true},"cell_type":"code","source":"class MetaClassifier(nn.Module):\n    def __init__(self, num_feature, num_class):\n        super(MetaClassifier, self).__init__()\n        self.layer_1 = nn.Linear(num_feature, 32)\n        self.layer_2 = nn.Linear(32, 16)\n        self.layer_3 = nn.Linear(16, 8)\n        self.layer_out = nn.Linear(8, num_class) \n        \n        self.relu = nn.ReLU()\n        self.dropout = nn.Dropout(p=DROPOUT_PROB)\n        self.batchnorm1 = nn.BatchNorm1d(32)\n        self.batchnorm2 = nn.BatchNorm1d(16)\n        self.batchnorm3 = nn.BatchNorm1d(8)\n        \n    def forward(self, x):\n        #print(x.shape)\n        x = self.layer_1(x)\n        x = self.batchnorm1(x)\n        x = self.relu(x)\n        \n        x = self.layer_2(x)\n        x = self.batchnorm2(x)\n        x = self.relu(x)\n        x = self.dropout(x)\n        \n        x = self.layer_3(x)\n        x = self.batchnorm3(x)\n        x = self.relu(x)\n        x = self.dropout(x)\n        \n        x = self.layer_out(x)\n        return x","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## device, optimizer"},{"metadata":{"trusted":true},"cell_type":"code","source":"device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\nprint(device)\n\nmodel = MetaClassifier(num_feature = N_FEATURES, num_class=N_CLASSES)\nmodel.to(device)\n#print(model)\ncriterion = nn.CrossEntropyLoss(weight=class_weights.to(device))\noptimizer = torch.optim.SGD(model.parameters(), lr=LEARNING_RATE, momentum=0.9)\n#scheduler = ReduceLROnPlateau(optimizer, 'min', patience=3, threshold=1e-4)\nscheduler = CosineAnnealingWarmRestarts(optimizer, T_0=5, T_mult=1, eta_min=1e-5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Evaluation metric function"},{"metadata":{"trusted":true},"cell_type":"code","source":"def multi_acc(y_pred, y_test):\n    y_pred_softmax = torch.softmax(y_pred, dim = 1)\n    y_pred_tags = torch.argmax(y_pred_softmax, dim = 1)    \n    #_, y_pred_tags = torch.argmax(y_pred_softmax, dim = 1)    \n    correct_pred = (y_pred_tags == y_test).float()\n    acc = correct_pred.sum() / len(correct_pred)\n    acc = torch.round(acc) * 100\n    return acc\n\naccuracy_stats = { 'train': [], \"val\": [] }\nloss_stats = { 'train': [], \"val\": [] }\nbest_val_acc = 0.0\nbest_epoch = 0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Begin training.\")\niters = len(train_loader)\nfor e in tqdm(range(1, N_EPOCHS+1)):\n    ## TRAINING\n    train_epoch_loss = 0\n    train_epoch_acc = 0\n    model.train()\n    for i, (X_train_batch, y_train_batch) in enumerate(train_loader):\n        X_train_batch, y_train_batch = X_train_batch.to(device), y_train_batch.to(device)\n        optimizer.zero_grad()\n        \n        # forward pass\n        y_train_pred = model(X_train_batch)\n        train_loss = criterion(y_train_pred, y_train_batch)\n        train_acc = multi_acc(y_train_pred, y_train_batch)\n        \n        # backward pass\n        train_loss.backward()\n        optimizer.step()\n        scheduler.step(e + i / iters)\n        \n        # log results\n        train_epoch_loss += train_loss.item()\n        train_epoch_acc += train_acc.item()\n        \n    ## VALIDATION    \n    with torch.no_grad():\n        val_epoch_loss = 0\n        val_epoch_acc = 0\n        model.eval()\n        \n        for X_val_batch, y_val_batch in val_loader:\n            X_val_batch, y_val_batch = X_val_batch.to(device), y_val_batch.to(device)    \n            y_val_pred = model(X_val_batch)\n            val_loss = criterion(y_val_pred, y_val_batch)\n            val_acc = multi_acc(y_val_pred, y_val_batch)\n            \n            # log results\n            val_epoch_loss += val_loss.item()\n            val_epoch_acc += val_acc.item()\n    #scheduler.step(val_loss.item())\n        \n    loss_stats['train'].append(train_epoch_loss/len(train_loader))\n    loss_stats['val'].append(val_epoch_loss/len(val_loader))\n    accuracy_stats['train'].append(train_epoch_acc/len(train_loader))\n    accuracy_stats['val'].append(val_epoch_acc/len(val_loader))\n    \n    if((val_epoch_acc/len(val_loader)) > best_val_acc):\n        torch.save(model.state_dict(), f'MetaClassifier_epoch{e}.pth')\n        best_val_acc = (val_epoch_acc/len(val_loader))\n        best_epoch = e\n    \n    if (e % PRINT_EVERY) == 0:\n        print(f'Epoch {e+0:3}: | Train Loss: {train_epoch_loss/len(train_loader):.5f} | Val Loss: {val_epoch_loss/len(val_loader):.5f} | Train Acc: {train_epoch_acc/len(train_loader):.3f}| Val Acc: {val_epoch_acc/len(val_loader):.3f}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'Best val_acc occured at epoch : {best_epoch}, val_acc : {best_val_acc}')\n#model.load_state_dict(torch.load('./MetaClassifier_epoch10.pth', map_location=device))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Plot results"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create dataframes\ntrain_val_acc_df = pd.DataFrame.from_dict(accuracy_stats).reset_index().melt(id_vars=['index']).rename(columns={\"index\":\"epochs\"})\ntrain_val_loss_df = pd.DataFrame.from_dict(loss_stats).reset_index().melt(id_vars=['index']).rename(columns={\"index\":\"epochs\"})\n# Plot the dataframes\nfig, axes = plt.subplots(nrows=1, ncols=2, figsize=(20,7))\nsns.lineplot(data=train_val_acc_df, x = \"epochs\", y=\"value\", hue=\"variable\",  ax=axes[0]).set_title('Train-Val Accuracy/Epoch')\nsns.lineplot(data=train_val_loss_df, x = \"epochs\", y=\"value\", hue=\"variable\", ax=axes[1]).set_title('Train-Val Loss/Epoch')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Test set results"},{"metadata":{},"cell_type":"markdown","source":"## **MetaClassifier performance**"},{"metadata":{"trusted":true},"cell_type":"code","source":"model.load_state_dict(torch.load(f'MetaClassifier_epoch{best_epoch}.pth',map_location=device))\n\ny_pred_list = []\nwith torch.no_grad():\n    model.eval()\n    for X_batch, _ in test_loader:\n        X_batch = X_batch.to(device)\n        y_test_pred = model(X_batch)        \n        y_pred_softmax = torch.softmax(y_test_pred, dim = 1)\n        y_pred_tags = torch.argmax(y_pred_softmax, dim = 1)    \n        y_pred_list.append(y_pred_tags.cpu().numpy())\ny_pred_list = [a.squeeze().tolist() for a in y_pred_list]\nprint(f'MetaClassifier accuracy = {100.0 * accuracy_score(y_test, y_pred_list)}')\n\n# confusion matrix \nMetaClassifier_cm = confusion_matrix(y_test, y_pred_list)\nplot_confusion_matrix(MetaClassifier_cm, class_names, normalize=True, title='MetaClassifier Confusion matrix')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def accuracy_avg_method(X_pred, y_true):\n    fold_split = np.array(np.split(X_pred, 5, axis=1))\n    classwise_avg = np.average(fold_split, axis=2)\n    y_preds_softmax = softmax(classwise_avg, axis=0)\n    y_pred_tags = np.argmax(y_preds_softmax, axis=0)\n    \n    print(f'avgmethod accuracy = {100.0 * accuracy_score(y_test, y_pred_tags)}')\n    avgmethod_cm = confusion_matrix(y_test, y_pred_tags)\n    plot_confusion_matrix(avgmethod_cm, class_names, normalize=True, title='avgmethod Confusion matrix')\n\naccuracy_avg_method(X_test, y_test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Results\n\n> **The MetaClassifier approach seems to have a lower overall accuracy, but is more accuracy in predicting the minority classes - label0(CBB) and label4(Healthy). This could be an option for ensembling different folds of the same model as well as ensembling different model outputs.**"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}