{"cells":[{"metadata":{},"cell_type":"markdown","source":"# SUMMARY\n\nThis notebook reproduces our best submission that scores **0.9016** on the private LB and reaches the **14th place**. The notebook implements a stacking ensemble of 33+2 CNN and ViT models (yes, they all fit into the 9-hour inference limit!) using pretrained weights saved as Kaggle datasets.\n- Complete codes and notebooks for base models and stacking are available [in my GitHub repo](https://github.com/kozodoi/Kaggle_Leaf_Disease_Classification).\n- A detailed summary of our solution is published [in this discussion topic](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/220751).\n\n![cassava](https://i.postimg.cc/d1dcZ6Zv/cassava.png)"},{"metadata":{},"cell_type":"markdown","source":"# PARAMETERS"},{"metadata":{"trusted":true},"cell_type":"code","source":"####### NOTEBOOK CONFIGURATION\n\nclass CFG:\n    \n    # environment\n    environment = 'kaggle'  # environment (kaggle, colab)\n    device      = 'GPU'     # device (CPU, GPU, TPU)\n    num_workers = 2         # no. cores\n    debug       = False     # debug runs inference on a subset of training images\n    \n    # randomness\n    seed = 13353  # random state\n\n    # inference\n    batch_size = 25  # no. images per batch\n    num_tta    = 4   # no. TTA flips (between 1 and 8)\n    one_fold   = 2   # use specific fold for all models\n   \n    # stacking parameters\n    lgb_folds       = 5    # no. folds for stacking\n    lgb_stop_rounds = 200  # no. early stopping rounds\n    lgb_params      = {'objective':         'multiclass',\n                       'metrics':           'multi_error',\n                       'n_estimators':      10000,\n                       'learning_rate':     0.01,\n                       'num_leaves':        10,\n                       'max_depth':         5,\n                       'min_child_samples': 20,\n                       'subsample':         0.3,\n                       'colsample_bytree':  0.3,\n                       'reg_alpha':         0.15,\n                       'reg_lambda':        0.15,\n                       'silent':            True,\n                       'verbosity':         -1,\n                       'n_jobs' :           2,\n                       'random_state':      13353}\n    # paths\n    data_path = '/kaggle/input/cassava-leaf-disease-classification/'\n    \n    # list of models\n    models = ['/kaggle/input/pytorch-v108/',\n              '/kaggle/input/pytorch-v89/',\n              '/kaggle/input/pytorch-v87/',\n              '/kaggle/input/pytorch-v107/',\n              '/kaggle/input/pytorch-v115/',\n              '/kaggle/input/pytorch-v138/',\n              '/kaggle/input/pytorch-v136/',\n              '/kaggle/input/pytorch-v117/',\n              '/kaggle/input/pytorch-v100/',\n              '/kaggle/input/pytorch-v119/',\n              '/kaggle/input/pytorch-v114/',\n              '/kaggle/input/pytorch-v125/',\n              '/kaggle/input/pytorch-v122/',\n              '/kaggle/input/pytorch-v120/',\n              '/kaggle/input/pytorch-v86/',\n              '/kaggle/input/pytorch-v84/',\n              '/kaggle/input/pytorch-v116/',\n              '/kaggle/input/pytorch-v124/',\n              '/kaggle/input/pytorch-v127/',\n              '/kaggle/input/pytorch-v118/',\n              '/kaggle/input/pytorch-v85/',\n              '/kaggle/input/pytorch-v113/',\n              '/kaggle/input/pytorch-v131/',\n              '/kaggle/input/pytorch-v1420/',\n              '/kaggle/input/pytorch-v121/',\n              '/kaggle/input/pytorch-v135/',\n              '/kaggle/input/pytorch-v106/',\n              '/kaggle/input/pytorch-v133/',\n              '/kaggle/input/pytorch-v83/',\n              '/kaggle/input/pytorch-v126/',\n              '/kaggle/input/pytorch-v130/',\n              '/kaggle/input/pytorch-v109/',\n              '/kaggle/input/pytorch-v129/']\n\n    # special models (imagenet + binary)\n    special_models = ['/kaggle/input/pytorch-basic-v0/',\n                      '/kaggle/input/pytorch-binary-v0/']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"During training we saved configuration file for each base model to make it easier to import their parameters and image sizes during inference. Below we load all model configurations in a list."},{"metadata":{"trusted":true},"cell_type":"code","source":"####### LOAD MODEL CONFIGURATIONS\n\nimport pandas as pd\n\nCFGs = []\n\nfor model in CFG.models:\n    \n    # load params\n    df_cfg   = pd.read_csv(model + 'tab_configuration.csv', index_col = 0).T\n    dict_cfg = df_cfg.reset_index(drop = True).astype(object).to_dict('records')[0]\n    \n    # adjust params saved incorrectly in a dict\n    dict_cfg['weights']     = 'empty'\n    dict_cfg['normalize']   = dict_cfg['normalize'] == 'imagenet'\n    dict_cfg['image_size']  = int(dict_cfg['image_size'])\n    dict_cfg['num_classes'] = int(dict_cfg['num_classes'])\n    if 'attention' not in dict_cfg.keys():\n        dict_cfg['attention'] = False\n    else:\n        dict_cfg['attention'] = dict_cfg['attention'] == 'True'\n    if dict_cfg['backbone'] == 'deit_base_patch16_384':\n        dict_cfg['backbone'] = 'vit_deit_base_patch16_384'\n    \n    # append params\n    CFGs.append(dict_cfg)\n    \nprint('Numer of models:', len(CFGs))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Apart from the 33 multi-class classification models, there are two special models:\n- ImageNet classifier that predicts ImageNet classes\n- binary cassava sickness classifier"},{"metadata":{"trusted":true},"cell_type":"code","source":"####### LOAD SPECIAL MODEL CONFIGURATIONS\n\nspecial_CFGs = []\n\nfor model in CFG.special_models:\n    \n    # load params\n    df_cfg   = pd.read_csv(model + 'tab_configuration.csv', index_col = 0).T\n    dict_cfg = df_cfg.reset_index(drop = True).astype(object).to_dict('records')[0]\n    \n    # adjust params saved incorrectly in a dict\n    dict_cfg['weights']     = 'empty'\n    dict_cfg['normalize']   = dict_cfg['normalize'] == 'imagenet'\n    dict_cfg['image_size']  = int(dict_cfg['image_size'])\n    dict_cfg['num_classes'] = int(dict_cfg['num_classes'])\n    if 'attention' not in dict_cfg.keys():\n        dict_cfg['attention'] = False\n    else:\n        dict_cfg['attention'] = dict_cfg['attention'] == 'True'\n    if dict_cfg['backbone'] == 'deit_base_patch16_384':\n        dict_cfg['backbone'] = 'vit_deit_base_patch16_384'\n        \n    # append params\n    special_CFGs.append(dict_cfg)\n    \nprint('Numer of special models:', len(special_CFGs))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# PREPARATIONS"},{"metadata":{"trusted":true},"cell_type":"code","source":"####### PACKAGES\n\nimport numpy as np\nimport pandas as pd\n\nimport torch\nimport torch.nn as nn\nimport torch.optim as optim\nfrom torch.utils.data import Dataset, DataLoader\nfrom torch.utils.data.sampler import RandomSampler, SequentialSampler\n\nimport albumentations as A\nfrom albumentations.pytorch import ToTensorV2\n\nfrom PIL import Image, ImageFile\nImageFile.LOAD_TRUNCATED_IMAGES = True\nimport cv2\n\nfrom sklearn.model_selection import StratifiedKFold\nimport lightgbm as lgb\n\nfrom tqdm import tqdm\n\nimport random\nimport time\nimport sys\nimport os\n\nimport gc\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n%matplotlib inline\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\nsys.path.append('../input/timm-pytorch-image-models/pytorch-image-models-master')\nimport timm","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"####### TRAINING DEVICE\n\nif CFG.device == 'TPU':\n    print('Training on TPU...')\n    \nif CFG.device == 'GPU':\n    print('Training on GPU...')\n    device = torch.device('cuda:0')\n\nif CFG.device == 'CPU':\n    print('Training on CPU...')\n    device = torch.device('cpu') ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"####### RANDOMNESS\n\ndef seed_everything(seed = 23):\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    random.seed(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.cuda.manual_seed_all(seed)\n    torch.backends.cudnn.deterministic = True\n    print('Setting random seed to {}...'.format(seed))\n    \nseed_everything(CFG.seed)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"###### TIMER\n\nnotebook_start = time.time()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# DATA PREP\n\nTo fit into the 9-hour limit, we tested our submissions on the training data using the debug mode. Since the hidden test included 15k images, we were running inference on 10% (1500) training images to get an estimate of the submission time."},{"metadata":{"trusted":true},"cell_type":"code","source":"####### DATA PREPARATION\n\n# import data\ndf = pd.read_csv(CFG.data_path + 'sample_submission.csv')\n\n# num classes\nCFG.num_classes = 5\n\n# debug mode\nif CFG.debug:\n    df = pd.read_csv(CFG.data_path + 'train.csv').head(int(15000 * 0.01))\n\nprint(df.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"####### DATASET\n\nclass LeafData(Dataset):\n    \n    # initialization\n    def __init__(self, \n                 data, \n                 directory, \n                 transform = None, \n                 labeled   = False):\n        self.data      = data\n        self.directory = directory\n        self.transform = transform\n        self.labeled   = labeled\n        \n    # length\n    def __len__(self):\n        return len(self.data)\n    \n    # get item  \n    def __getitem__(self, idx):\n        \n        # import\n        path  = os.path.join(self.directory, self.data.iloc[idx]['image_id'])\n        image = cv2.imread(path)\n        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n            \n        # augmentations\n        if self.transform is not None:\n            image = self.transform(image = image)['image']\n        \n        # output\n        if self.labeled:\n            labels = torch.tensor(self.data.iloc[idx]['label']).long()\n            return image, labels\n        else:\n            return image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"####### CHECK OOF PERFORMANCE\n\n# load preds\nfor m in CFG.models:\n\n    tmp_train_preds         = pd.read_csv(m + '/oof.csv')\n    tmp_train_preds.columns = ['image_id', 'label', 'source', 'fold'] + [str(m) + str(c) for c in ['c0', 'c1', 'c2', 'c3', 'c4']]\n\n    if m == CFG.models[0]:\n        train_preds = tmp_train_preds        \n    else:\n        train_preds = train_preds.merge(tmp_train_preds[['image_id'] + [str(m) + str(c) for c in ['c0', 'c1', 'c2', 'c3', 'c4']]], how = 'left', on = 'image_id')\n\n# display performance \nweights = []\nfor model_idx, m in enumerate(CFG.models):\n    weights.append((np.argmax(train_preds.filter(like = m).values, axis = 1) == train_preds['label']).sum() / len(train_preds))\n    print('- model {}: accuracy = {:.4f}'.format(model_idx + 1, weights[model_idx]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# AUGMENTATIONS"},{"metadata":{"trusted":true},"cell_type":"code","source":"####### AUGMENTATIONS\n\ndef define_augs(CFG, image_size = None):\n\n    # image size\n    if image_size is None:\n        image_size = CFG['image_size']\n\n    # normalization\n    if CFG['normalize']:\n        CFG['pixel_mean'] = (0.485, 0.456, 0.406)\n        CFG['pixels_std'] = (0.229, 0.224, 0.225)\n    else:\n        CFG['pixel_mean'] = (0, 0, 0)\n        CFG['pixels_std'] = (1, 1, 1)\n\n    # test augmentations\n    test_augs = A.Compose([A.SmallestMaxSize(max_size = image_size),\n                           A.CenterCrop(height = image_size, \n                                        width  = image_size),\n                           A.Normalize(mean = CFG['pixel_mean'],\n                                       std  = CFG['pixels_std']),\n                           ToTensorV2()\n                           ])\n    \n    # output\n    return test_augs","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"####### TTA HELPER FUNCTION\n\n'''\nBorrowed from https://github.com/haqishen/SIIM-ISIC-Melanoma-Classification-1st-Place-Solution\n'''\n\ndef get_tta_flips(img, i):\n\n    if i >= 4:\n        img = img.transpose(2, 3)\n    if i % 4 == 0:\n        return img\n    elif i % 4 == 1:\n        return img.flip(2)\n    elif i % 4 == 2:\n        return img.flip(3)\n    elif i % 4 == 3:\n        return img.flip(2).flip(3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"###### EXAMINE SAMPLE BATCH\n\nif len(df) == 1 or CFG.debug:\n\n    # dataset\n    test_dataset = LeafData(data      = df, \n                            directory = CFG.data_path + 'test_images/' if not CFG.debug else CFG.data_path + 'train_images/',\n                            transform = define_augs(CFGs[0]),\n                            labeled   = False)\n\n    # data loader\n    test_loader = torch.utils.data.DataLoader(test_dataset, \n                                              batch_size  = CFG.batch_size, \n                                              shuffle     = False, \n                                              num_workers = 0)\n\n    # display 1st image with TTA\n    for batch_idx, (inputs) in enumerate(test_loader):\n        fig = plt.figure(figsize = (14, 7))\n        for i in range(1):\n            for tta_idx in range(CFG.num_tta):\n                inputs_tta = get_tta_flips(inputs, tta_idx)\n                ax         = fig.add_subplot(2, 4, tta_idx + 1, xticks = [], yticks = [])     \n                plt.imshow(inputs_tta[i].numpy().transpose(1, 2, 0))\n        break\n\n    # clean up\n    del inputs, batch_idx","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# MODEL PREP"},{"metadata":{"trusted":true},"cell_type":"code","source":"####### ATTENTION MODULES\n\n'''\nBorrowed from https://www.kaggle.com/jy2tong/efficientnet-b2-soft-attention\n'''\n\nclass PAM_Module(nn.Module):\n    ''' Position attention module'''\n    #Ref from SAGAN\n    def __init__(self, in_dim):\n        super(PAM_Module, self).__init__()\n        self.chanel_in  = in_dim\n        self.query_conv = nn.Conv2d(in_channels=in_dim, out_channels=in_dim//8, kernel_size=1)\n        self.key_conv   = nn.Conv2d(in_channels=in_dim, out_channels=in_dim//8, kernel_size=1)\n        self.value_conv = nn.Conv2d(in_channels=in_dim, out_channels=in_dim, kernel_size=1)\n        self.gamma      = nn.Parameter(torch.zeros(1))\n\n    def forward(self, x):\n        '''\n            inputs :\n                x : input feature maps( B X C X H X W)\n            returns :\n                out : attention value + input feature\n                attention: B X (HxW) X (HxW)\n        '''\n        m_batchsize, C, height, width = x.size()\n        proj_query = self.query_conv(x).view(m_batchsize, -1, width*height).permute(0, 2, 1)\n        proj_key   = self.key_conv(x).view(m_batchsize, -1, width*height)\n        energy     = torch.bmm(proj_query, proj_key)\n        attention  = torch.softmax(energy, dim=-1)\n        proj_value = self.value_conv(x).view(m_batchsize, -1, width*height)\n\n        out = torch.bmm(proj_value, attention.permute(0, 2, 1))\n        out = out.view(m_batchsize, C, height, width)\n\n        out = self.gamma*out + x\n        return out\n\n\nclass CAM_Module(nn.Module):\n    ''' Channel attention module'''\n    def __init__(self, in_dim):\n        super(CAM_Module, self).__init__()\n        self.chanel_in = in_dim\n        self.gamma = nn.Parameter(torch.zeros(1))\n\n    def forward(self,x):\n        '''\n            inputs :\n                x : input feature maps( B X C X H X W)\n            returns :\n                out : attention value + input feature\n                attention: B X C X C\n        '''\n        m_batchsize, C, height, width = x.size()\n        proj_query = x.view(m_batchsize, C, -1)\n        proj_key   = x.view(m_batchsize, C, -1).permute(0, 2, 1)\n        energy     = torch.bmm(proj_query, proj_key)\n        energy_new = torch.max(energy, -1, keepdim=True)[0].expand_as(energy)-energy\n        attention  = torch.softmax(energy_new, dim=-1)\n        proj_value = x.view(m_batchsize, C, -1)\n\n        out = torch.bmm(attention, proj_value)\n        out = out.view(m_batchsize, C, height, width)\n        out = self.gamma*out + x\n        \n        return out\n\n\nclass CBAM(nn.Module):\n    def __init__(self, in_channels):\n        # def __init__(self):\n        super(CBAM, self).__init__()\n        inter_channels = in_channels // 4\n        self.conv1_c = nn.Sequential(nn.Conv2d(in_channels, inter_channels, 3, padding=1, bias=False),\n                                     nn.BatchNorm2d(inter_channels),\n                                     nn.ReLU())\n        \n        self.conv1_s = nn.Sequential(nn.Conv2d(in_channels, inter_channels, 3, padding=1, bias=False),\n                                     nn.BatchNorm2d(inter_channels),\n                                     nn.ReLU())\n\n        self.channel_gate = CAM_Module(inter_channels)\n        self.spatial_gate = PAM_Module(inter_channels)\n\n        self.conv2_c = nn.Sequential(nn.Conv2d(inter_channels, in_channels, 3, padding=1, bias=False),\n                                     nn.BatchNorm2d(in_channels),\n                                     nn.ReLU())\n        self.conv2_a = nn.Sequential(nn.Conv2d(inter_channels, in_channels, 3, padding=1, bias=False),\n                                     nn.BatchNorm2d(in_channels),\n                                     nn.ReLU())\n\n    def forward(self, x):\n        feat1    = self.conv1_c(x)\n        chnl_att = self.channel_gate(feat1)\n        chnl_att = self.conv2_c(chnl_att)\n\n        feat2    = self.conv1_s(x)\n        spat_att = self.spatial_gate(feat2)\n        spat_att = self.conv2_a(spat_att)\n\n        x_out = chnl_att + spat_att\n\n        return x_out","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"All models were imported from the great `timm` library. I have used [this dataset](https://www.kaggle.com/kozodoi/timm-pytorch-image-models) with the most recent version of `timm`."},{"metadata":{"trusted":true},"cell_type":"code","source":"####### MODEL ARCHITECTURE\n\ndef init_model(CFG):\n\n    ##### CONVOLUTIONAL PART\n\n    model = timm.create_model(model_name = CFG['backbone'], pretrained = False)\n    \n        \n    ##### CLASSIFIER PART\n    \n    if 'efficient' in CFG['backbone']:\n        model.classifier = nn.Linear(model.classifier.in_features, CFG['num_classes'])\n    elif ('vit' in CFG['backbone']) or ('deit' in CFG['backbone']):\n        model.head = nn.Linear(model.head.in_features, CFG['num_classes'])\n    else:\n        model.fc = nn.Linear(model.fc.in_features, CFG['num_classes'])\n            \n            \n    ##### MODEL WITH ATTENTION [EFFICIENTNET ONLY]\n    \n    if CFG['attention']:\n        \n        class model_with_attention(nn.Module):\n            \n            def __init__(self, CFG):\n                super().__init__()\n                self.backbone            = timm.create_model(model_name = CFG['backbone'], pretrained = False)\n                self.backbone._dropout   = nn.Dropout(0.1)\n                n_features               = self.backbone.classifier.in_features\n                self.backbone.classifier = nn.Linear(n_features, CFG['num_classes'])\n                self.local_fe            = CBAM(n_features)\n                self.dropout             = nn.Dropout(0.1)\n                self.classifier          = nn.Sequential(nn.Linear(n_features + n_features, n_features),\n                                                        nn.BatchNorm1d(n_features),\n                                                        nn.Dropout(0.1),\n                                                        nn.ReLU(),\n                                                        nn.Linear(n_features, CFG['num_classes']))\n\n            def forward(self, image):\n                enc_feas    = self.backbone.forward_features(image)\n                global_feas = self.backbone.global_pool(enc_feas)\n                global_feas = global_feas.flatten(start_dim = 1)\n                global_feas = self.dropout(global_feas)\n                local_feas  = self.local_fe(enc_feas)\n                local_feas  = torch.sum(local_feas, dim = [2, 3])\n                local_feas  = self.dropout(local_feas)\n                all_feas    = torch.cat([global_feas, local_feas], dim = 1)\n                outputs     = self.classifier(all_feas)\n                return outputs\n            \n        model = model_with_attention(CFG)\n    \n\n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# INFERENCE\n\nThe inference loop goes through all 33 models and stacks their predictions in a pandas dataframe."},{"metadata":{"trusted":true},"cell_type":"code","source":"########## INFERENCE\n\n# timer\ncv_start = time.time()\ngc.collect()\n\n# counter\nall_counter = 0\n\n# loop through models\nfor model_idx in tqdm(range(len(CFG.models))):\n        \n    # placeholders\n    cnn_preds = np.zeros((len(df), CFG.num_classes))\n    \n    # data prep\n    test_dataset = LeafData(data      = df, \n                            directory = CFG.data_path + 'test_images/' if not CFG.debug else CFG.data_path + 'train_images/',\n                            transform = define_augs(CFGs[model_idx]),\n                            labeled   = False)\n    test_loader = torch.utils.data.DataLoader(test_dataset, \n                                              batch_size  = CFG.batch_size, \n                                              shuffle     = False, \n                                              num_workers = CFG.num_workers,\n                                              pin_memory  = True)\n           \n    # model prep\n    model = init_model(CFGs[model_idx])\n    model = model.to(device)\n    model.load_state_dict(torch.load(CFG.models[model_idx] + 'weights_fold{}.pth'.format(CFG.one_fold),\n                                     map_location = device))\n    model.eval()\n\n    # probs placeholder\n    PROBS = []\n\n    # loop through batches \n    with torch.no_grad():\n        for batch_idx, inputs in enumerate(test_loader):\n\n            # extract inputs \n            inputs = inputs.to(device)\n\n            # preds placeholders\n            probs = torch.zeros((inputs.shape[0], CFG.num_classes), device = device)\n\n            # inference with TTA\n            for tta_idx in range(CFG.num_tta): \n                preds  = model(get_tta_flips(inputs, tta_idx))\n                probs += preds.softmax(axis = 1)\n\n            # store predictions\n            PROBS.append(probs.detach().cpu() / CFG.num_tta)\n\n    # transform predictions\n    PROBS      = torch.cat(PROBS).numpy()\n    cnn_preds += PROBS\n\n    # update counter\n    all_counter += 1\n\n    # clear memory\n    del model, inputs, preds, probs, PROBS, test_dataset, test_loader\n    gc.collect()\n    \n    # convert to DF\n    cnn_preds = pd.DataFrame(cnn_preds, columns = [str(CFG.models[model_idx]) + str(c) for c in ['c0', 'c1', 'c2', 'c3', 'c4']])\n    if model_idx == 0:\n        all_cnn_preds = cnn_preds.copy()\n    else:\n        all_cnn_preds = pd.concat([all_cnn_preds, cnn_preds], axis = 1)\n        \n# print performance\nprint('Finished {} preds x {} TTA in {:.2f} minutes'.format(all_counter, CFG.num_tta, (time.time() - cv_start) / 60))\nif CFG.debug:\n    print('Estimated time to run on 15k images: {:.2f} hours'.format((time.time() - cv_start) / 360))\ndisplay(all_cnn_preds.head())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Two special models have a separate inference loop:\n- for ImageNet classifier, we don't save probabilisitc predictions and only extract predicted classes\n- for binary cassava classifier, we extract the probability of the 1st class"},{"metadata":{"trusted":true},"cell_type":"code","source":"########## INFERENCE FOR IMAGENET MODEL\n\nif '/kaggle/input/pytorch-basic-v0/' in CFG.special_models:\n    \n    # timer\n    cv_start = time.time()\n\n    # clear memory\n    gc.collect()\n\n    # placeholders\n    cnn_preds = np.zeros((len(df), 1000))\n\n    # data prep\n    test_dataset = LeafData(data      = df, \n                            directory = CFG.data_path + 'test_images/' if not CFG.debug else CFG.data_path + 'train_images/',\n                            transform = define_augs(special_CFGs[0]),\n                            labeled   = False)\n    test_loader = torch.utils.data.DataLoader(test_dataset, \n                                              batch_size  = CFG.batch_size, \n                                              shuffle     = False, \n                                              num_workers = CFG.num_workers,\n                                              pin_memory  = True)\n\n\n    # model prep\n    model = init_model(special_CFGs[0])\n    model = model.to(device)\n    model.load_state_dict(torch.load(CFG.special_models[0] + 'weights_fold0.pth', map_location = device))\n    model.eval()\n\n    # probs placeholder\n    PROBS = []\n\n    # loop through batches \n    with torch.no_grad():\n        for batch_idx, inputs in enumerate(test_loader):\n\n            # extract inputs \n            inputs = inputs.to(device)\n\n            # preds placeholders\n            probs = torch.zeros((inputs.shape[0], 1000), device = device)\n\n            # inference with TTA\n            for tta_idx in range(CFG.num_tta): \n                preds  = model(get_tta_flips(inputs, tta_idx))\n                probs += preds.softmax(axis = 1)\n\n            # store predictions\n            PROBS.append(probs.detach().cpu() / CFG.num_tta)\n\n    # transform predictions\n    PROBS = torch.cat(PROBS).numpy()\n\n    # clear memory\n    del model, inputs, preds, probs\n    del test_dataset, test_loader\n    gc.collect()  \n\n    # convert to DF\n    PREDS_basic = np.argmax(PROBS, axis = 1)\n    \n    # print performance\n    print('Finished in {:.2f} minutes'.format((time.time() - cv_start) / 60))\n    if CFG.debug:\n        print('Estimated time to run on 15k images: {:.2f} hours'.format((time.time() - cv_start) / 360))\n    display(PREDS_basic[0:5])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"########## INFERENCE FOR BINARY MODEL\n\nif '/kaggle/input/pytorch-binary-v0/' in CFG.special_models:\n    \n    # timer\n    cv_start = time.time()\n\n    # clear memory\n    gc.collect()\n\n    # placeholders\n    cnn_preds = np.zeros((len(df), 2))\n\n    # data prep\n    test_dataset = LeafData(data      = df, \n                            directory = CFG.data_path + 'test_images/' if not CFG.debug else CFG.data_path + 'train_images/',\n                            transform = define_augs(special_CFGs[1]),\n                            labeled   = False)\n    test_loader = torch.utils.data.DataLoader(test_dataset, \n                                              batch_size  = CFG.batch_size, \n                                              shuffle     = False, \n                                              num_workers = CFG.num_workers,\n                                              pin_memory  = True)\n\n\n    # model prep\n    model = init_model(special_CFGs[1])\n    model = model.to(device)\n    model.load_state_dict(torch.load(CFG.special_models[1] + 'weights_fold0.pth', map_location = device))\n    model.eval()\n\n    # probs placeholder\n    PROBS = []\n\n    # loop through batches \n    with torch.no_grad():\n        for batch_idx, inputs in enumerate(test_loader):\n\n            # extract inputs \n            inputs = inputs.to(device)\n\n            # preds placeholders\n            probs = torch.zeros((inputs.shape[0], 2), device = device)\n\n            # inference with TTA\n            for tta_idx in range(CFG.num_tta): \n                preds  = model(get_tta_flips(inputs, tta_idx))\n                probs += preds.softmax(axis = 1)\n\n            # store predictions\n            PROBS.append(probs.detach().cpu() / CFG.num_tta)\n\n    # transform predictions\n    PROBS = torch.cat(PROBS).numpy()\n\n    # clear memory\n    del model, inputs, preds, probs\n    del test_dataset, test_loader\n    gc.collect()  \n\n    # convert to DF\n    PREDS_binary = PROBS[:, 0]\n    \n    # print performance\n    print('Finished in {:.2f} minutes'.format((time.time() - cv_start) / 60))\n    if CFG.debug:\n        print('Estimated time to run on 15k images: {:.2f} hours'.format((time.time() - cv_start) / 360))\n    display(PREDS_binary[0:5])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# STACKING\n\nFirst, we load all OOF predictions from the used models and make sure train / test predictions have the same format."},{"metadata":{"trusted":true},"cell_type":"code","source":"####### PREPARE OOF PREDS\n\nfor m in CFG.models:\n\n    tmp_train_preds = pd.read_csv(m + '/oof.csv')\n    tmp_train_preds.columns = ['image_id', 'label', 'source', 'fold'] + [str(m) + str(c) for c in ['c0', 'c1', 'c2', 'c3', 'c4']]\n    \n    if m == CFG.models[0]:\n        train_preds = tmp_train_preds        \n    else:\n        train_preds = train_preds.merge(tmp_train_preds[['image_id'] + [str(m) + str(c) for c in ['c0', 'c1', 'c2', 'c3', 'c4']]], how = 'left', on = 'image_id')\n \ntrain_preds.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"####### PREPARE TEST PREDS\n\ntest_preds = all_cnn_preds.copy()\ntest_preds = pd.concat([df['image_id'], test_preds], axis = 1)\ntest_preds.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We add a couple of meta-features based on the model predictions:\n- most common predicted class across the 33 models\n- from our experiments, working with class predictions worked better than probabilistic predictions. Here, we compute top-4 predicted classes per each model\n- finally, we add predictions from the two special models"},{"metadata":{"trusted":true},"cell_type":"code","source":"####### ADD FEATURES\n\n# most common predcted class\nfor m in CFG.models:\n    train_preds[m] = np.argmax(train_preds.filter(like = m).values, axis = 1)\n    test_preds[m]  = np.argmax(test_preds.filter(like  = m).values, axis = 1)\ntrain_preds['mode_pred'] = train_preds[CFG.models].mode(axis = 1)[0].astype('int')\ntest_preds['mode_pred']  = test_preds[CFG.models].mode(axis  = 1)[0].astype('int')\nfor m in CFG.models:\n    del train_preds[m], test_preds[m]\n    \n# top-4 class predictions\nfor m in CFG.models:\n\n    columns = train_preds.filter(like = m).columns\n\n    # predicted class\n    train_preds[m + '1'] = np.argsort(train_preds[columns].values, axis = 1)[:,-1]\n    test_preds[m  + '1'] = np.argsort(test_preds[columns].values,  axis = 1)[:,-1]\n\n    # second-best predicted class\n    train_preds[m + '2'] = np.argsort(train_preds[columns].values, axis = 1)[:,-2]\n    test_preds[m  + '2'] = np.argsort(test_preds[columns].values,  axis = 1)[:,-2]\n\n    # third-best predicted class\n    train_preds[m + '3'] = np.argsort(train_preds[columns].values, axis = 1)[:,-3]\n    test_preds[m  + '3'] = np.argsort(test_preds[columns].values,  axis = 1)[:,-3]\n    \n    # forth-best predicted class\n    train_preds[m + '4'] = np.argsort(train_preds[columns].values, axis = 1)[:,-4]\n    test_preds[m  + '4'] = np.argsort(test_preds[columns].values,  axis = 1)[:,-4]\n\n    to_drop = [str(m) + str(c) for c in ['c0', 'c1', 'c2', 'c3', 'c4']]\n    for col in to_drop:\n        del train_preds[col]\n        del test_preds[col]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"####### ADD IMAGENET MODEL\n\nif '/kaggle/input/pytorch-basic-v0/' in CFG.special_models:\n    \n    tmp_train_preds = pd.read_csv(CFG.special_models[0] + '/oof.csv')\n    tmp_train_preds[CFG.special_models[0]] = tmp_train_preds['pred']\n    \n    train_preds = train_preds.merge(tmp_train_preds[['image_id', CFG.special_models[0]]], how = 'left', on = 'image_id')\n                                                    \n    test_preds[CFG.special_models[0]] = PREDS_basic\n    \n    display(train_preds.head(1))\n    display(test_preds.head(1))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"####### ADD BINARY MODEL\n\nif '/kaggle/input/pytorch-binary-v0/' in CFG.special_models:\n    \n    tmp_train_preds = pd.read_csv(CFG.special_models[1] + '/oof.csv')\n    tmp_train_preds[CFG.special_models[1]] = tmp_train_preds['c0']\n    \n    train_preds = train_preds.merge(tmp_train_preds[['image_id', CFG.special_models[1]]], how = 'left', on = 'image_id')\n                                                    \n    test_preds[CFG.special_models[1]] = PREDS_basic\n    \n    display(train_preds.head(1))\n    display(test_preds.head(1))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"####### TRANSFORM DATA\n\ny      = train_preds['label']\nX      = train_preds.copy()\nX_test = test_preds.copy()\nprint(X.shape, y.shape, X_test.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"####### SELECT RELEVANT FEATURES\n\ndrop_features = ['image_id', 'label', 'source', 'fold']\nfeatures      = [f for f in X.columns if f not in drop_features]\nprint(len(features), 'features')\nprint(features)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Stacking is performed with LightGBM on 5-fold stratfied CV over the OOFs."},{"metadata":{"trusted":true},"cell_type":"code","source":"####### STACKING LOOP\n\n# partitinonig\nskf = StratifiedKFold(n_splits = CFG.lgb_folds, random_state = CFG.seed, shuffle = True)\n\n# placeholders\noof_preds   = np.zeros((len(X), CFG.num_classes))\nimportances = pd.DataFrame()\n\n# cross-validation\nprint('-' * 30)    \nfor fold, (trn_idx, val_idx) in enumerate(skf.split(X, y)):\n    \n    # placeholders\n    lgb_preds = np.zeros((len(X_test), CFG.num_classes))\n     \n    # extract samples\n    X_train, y_train = X.iloc[trn_idx][features], y.iloc[trn_idx]\n    X_valid, y_valid = X.iloc[val_idx][features], y.iloc[val_idx]\n    X_test           = X_test[features]\n    \n    # modeling\n    clf = lgb.LGBMClassifier(**CFG.lgb_params) \n    clf = clf.fit(X_train, y_train, \n                  eval_set              = [(X_valid, y_valid)],\n                  early_stopping_rounds = CFG.lgb_stop_rounds,\n                  verbose               = False)\n    \n    # prediction\n    oof_preds[val_idx, :] = clf.predict_proba(X_valid)\n    lgb_preds             = clf.predict_proba(X_test)\n    \n    # save preditions\n    lgb_preds = pd.DataFrame(lgb_preds, columns = ['f' + str(fold) + '_' + str(c) for c in ['c0', 'c1', 'c2', 'c3', 'c4']])\n    if fold == 0:\n        all_lgb_preds = lgb_preds.copy()\n    else:\n        all_lgb_preds = pd.concat([all_lgb_preds, lgb_preds], axis = 1)\n        \n    # feature importance\n    fold_importance_df = pd.DataFrame()\n    fold_importance_df['Feature']    = features\n    fold_importance_df['Importance'] = clf.feature_importances_\n    fold_importance_df['Fold']       = fold + 1\n    importances = pd.concat([importances, fold_importance_df], axis = 0)\n\n    # information\n    print('- FOLD {}: accuracy = {:.4f}'.format(fold + 1, (np.argmax(oof_preds[val_idx, :], axis = 1) == y_valid).sum() / len(y_valid)))\n    \n# print performance\nprint('-' * 30)\nprint('OOF accuracy = {:.4f}'.format((np.argmax(oof_preds, axis = 1) == y).sum() / len(y)))\nprint('-' * 30)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"###### FEATURE IMPORTANCE\n\nfig  = plt.figure(figsize = (10, 18))\ncols = importances[['Feature', 'Importance']].groupby('Feature').mean().sort_values(by = 'Importance', ascending = False).index\nimportance = importances.loc[importances.Feature.isin(cols)].groupby('Feature').mean().reset_index(drop = False)\nsns.barplot(x = 'Importance', y = 'Feature', data = importance.sort_values(by = 'Importance', ascending = False), ci = 0)\nplt.title('Feature Importance')\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Looking at feature importance, you can notice that two special models are ranked fairly high. Their contribution helps to further improve the performance of the ensemble."},{"metadata":{"trusted":true},"cell_type":"code","source":"####### BLEND STACKING FOLD PREDCITIONS\n\nfor c in ['c0', 'c1', 'c2', 'c3', 'c4']:\n    all_lgb_preds[c] = all_lgb_preds.filter(like = c).mean(axis = 1)\n\ndisplay(all_lgb_preds[['c0', 'c1', 'c2', 'c3', 'c4']].head(1))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# SUBMISSION"},{"metadata":{"trusted":true},"cell_type":"code","source":"####### TIMER\n\nif CFG.debug:\n    print('Estimated time to run inference on 15k images: {:.2f} hours'.format((time.time() - notebook_start) / 360))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"####### SUBMISSION FILE\n\nif all_counter == len(CFG.models): \n    \n    df['label'] = np.argmax(all_lgb_preds[['c0', 'c1', 'c2', 'c3', 'c4']].values, axis = 1)\n                                \n    if not CFG.debug: \n        df.to_csv('submission.csv', index = False)\n    \n    if len(df) == 1:\n        display(df)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}