{"cells":[{"metadata":{},"cell_type":"markdown","source":"# SUMMARY\n\nThis notebook reproduces my submission that scores **0.971** on private LB and reaches the **71st** place. The notebook implements ensemble of 5+2 CNN models using pretrained weights saved as Kaggle datasets.\n\n![ensemble](https://i.postimg.cc/c4cPcXng/ranzcr.png)\n- A detailed summary of my solution is published [in this discussion topic](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/226664)\n- Complete code including base model training is available in [this GitHub repo](https://github.com/kozodoi/Kaggle_RANZCR_Challenge)"},{"metadata":{},"cell_type":"markdown","source":"# PARAMETERS"},{"metadata":{"trusted":true},"cell_type":"code","source":"####### NOTEBOOK CONFIGURATION\n\nclass CFG:\n    \n    # environment\n    device      = 'GPU'  # device ['CPU', 'GPU']\n    cpu_workers = 2      # no. cores\n    debug       = True   # debug runs inference on a single batch\n    seed        = 13353  # random state\n\n    # inference\n    batch_size  = 50     # no. images per batch\n    num_tta     = 2      # no. TTA flips (between 1 and 8)\n    num_folds   = 3      # no. folds per model (between 1 and 5)\n    fold_idx    = False  # load weights from one fold [False, fold index]\n      \n    # blending\n    fold_blend  = 'pmean'  # how to blend folds  ['amean', 'median', 'gmean', 'pmean', 'rmean']\n    model_blend = 'pmean'  # how to blend models ['amean', 'median', 'gmean', 'pmean', 'rmean']\n    power       = 1/11     # power parameter for pmean\n    w_public    = 0.25     # weight of public models in the final blend   \n\n    # stacking\n    lgb_folds       = 5      # no. folds for stacking\n    label_features  = False  # use only label-specific features\n    sort_targets    = True   # sort targets by AUC\n    pred_as_feature = True   # include class predictions as features for other classes\n    lgb_stop_rounds = 200    # no. early stopping rounds\n    lgb_params      = {'objective':         'binary',\n                       'metrics':           'auc',\n                       'n_estimators':      10000,\n                       'learning_rate':     0.01,\n                       'num_leaves':        8,\n                       'max_depth':         5,\n                       'min_child_samples': 20,\n                       'subsample':         0.3,\n                       'colsample_bytree':  0.5,\n                       'reg_alpha':         0.1,\n                       'reg_lambda':        0.1,\n                       'silent':            True,\n                       'verbosity':         -1,\n                       'n_jobs' :           -1,\n                       'random_state':      13353}\n\n    # paths\n    data_path = '/kaggle/input/ranzcr-clip-catheter-line-classification/'\n    \n    # list of models\n    models = ['/kaggle/input/ranzcr-v12/', \n              '/kaggle/input/ranzcr-v15-pub/', \n              '/kaggle/input/ranzcr-v17-pub/', \n              '/kaggle/input/ranzcr-v13-pub/', \n              '/kaggle/input/ranzcr-v14-pub/']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"####### CONVERT CONFIGURATION\n\nCFG = dict(vars(CFG))\nfor key in ['__dict__', '__doc__', '__module__', '__weakref__']:\n    del CFG[key]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"During training, I save a `pickle` configuration file for each base model to make it convenient to import their weights, meta-parameters and image sizes during inference. Below I load all model configurations in a list."},{"metadata":{"trusted":true},"cell_type":"code","source":"####### LOAD MODEL CONFIGURATIONS\n\nimport pickle\n\nCFGs = []\n\nfor model in CFG['models']:\n    model_cfg = pickle.load(open(model + 'configuration.pkl', 'rb'))\n    CFGs.append(model_cfg)\n    \nprint('Numer of models:', len(CFGs))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# PREPARATIONS"},{"metadata":{"trusted":true},"cell_type":"code","source":"####### PACKAGES\n\nimport numpy as np\nimport pandas as pd\npd.set_option('display.max_columns', 100)\n\nimport torch\nimport torch.nn as nn\nimport torch.optim as optim\nfrom torch.utils.data import Dataset, DataLoader\nfrom torch.utils.data.sampler import RandomSampler, SequentialSampler\n\nimport albumentations as A\nfrom albumentations.pytorch import ToTensorV2\n\nfrom PIL import Image, ImageFile\nImageFile.LOAD_TRUNCATED_IMAGES = True\nimport cv2\n\nfrom scipy.stats import rankdata, gmean\nfrom sklearn.metrics import roc_auc_score\nfrom sklearn.model_selection import KFold, StratifiedKFold, GroupKFold\n\nimport lightgbm as lgb\n\nfrom tqdm import tqdm\n\nimport random\nimport time\nimport sys\nimport os\n\nimport gc\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n%matplotlib inline\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\nsys.path.append('../input/timm-pytorch-image-models/pytorch-image-models-master')\nimport timm","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"####### TRAINING DEVICE\n    \nif CFG['device'] == 'GPU':\n    print('Training on GPU...')\n    device = torch.device('cuda:0')\n\nif CFG['device'] == 'CPU':\n    print('Training on CPU...')\n    device = torch.device('cpu') ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The function `compute_blend()` implements simple blends (arithmetic, geometric, power and rank mean)."},{"metadata":{"trusted":true},"cell_type":"code","source":"####### UTILITIES\n\n# competition metric\ndef get_score(y_true, y_pred):\n    scores = []\n    for i in range(y_true.shape[1]):\n        score = roc_auc_score(y_true[:,i], y_pred[:,i])\n        scores.append(score)\n    avg_score = np.mean(scores)\n    return avg_score, scores\n\n# simple ensembles\ndef compute_blend(df, preds, blend, CFG, weights = None):\n    \n    if weights is None:\n        weights = np.ones(len(preds)) / len(preds)\n        \n    if blend == 'amean':\n        out = np.sum(df[preds].values * weights, axis = 1)\n    elif blend == 'median':\n        out = df[preds].median(axis = 1)\n    elif blend == 'gmean':\n        out = np.prod(np.power(df[preds].values, weights), axis = 1)\n    elif blend == 'pmean':\n        out = np.sum(np.power(df[preds].values, CFG['power']) * weights, axis = 1) ** (1 / CFG['power'])\n    elif blend == 'rmean':\n        out = np.sum(df[preds].rank(pct = True).values * weights, axis = 1)\n    return out\n\n# randomness\ndef seed_everything(seed):\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    random.seed(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.cuda.manual_seed_all(seed)\n    torch.backends.cudnn.deterministic = True\n    print('setting random seed to {}...'.format(seed))\n    \nseed_everything(CFG['seed'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# DATA PREP\n\nTo save GPU quota, I was subsetting the test images to a single batch when commiting the notebook. During submission, the condition `len(df) == 3582` is not fulfilled and the code is run on the full test set."},{"metadata":{"trusted":true},"cell_type":"code","source":"####### DATA PREPARATION\n\n# import data\ndf = pd.read_csv(CFG['data_path'] + 'sample_submission.csv')\n\n# num classes\nCFG['targets'] = ['ETT - Abnormal', \n                  'ETT - Borderline', \n                  'ETT - Normal', \n                  'NGT - Abnormal', \n                  'NGT - Borderline', \n                  'NGT - Incompletely Imaged', \n                  'NGT - Normal', \n                  'CVC - Abnormal',\n                  'CVC - Borderline', \n                  'CVC - Normal', \n                  'Swan Ganz Catheter Present']\nCFG['num_classes'] = len(CFG['targets'])\n\n# debug mode\nif CFG['debug'] and len(df) == 3582:\n    print('Subsetting data in the debug mode...')\n    df = df.head(CFG['batch_size'])\n    \nprint(df.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To evaluate OOF performance, I am averaging AUCs on validation folds instead of computing OOF AUC on the full training data. Since AUC is a ranking indicator, this way to evaluate models provided a more robust CV/LB correlation."},{"metadata":{"trusted":true},"cell_type":"code","source":"####### CHECK OOF PERFORMANCE\n\n# load preds\nfor m in CFG['models']:\n\n    tmp_train_preds         = pd.read_csv(m + '/oof.csv')\n    tmp_train_preds.columns = ['StudyInstanceUID'] + CFG['targets'] + ['PatientID', 'fold'] + [m + ' ' + c for c in CFG['targets']]\n\n    if m == CFG['models'][0]:\n        train_preds = tmp_train_preds        \n    else:\n        train_preds = train_preds.merge(tmp_train_preds[['StudyInstanceUID'] + [m + ' ' + c for c in CFG['targets']]], how = 'left', on = 'StudyInstanceUID')\n\n# sort models by performance\nweights = []\nfor model_idx, m in enumerate(CFG['models']):\n    score = 0\n    for fold_idx in range(5):\n        tmp_train_preds = train_preds.loc[train_preds['fold'] == fold_idx]\n        score += get_score(tmp_train_preds[CFG['targets']].values, tmp_train_preds[[m + ' ' + c for c in CFG['targets']]].values)[0] / 5\n    weights.append(score)\nsorted_ids     = list(np.argsort(np.array(weights)))\nsorted_weights = [weights[i] for i in sorted_ids]\nCFG['models']  = [CFG['models'][i] for i in sorted_ids]\nCFGs           = [CFGs[i] for i in sorted_ids]\n\n# display performance \nprint('-' * 45)\nprint('{:<5}{:<33}{:>5}'.format('ID', 'Model', 'AUC'))\nprint('-' * 45)\nfor model_idx, m in enumerate(CFG['models']):\n    print('{:<5}{:<33}{:.4f}'.format(model_idx + 1, m, sorted_weights[model_idx]))\nprint('-' * 45)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Below I evaluate different ensembles on OOF predictions. Unweighted power mean with `power = 1/11` worked best in my experiments."},{"metadata":{"trusted":true},"cell_type":"code","source":"####### CHECK BLEND PERFORMANCE\n\n# compute predcitions\nfor c in CFG['targets']:   \n    \n    class_preds = train_preds.filter(like = 'kaggle').filter(like = c).columns\n    \n    for blend in ['amean', 'median', 'gmean', 'pmean', 'rmean']:\n        train_preds[blend + ' ' + c] = compute_blend(train_preds, class_preds, blend, CFG)\n        \n    for blend in ['amean', 'median', 'gmean', 'pmean', 'rmean']:\n        train_preds['w' + blend + ' ' + c] = compute_blend(train_preds, class_preds, blend, CFG, weights = np.array(sorted_weights))\n    \n# compute performance\nprint('-' * 18)\nprint('{:<10}{:>5}'.format('Blend', 'AUC'))\nprint('-' * 18)\nfor blend in ['amean', 'median', 'gmean', 'pmean', 'rmean']:\n    score = 0\n    for fold_idx in range(5):\n        tmp_train_preds = train_preds.loc[train_preds['fold'] == fold_idx]\n        score += get_score(tmp_train_preds[CFG['targets']].values, tmp_train_preds[[blend + ' ' + c for c in CFG['targets']]].values)[0] / 5\n    print('{:<10}{:>5.4f}'.format(blend, score))    \nprint('-' * 18)\nfor blend in ['amean', 'median', 'gmean', 'pmean', 'rmean']:\n    score = 0\n    for fold_idx in range(5):\n        tmp_train_preds = train_preds.loc[train_preds['fold'] == fold_idx]\n        score += get_score(tmp_train_preds[CFG['targets']].values, tmp_train_preds[['w' + blend + ' ' + c for c in CFG['targets']]].values)[0] / 5\n    print('{:<10}{:>5.4f}'.format('w' + blend, score))    \nprint('-' * 18)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# IMAGE PROCESSING"},{"metadata":{"trusted":true},"cell_type":"code","source":"####### DATASET\n\ndef get_dataset(CFG):\n\n    class ImageData(Dataset):\n\n        # initialization\n        def __init__(self, \n                     df, \n                     path, \n                     transform = None, \n                     labeled   = False,\n                     indexed   = False):\n            self.df        = df\n            self.path      = path\n            self.transform = transform\n            self.labeled   = labeled\n            self.indexed   = indexed\n\n        # length\n        def __len__(self):\n            return len(self.df)\n\n        # get item  \n        def __getitem__(self, idx):\n\n            # import\n            path  = os.path.join(self.path, self.df.iloc[idx]['StudyInstanceUID'] + '.jpg')\n            image = cv2.imread(path, cv2.IMREAD_GRAYSCALE)\n            if image is None:\n                raise FileNotFoundError(path)\n        \n            # crop black borders\n            '''\n            Borrowed from https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/224146\n            '''\n            mask  = image > 0\n            image = image[np.ix_(mask.any(1), mask.any(0))]\n\n            # convert to RGB\n            image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n            \n            # augmentations\n            if self.transform is not None:\n                image = self.transform(image = image)['image']\n                \n            # transform \n            if CFG['normalize'] == 'public':\n                image = image.astype(np.float32)\n                image = image.transpose(2, 0, 1)\n                image = torch.tensor(image).float()\n\n            # output\n            if self.labeled:\n                label = torch.tensor(self.df.iloc[idx][CFG['targets']]).float()\n                if self.indexed:\n                    idx = torch.tensor(idx)\n                    return idx, image, label\n                else: \n                    return image, label\n            else:\n                return image\n            \n    return ImageData","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"####### AUGMENTATIONS\n\ndef get_augs(CFG, image_size = None):\n\n    if CFG['normalize'] != 'public':\n        test_augs = A.Compose([A.Resize(height = image_size, \n                                        width  = image_size),\n                               A.Normalize(mean = CFG['pixel_mean'],\n                                           std  = CFG['pixels_std']),\n                               ToTensorV2()])\n    else:\n        test_augs = A.Compose([A.Resize(height = image_size, \n                                        width  = image_size),\n                               A.Normalize()])\n\n    # output\n    return test_augs","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"####### TTA HELPER FUNCTION\n\n'''\nBased on https://github.com/haqishen/SIIM-ISIC-Melanoma-Classification-1st-Place-Solution\n'''\ndef get_tta_flips(img, i):\n\n    if i >= 4:\n        img = img.transpose(2, 3)\n    if i % 4 == 0:\n        return img\n    elif i % 4 == 1:\n        return img.flip(3)\n    elif i % 4 == 2:\n        return img.flip(2)\n    elif i % 4 == 3:\n        return img.flip(3).flip(2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# MODEL PREP\n\nAll models were imported from the `timm` library. I use [this dataset](https://www.kaggle.com/kozodoi/timm-pytorch-image-models) with the most recent version of `timm`."},{"metadata":{"trusted":true},"cell_type":"code","source":"####### MODEL ARCHITECTURE\n\ndef get_model(CFG, device, num_classes):\n    \n    if CFG['weights'] != 'public':\n\n        model = timm.create_model(model_name = CFG['backbone'], \n                                  pretrained = False,\n                                  in_chans   = CFG['channels'])\n\n        if 'efficient' in CFG['backbone']:\n            model.classifier = nn.Linear(model.classifier.in_features, num_classes)\n        else:\n            model.fc = nn.Linear(model.fc.in_features, num_classes)\n\n    else:\n        \n        class CustomModel(nn.Module):\n            \n            def __init__(self, model_name = 'resnet200d', out_dim = 11, pretrained = False):\n                super().__init__()\n                self.model             = timm.create_model(model_name, pretrained=False)\n                n_features             = self.model.fc.in_features\n                self.model.global_pool = nn.Identity()\n                self.model.fc          = nn.Identity()\n                self.pooling           = nn.AdaptiveAvgPool2d(1)\n                self.fc                = nn.Linear(n_features, out_dim)\n\n            def forward(self, x):\n                bs              = x.size(0)\n                features        = self.model(x)\n                pooled_features = self.pooling(features).view(bs, -1)\n                output          = self.fc(pooled_features)\n                return output\n\n        model = CustomModel(CFG['backbone'], num_classes, True)\n    \n        \n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# INFERENCE\n\nThe inference loop goes through 5 CNN models and stacks their predictions in a pandas dataframe. For each model, I also loop through the specified number of folds and iterate fold indices between 1 and 5."},{"metadata":{"trusted":true},"cell_type":"code","source":"########## INFERENCE FOR NORMAL MODELS\n\n# timer\ncv_start = time.time()\ngc.collect()\n\n# counter\nall_counter  = 0\nfold_counter = 0 if not CFG['fold_idx'] else CFG['fold_idx']\n\n# placeholder\nall_cnn_preds = None\n\n# loop through models\nfor model_idx in range(len(CFG['models'])):\n           \n    # data prep\n    ImageData    = get_dataset(CFGs[model_idx])\n    test_dataset = ImageData(df        = df, \n                             path      = CFG['data_path'] + 'test/',\n                             transform = get_augs(CFGs[model_idx], image_size = CFGs[model_idx]['image_size']),\n                             labeled   = False,\n                             indexed   = False)\n    test_loader = DataLoader(test_dataset, \n                             batch_size  = CFG['batch_size'], \n                             shuffle     = False, \n                             num_workers = CFG['cpu_workers'],\n                             pin_memory  = True)\n    \n    # loop thgough folds\n    for fold_idx in tqdm(range(CFG['num_folds'])):\n           \n        # model prep\n        model = get_model(CFGs[model_idx], device = device, num_classes = CFG['num_classes'])\n        model = model.to(device)\n        model.load_state_dict(torch.load(CFG['models'][model_idx] + 'weights_fold{}.pth'.format(fold_counter),map_location = device))\n        model.eval()\n\n        # probs placeholder\n        PROBS = []\n\n        # loop through batches \n        with torch.no_grad():\n            for batch_idx, inputs in enumerate(test_loader):\n\n                # extract inputs \n                inputs = inputs.to(device)\n                \n                # preds placeholders\n                probs = torch.zeros((inputs.shape[0], CFG['num_classes']), device = device)\n\n                # inference with TTA\n                for tta_idx in range(CFG['num_tta']): \n                    preds  = model(get_tta_flips(inputs, tta_idx))\n                    probs += preds.sigmoid()\n\n                # store predictions\n                PROBS.append(probs.detach().cpu() / CFG['num_tta'])\n                \n        # store predictions\n        cnn_preds     = pd.DataFrame(torch.cat(PROBS).numpy(), columns = [CFG['models'][model_idx] + str(fold_idx) + '/' + c for c in CFG['targets']])\n        all_cnn_preds = pd.concat([all_cnn_preds, cnn_preds], axis = 1)\n\n        # update counters\n        all_counter += 1\n        if not CFG['fold_idx']:\n            fold_counter += 1\n            if fold_counter == CFG['num_folds']:\n                fold_counter = 0\n\n        # clear memory\n        del model, inputs, preds, probs, PROBS\n        gc.collect()\n        \n    # clear memory\n    del test_loader, test_dataset\n    gc.collect()\n        \n# print performance\nprint('Finished {} preds x {} TTA in {:.2f} hours'.format(all_counter, CFG['num_tta'], (time.time() - cv_start) / 3600))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"####### BLEND FOLD PREDICTIONS\n\nprint('Blending fold predictions with: ' + CFG['fold_blend'])\n\nfor m in CFG['models']:\n    for c in CFG['targets']:\n        class_preds = all_cnn_preds.filter(like = m).filter(like = c).columns\n        all_cnn_preds[m + c] = compute_blend(all_cnn_preds, class_preds, CFG['fold_blend'], CFG)\n        all_cnn_preds.drop(class_preds, axis = 1, inplace = True)\n           \nall_cnn_preds.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# STACKING\n\nFirst, I load OOF predictions from CNNs and make sure train / test dataframes have the same format."},{"metadata":{"trusted":true},"cell_type":"code","source":"####### PREPARE OOF PREDS\n\nfor m in CFG['models']:\n\n    tmp_train_preds         = pd.read_csv(m + '/oof.csv')\n    tmp_train_preds.columns = ['StudyInstanceUID'] + CFG['targets'] + ['PatientID', 'fold'] + [m + '' + c for c in CFG['targets']]\n\n    if m == CFG['models'][0]:\n        train_preds = tmp_train_preds        \n    else:\n        train_preds = train_preds.merge(tmp_train_preds[['StudyInstanceUID'] + [m + '' + c for c in CFG['targets']]], how = 'left', on = 'StudyInstanceUID')\n\ntrain_preds.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"####### PREPARE TEST PREDS\n\ntest_preds = all_cnn_preds.copy()\ntest_preds = pd.concat([df['StudyInstanceUID'], test_preds], axis = 1)    \ntest_preds.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"####### TRANSFORM DATA\n\nX      = train_preds.copy()\nX_test = test_preds.copy()\n\ndrop_features = ['StudyInstanceUID', 'PatientID', 'fold'] + CFG['targets']\nfeatures      = [f for f in X.columns if f not in drop_features]\nprint(len(features), 'features')\ndisplay(features)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Stacking is performed on OOFs using [group stratified 5-fold partitiioning](https://www.kaggle.com/underwearfitting/how-to-properly-split-folds) shared by [@underwearfitting](https://www.kaggle.com/underwearfitting). I am using the same partitinoing to train base models as well."},{"metadata":{"trusted":true},"cell_type":"code","source":"########## PARTITIONING\n\nfolds = pd.read_csv('/kaggle/input/how-to-properly-split-folds/train_folds.csv')\ndel X['fold']\nX = X.merge(folds[['StudyInstanceUID', 'fold']], how = 'left', on = 'StudyInstanceUID')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Sorting targets by AUC helps to improve stacking performance. When starting with easier labels, meta-models for more difficult labels have more features (that include OOF meta predictions of previous labels)."},{"metadata":{"trusted":true},"cell_type":"code","source":"####### SORT TARGETS BY AUC\n\nif CFG['sort_targets']:\n\n    sorted_targets = ['Swan Ganz Catheter Present',\n                      'ETT - Normal', \n                      'ETT - Abnormal', \n                      'ETT - Borderline', \n                      'NGT - Abnormal', \n                      'NGT - Normal', \n                      'NGT - Incompletely Imaged', \n                      'NGT - Borderline', \n                      'CVC - Abnormal',\n                      'CVC - Normal',\n                      'CVC - Borderline']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For each label, I do the following:\n- training a separate stacking model on 5 folds\n- comparing performance of stacking with the best simple blend (power mean)\n- saving predictions of the method with the highest OOF AUC\n- saving meta predictions as a feature for the stacking model for the next label"},{"metadata":{"trusted":true},"cell_type":"code","source":"########## STACKING\n\n# placeholders\ncnn_oof = np.zeros((len(X),      CFG['num_classes']))\nlgb_oof = np.zeros((len(X),      CFG['num_classes']))\nlgb_tst = np.zeros((len(X_test), CFG['lgb_folds'], CFG['num_classes']))\nall_lgb_preds = None\n\n# modeling loop\ncv_start = time.time()\nprint('-' * 45)\nprint('{:<28}{:<7}{:>5}'.format('Label', 'Model', 'AUC'))\nprint('-' * 45)\nfor label in sorted_targets:\n\n    # extract label\n    y = X[label]\n\n    # extract features\n    label_features = [f for f in features if label in f] if CFG['label_features'] else features\n    \n    # placeholders\n    lgb_auc = 0\n    cnn_auc = 0\n\n    # cross-validation\n    for fold in range(CFG['lgb_folds']):\n        \n        # extract index\n        trn_idx = X.loc[X['fold'] != fold].index\n        val_idx = X.loc[X['fold'] == fold].index\n        \n        # extract samples\n        X_train, y_train = X.iloc[trn_idx][label_features], y.iloc[trn_idx]\n        X_valid, y_valid = X.iloc[val_idx][label_features], y.iloc[val_idx]\n        X_test_label     = X_test[label_features]\n\n        # modeling\n        clf = lgb.LGBMClassifier(**CFG['lgb_params']) \n        clf = clf.fit(X_train, y_train, \n                      eval_set              = [(X_valid, y_valid)],\n                      early_stopping_rounds = CFG['lgb_stop_rounds'],\n                      verbose               = False)\n\n        # prediction\n        cnn_oof[val_idx, CFG['targets'].index(label)] = compute_blend(X_valid, list(X_valid.filter(like = label).columns), CFG['fold_blend'], CFG)\n        lgb_oof[val_idx, CFG['targets'].index(label)] = clf.predict_proba(X_valid)[:,      1]\n        lgb_tst[:, fold, CFG['targets'].index(label)] = clf.predict_proba(X_test_label)[:, 1]\n        \n        # performance\n        cnn_auc += roc_auc_score(y_valid ,cnn_oof[val_idx, CFG['targets'].index(label)]) / CFG['lgb_folds']\n        lgb_auc += roc_auc_score(y_valid, lgb_oof[val_idx, CFG['targets'].index(label)]) / CFG['lgb_folds']\n\n    # print label performance\n    print('{:<29}{:<7}{:>5.4f}'.format(label, 'CNN', cnn_auc))\n    print('{:<29}{:<7}{:>5.4f}'.format(label, 'LGB', lgb_auc))\n    print('-' * 45)    \n    \n    # replace LGB pred if CNN pred is better\n    if cnn_auc >= lgb_auc:\n        for fold in range(CFG['lgb_folds']):\n\n            # extract data\n            val_idx = X.loc[X['fold'] == fold].index\n            X_valid, y_valid = X.iloc[val_idx][label_features], y.iloc[val_idx]\n\n            # replace predcitions\n            lgb_oof[val_idx, CFG['targets'].index(label)] = compute_blend(X_valid,      list(X_valid.filter(like      = label).columns), CFG['fold_blend'], CFG)\n            lgb_tst[:, fold, CFG['targets'].index(label)] = compute_blend(X_test_label, list(X_test_label.filter(like = label).columns), CFG['fold_blend'], CFG)\n            \n    # store predictions\n    for fold in range(CFG['lgb_folds']):\n        lgb_preds     = pd.DataFrame(lgb_tst[:, fold, CFG['targets'].index(label)], columns = [str(fold) + '/' + label])\n        all_lgb_preds = pd.concat([all_lgb_preds, lgb_preds], axis = 1)\n    \n    # add prediction as feature to next model\n    if CFG['pred_as_feature']:\n        X['LGB '      + label] = lgb_oof[:, CFG['targets'].index(label)]\n        X_test['LGB ' + label] = np.mean(lgb_tst[:, :, CFG['targets'].index(label)], axis = 1)\n        features.append('LGB ' + label)\n        \n# print overall performance\nscore_cnn = 0\nscore_lgb = 0\nfor fold in range(CFG['lgb_folds']):\n    val_idx    = X.loc[X['fold'] == fold].index\n    score_cnn += get_score(X.iloc[val_idx][CFG['targets']].values, cnn_oof[val_idx, :])[0] / CFG['lgb_folds']\n    score_lgb += get_score(X.iloc[val_idx][CFG['targets']].values, lgb_oof[val_idx, :])[0] / CFG['lgb_folds']\nprint('{:<29}{:<7}{:>5.4f}'.format('OVERALL', 'CNN', score_cnn))\nprint('{:<29}{:<7}{:>5.4f}'.format('OVERALL', 'LGB', score_lgb))\nprint('-' * 45)\nprint('Finished in {:.2f} minutes'.format((time.time() - cv_start) / 60))\n\n# clear memory\ndel cnn_oof, lgb_tst, lgb_oof, clf, X_train, X_valid, X_test, X_test_label, y_train, y_valid\ndel features, label_features, trn_idx, val_idx, folds, train_preds, test_preds\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"####### BLEND FOLD PREDICTIONS\n\nprint('Blending fold predictions with: ' + CFG['fold_blend'])\n\nfor c in CFG['targets']:\n    class_preds      = all_lgb_preds.filter(like = c).columns\n    all_lgb_preds[c] = compute_blend(all_lgb_preds, class_preds, CFG['fold_blend'], CFG)\n    all_lgb_preds.drop(class_preds, axis = 1, inplace = True)\n           \nall_lgb_preds.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# PUBLIC MODELS INFERENCE\nIn addition to 5 CNN models used in the stacking ensemble, I add 2 public models:\n- [SeResNet152D](https://www.kaggle.com/ammarali32/seresnet152d-cv9615) by [@ammarali32](https://www.kaggle.com/ammarali32)\n- [ResNet200D](https://www.kaggle.com/ammarali32/resnet200d-public) by [@ammarali32](https://www.kaggle.com/ammarali32)"},{"metadata":{"trusted":true},"cell_type":"code","source":"####### INFERENCE FOR PUBLIC MODELS \n\nif CFG['w_public'] > 0:\n\n    gc.collect()\n\n    BATCH_SIZE = 96\n    IMAGE_SIZE = 640\n    TEST_PATH               = '../input/ranzcr-clip-catheter-line-classification/test'\n    MODEL_PATH_resnet200d   = '../input/resnet200d-public/resnet200d_320_CV9632.pth'\n    MODEL_PATH_seresnet152d = '../input/seresnet152d-cv9615/seresnet152d_320_CV96.15.pth'\n\n\n    class TestDataset(Dataset):\n        def __init__(self, df, transform=None):\n            self.df = df\n            self.file_names = df['StudyInstanceUID'].values\n            self.transform = transform\n\n        def __len__(self):\n            return len(self.df)\n\n        def __getitem__(self, idx):\n            file_name = self.file_names[idx]\n            file_path = f'{TEST_PATH}/{file_name}.jpg'   \n            image = cv2.imread(file_path, cv2.IMREAD_GRAYSCALE)\n            mask  = image > 0\n            image = image[np.ix_(mask.any(1), mask.any(0))]\n            image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n            if self.transform:\n                augmented = self.transform(image=image)\n                image = augmented['image']\n            return image\n\n\n    def get_transforms(CFG):\n        return A.Compose([A.Resize(IMAGE_SIZE, IMAGE_SIZE),\n                          A.Normalize(),\n                          ToTensorV2()])\n\n\n    class ResNet200D(nn.Module):\n        def __init__(self, model_name = 'resnet200d'):\n            super().__init__()\n            self.model = timm.create_model(model_name, pretrained=False)\n            n_features = self.model.fc.in_features\n            self.model.global_pool = nn.Identity()\n            self.model.fc = nn.Identity()\n            self.pooling = nn.AdaptiveAvgPool2d(1)\n            self.fc = nn.Linear(n_features, 11)\n\n        def forward(self, x):\n            bs = x.size(0)\n            features = self.model(x)\n            pooled_features = self.pooling(features).view(bs, -1)\n            output = self.fc(pooled_features)\n            return output\n\n\n    class SeResNet152D(nn.Module):\n        def __init__(self, model_name = 'seresnet152d'):\n            super().__init__()\n            self.model = timm.create_model(model_name, pretrained=False)\n            n_features = self.model.fc.in_features\n            self.model.global_pool = nn.Identity()\n            self.model.fc = nn.Identity()\n            self.pooling = nn.AdaptiveAvgPool2d(1)\n            self.fc = nn.Linear(n_features, 11)\n\n        def forward(self, x):\n            bs = x.size(0)\n            features = self.model(x)\n            pooled_features = self.pooling(features).view(bs, -1)\n            output = self.fc(pooled_features)\n            return output\n\n\n    def inference(models, test_loader, device):\n        tk0 = tqdm(enumerate(test_loader), total = len(test_loader))\n        probs = []\n        for i, (images) in tk0:\n            images = images.to(device)\n            avg_preds = []\n            for model in models:\n                with torch.no_grad():\n                    y_preds1 = model(images)\n                    y_preds2 = model(images.flip(-1))\n                y_preds = (y_preds1.sigmoid().to('cpu').numpy() + y_preds2.sigmoid().to('cpu').numpy()) / 2\n                avg_preds.append(y_preds)\n            avg_preds = np.mean(avg_preds, axis = 0)\n            probs.append(avg_preds)\n        probs = np.concatenate(probs)\n        return probs\n\n\n    models200D = []\n    model = ResNet200D()\n    model.load_state_dict(torch.load(MODEL_PATH_resnet200d)['model'])\n    model.eval()\n    model.to(device)\n    models200D.append(model)\n    del model\n\n    models152D = []\n    model = SeResNet152D()\n    model.load_state_dict(torch.load(MODEL_PATH_seresnet152d)['model'])\n    model.eval()\n    model.to(device)\n    models152D.append(model)\n    del model\n    \n\n    test_dataset    = TestDataset(df, transform = get_transforms(CFG))\n    test_loader     = DataLoader(test_dataset, batch_size = BATCH_SIZE, shuffle = False, num_workers = CFG['cpu_workers'], pin_memory = True)\n    predictions200d = inference(models200D, test_loader, device)\n    predictions152d = inference(models152D, test_loader, device)\n    \n    target_cols = df.iloc[:, 1:12].columns.tolist()\n    \n    predictions200d = pd.DataFrame(predictions200d, columns = ['200d/' + c for c in target_cols])\n    predictions152d = pd.DataFrame(predictions152d, columns = ['152d/' + c for c in target_cols])\n    predictions     = pd.concat([predictions200d, predictions152d], axis = 1)\n    \n    df_pub = predictions.copy()\n    display(df_pub.head())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"####### BLEND PUBLIC MODELS\n\nif CFG['w_public'] == 0:\n    \n    df_pub = all_lgb_preds.copy()\n    \nelse:\n\n    for c in CFG['targets']:\n        class_preds = df_pub.filter(like = c).columns\n        df_pub[c] = compute_blend(df_pub, class_preds, CFG['model_blend'], CFG, weights = np.array([2/3, 1/3]))\n        df_pub.drop(class_preds, axis = 1, inplace = True)\n        \ndf_pub.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# SUBMISSION"},{"metadata":{"trusted":true},"cell_type":"code","source":"####### BLEND MY AND PUBLIC MODELS\n\nall_preds = all_lgb_preds.copy()\nall_preds.columns = ['my/'     + c for c in all_preds.columns]\ndf_pub.columns    = ['public/' + c for c in df_pub.columns]\n\npreds = pd.concat([all_preds, df_pub], axis = 1)\n\nfor c in CFG['targets']:\n    class_preds = preds.filter(like = c).columns\n    preds[c] = compute_blend(preds, class_preds, CFG['model_blend'], CFG, weights = np.array([1 - CFG['w_public'], CFG['w_public']]))\n    preds.drop(class_preds, axis = 1, inplace = True)\n\npreds.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Before saving `submisssion.csv`, I check that all model/fold combinations have succeffuly completed inference. This helps to catch memory errors that sometimes do not stop the notebook from running and can produce poor LB performance because some models have not been run."},{"metadata":{"trusted":true},"cell_type":"code","source":"####### SUBMISSION FILE\n\nif all_counter == len(CFG['models'] * CFG['num_folds']): \n        \n    for c in CFG['targets']:\n        df[c] = preds[c].rank(pct = True)\n        \n    df.to_csv('submission.csv', index = False)\n    display(df.head())","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}