{"cells":[{"metadata":{},"cell_type":"markdown","source":"# About\n\nI'm trying the suggested concept in this topic:  \nhttps://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/205208\n\nI share all the experimental results in this topic:  \nhttps://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/207230\n\nFor the comparison, I used the similar settings with @yasufuminakama 's [baseline](https://www.kaggle.com/yasufuminakama/ranzcr-resnext50-32x4d-starter-training)(ResNeXt50_32x4d). Great thanks :)\n\n## Experimental Settings\n\n### model\nBase model: **ResNet200D** (used the pretrained model provided by [timm](https://github.com/rwightman/pytorch-image-models/tree/392595c7eb02c3f6353a7806aa9d1e3f569d47e7))\n\n* **NOTE: I use [the pre-trained model](https://www.kaggle.com/ammarali32/startingpointschestx) shared by @ammarali32 . Thanks!**\n\nThe model is branched at the CNN backbone's output.  \nSeparated **Spatial-Attention Modules** and MLP(Linear -> ReLU -> Dropout -> Linear)s are prepared for each group(`ETT(3)`, `NGT(4)`, `CVC(3)`, and `Swan(1)`).\n\nI use only Spatial-Attention but @ipythonx tries more complicated attention. For more details, see this notebook:  \nhttps://www.kaggle.com/ipythonx/tf-keras-ranzcr-multi-attention-efficientnet/\n\n### data augmentation\n\n* implemented by [albumentations](https://albumentations.ai/docs/)\n* Train:\n    - HorizontalFlip\n    - ShiftScaleRotate\n    - RandomResizedCrop\n    - Cutout\n    - Normalize\n* Valid (no augmnentation):\n    * Normalize\n\n### learning settings\n* CV Strategy: Multi-Label Stratified Group KFold\n    * K=5 (this notebook is `fold 0`)\n    * use `PatientID` as group id\n* max epochs: 16\n* data:\n    * input image size: 3x512x512\n    * batch size: 16 (with mixed precision)\n* loss: [BCEWithLogitsLoss](https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html#torch.nn.BCEWithLogitsLoss)\n* optimizer: [Adam](https://pytorch.org/docs/stable/optim.html#torch.optim.Adam)\n    * learning rate: 2.5e-04\n* learning rate scheduler: [CosineAnnealingWarmRestarts](https://pytorch.org/docs/stable/optim.html#torch.optim.lr_scheduler.CosineAnnealingWarmRestarts) (only 1 cycle)\n    * T_0: 16\n    * T_mult: 1\n\n## Data Preprocess\n\nI prepared `.npy` files by image size for faster training.  \n(I shared them as a Kaggle Dataset: https://www.kaggle.com/ttahara/ranzcr-clip-train-numpy )\n\n\nIt is better to load them using `mmap_mode` option for preventing RAM's OOM Error.\n```python\nimport numpy as np\ntrain_data_arr = np.load(\"../input/ranzcr-clip-train-numpy/train_448x448.npy\", mmap_mode=\"r\")\n```"},{"metadata":{},"cell_type":"markdown","source":"# Prepare"},{"metadata":{},"cell_type":"markdown","source":"## import"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import gc\nimport os\nimport sys\nimport time\nimport copy\nimport random\nimport shutil\nimport typing as tp\nfrom pathlib import Path\nfrom argparse import ArgumentParser\n\nimport yaml\nimport numpy as np\nimport pandas as pd\nfrom scipy.sparse import coo_matrix\nfrom sklearn.metrics import roc_auc_score\n\nfrom tqdm import tqdm\nfrom joblib import Parallel, delayed\n\nimport cv2\nimport albumentations\n\nfrom albumentations.core.transforms_interface import ImageOnlyTransform, DualTransform\nfrom albumentations.pytorch import ToTensorV2\n\nimport torch\nfrom torch import nn\nfrom torch.utils import data\nfrom torchvision import models as torchvision_models\n\nsys.path.append('../input/pytorch-image-models/pytorch-image-models-master')\nimport timm\n\nsys.path.append(\"../input/pytorch-pfn-extras/pytorch-pfn-extras-0.3.2/\")\nimport pytorch_pfn_extras as ppe\nfrom pytorch_pfn_extras.training import extensions as ppe_extensions","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ROOT = Path.cwd().parent\nINPUT = ROOT / \"input\"\nOUTPUT = ROOT / \"output\"\nDATA = INPUT / \"ranzcr-clip-catheter-line-classification\"\nTRAIN = DATA / \"train\"\nTEST = DATA / \"test\"\n\n\nTRAIN_NPY = INPUT / \"ranzcr-clip-train-numpy\"\nTMP = ROOT / \"tmp\"\nTMP.mkdir(exist_ok=True)\n\nRANDAM_SEED = 1086\nN_CLASSES = 11\n# FOLDS = [0, 1, 2, 3, 4]\n# N_FOLD = len(FOLDS)\nFOLDS = [1,]\nN_FOLD = 5\n\nCLASSES = [\n    'ETT - Abnormal',\n    'ETT - Borderline',\n    'ETT - Normal',\n    'NGT - Abnormal',\n    'NGT - Borderline',\n    'NGT - Incompletely Imaged',\n    'NGT - Normal',\n    'CVC - Abnormal',\n    'CVC - Borderline',\n    'CVC - Normal',\n    'Swan Ganz Catheter Present'\n]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## read data"},{"metadata":{"trusted":true},"cell_type":"code","source":"for p in DATA.iterdir():\n    print(p.name)\n\ntrain = pd.read_csv(DATA / \"train.csv\")\nsmpl_sub =  pd.read_csv(DATA / \"sample_submission.csv\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## split fold"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def multi_label_stratified_group_k_fold(label_arr: np.array, gid_arr: np.array, n_fold: int, seed: int=42):\n    \"\"\"\n    create multi-label stratified group kfold indexs.\n\n    reference: https://www.kaggle.com/jakubwasikowski/stratified-group-k-fold-cross-validation\n    input:\n        label_arr: numpy.ndarray, shape = (n_train, n_class)\n            multi-label for each sample's index using multi-hot vectors\n        gid_arr: numpy.array, shape = (n_train,)\n            group id for each sample's index\n        n_fold: int. number of fold.\n        seed: random seed.\n    output:\n        yield indexs array list for each fold's train and validation.\n    \"\"\"\n    np.random.seed(seed)\n    random.seed(seed)\n    start_time = time.time()\n    n_train, n_class = label_arr.shape\n    gid_unique = sorted(set(gid_arr))\n    n_group = len(gid_unique)\n\n    # # aid_arr: (n_train,), indicates alternative id for group id.\n    # # generally, group ids are not 0-index and continuous or not integer.\n    gid2aid = dict(zip(gid_unique, range(n_group)))\n#     aid2gid = dict(zip(range(n_group), gid_unique))\n    aid_arr = np.vectorize(lambda x: gid2aid[x])(gid_arr)\n\n    # # count labels by class\n    cnts_by_class = label_arr.sum(axis=0)  # (n_class, )\n\n    # # count labels by group id.\n    col, row = np.array(sorted(enumerate(aid_arr), key=lambda x: x[1])).T\n    cnts_by_group = coo_matrix(\n        (np.ones(len(label_arr)), (row, col))\n    ).dot(coo_matrix(label_arr)).toarray().astype(int)\n    del col\n    del row\n    cnts_by_fold = np.zeros((n_fold, n_class), int)\n\n    groups_by_fold = [[] for fid in range(n_fold)]\n    group_and_cnts = list(enumerate(cnts_by_group))  # pair of aid and cnt by group\n    np.random.shuffle(group_and_cnts)\n    print(\"finished preparation\", time.time() - start_time)\n    for aid, cnt_by_g in sorted(group_and_cnts, key=lambda x: -np.std(x[1])):\n        best_fold = None\n        min_eval = None\n        for fid in range(n_fold):\n            # # eval assignment.\n            cnts_by_fold[fid] += cnt_by_g\n            fold_eval = (cnts_by_fold / cnts_by_class).std(axis=0).mean()\n            cnts_by_fold[fid] -= cnt_by_g\n\n            if min_eval is None or fold_eval < min_eval:\n                min_eval = fold_eval\n                best_fold = fid\n\n        cnts_by_fold[best_fold] += cnt_by_g\n        groups_by_fold[best_fold].append(aid)\n    print(\"finished assignment.\", time.time() - start_time)\n\n    gc.collect()\n    idx_arr = np.arange(n_train)\n    for fid in range(n_fold):\n        val_groups = groups_by_fold[fid]\n\n        val_indexs_bool = np.isin(aid_arr, val_groups)\n        train_indexs = idx_arr[~val_indexs_bool]\n        val_indexs = idx_arr[val_indexs_bool]\n\n        print(\"[fold {}]\".format(fid), end=\" \")\n        print(\"n_group: (train, val) = ({}, {})\".format(n_group - len(val_groups), len(val_groups)), end=\" \")\n        print(\"n_sample: (train, val) = ({}, {})\".format(len(train_indexs), len(val_indexs)))\n\n        yield train_indexs, val_indexs","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"label_arr = train[CLASSES].values\ngroup_id = train.PatientID.values\n\ntrain_val_indexs = list(\n    multi_label_stratified_group_k_fold(label_arr, group_id, N_FOLD, RANDAM_SEED))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"train[\"fold\"] = -1\nfor fold_id, (trn_idx, val_idx) in enumerate(train_val_indexs):\n    train.loc[val_idx, \"fold\"] = fold_id\n    \ntrain.groupby(\"fold\")[CLASSES].sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## preprocess train images"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# def resize_images(img_id, input_dir, output_dir, resize_to=(512, 512)):\n#     img_path = input_dir / (img_id + \".jpg\")\n#     save_path = output_dir / (img_id + \".jpg\")\n    \n#     img = cv2.imread(str(img_path), cv2.IMREAD_GRAYSCALE)\n#     img = cv2.resize(img, resize_to)\n#     cv2.imwrite(str(save_path), img, )\n\n# IMAGE_SIZE = (320, 320)\n# TRAIN_RESIZED = TMP / \"train_{0}x{1}\".format(*IMAGE_SIZE)\n# TRAIN_RESIZED.mkdir(exist_ok=True)\n# TRAIN_RESIZED\n\n# _ = Parallel(n_jobs=2, verbose=5)([\n#     delayed(resize_images)(img_id, TRAIN, TRAIN_RESIZED, IMAGE_SIZE)\n#     for img_id in train.StudyInstanceUID.values\n# ])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# def save_as_numpy(\n#     input_dir,\n#     output_path,\n#     meta_file, size=(512, 512), ext=\"png\"):\n#     arr = np.zeros((len(meta_file), *size), dtype=\"uint8\")\n#     for idx, img_id in tqdm(enumerate(meta_file[\"StudyInstanceUID\"].values)):\n#         img_path = input_dir / f\"{img_id}.{ext}\"\n#         img = cv2.imread(str(img_path), cv2.IMREAD_GRAYSCALE)\n#         arr[idx] = img\n        \n#     np.save(output_path, arr)\n#     return arr\n\n# train_arr_320 = save_as_numpy(\n#     TRAIN_RESIZED,\n#     TMP / \"train_{0}x{1}.npy\".format(*IMAGE_SIZE),\n#     train, IMAGE_SIZE)\n\n# train_arr_320.shape\n\n# del train_arr_320\n# gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training"},{"metadata":{},"cell_type":"markdown","source":"## definition"},{"metadata":{},"cell_type":"markdown","source":"### custom model"},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"def get_activation(activ_name: str=\"relu\"):\n    \"\"\"\"\"\"\n    act_dict = {\n        \"relu\": nn.ReLU(inplace=True),\n        \"tanh\": nn.Tanh(),\n        \"sigmoid\": nn.Sigmoid(),\n        \"identity\": nn.Identity()}\n    if activ_name in act_dict:\n        return act_dict[activ_name]\n    else:\n        raise NotImplementedError\n        \n\nclass Conv2dBNActiv(nn.Module):\n    \"\"\"Conv2d -> (BN ->) -> Activation\"\"\"\n    \n    def __init__(\n        self, in_channels: int, out_channels: int,\n        kernel_size: int, stride: int=1, padding: int=0,\n        bias: bool=False, use_bn: bool=True, activ: str=\"relu\"\n    ):\n        \"\"\"\"\"\"\n        super(Conv2dBNActiv, self).__init__()\n        layers = []\n        layers.append(nn.Conv2d(\n            in_channels, out_channels,\n            kernel_size, stride, padding, bias=bias))\n        if use_bn:\n            layers.append(nn.BatchNorm2d(out_channels))\n            \n        layers.append(get_activation(activ))\n        self.layers = nn.Sequential(*layers)\n        \n    def forward(self, x):\n        \"\"\"Forward\"\"\"\n        return self.layers(x)\n        \n\nclass SSEBlock(nn.Module):\n    \"\"\"channel `S`queeze and `s`patial `E`xcitation Block.\"\"\"\n\n    def __init__(self, in_channels: int):\n        \"\"\"Initialize.\"\"\"\n        super(SSEBlock, self).__init__()\n        self.channel_squeeze = nn.Conv2d(\n            in_channels=in_channels, out_channels=1,\n            kernel_size=1, stride=1, padding=0, bias=False)\n        self.sigmoid = nn.Sigmoid()\n\n    def forward(self, x):\n        \"\"\"Forward.\"\"\"\n        # # x: (bs, ch, h, w) => h: (bs, 1, h, w)\n        h = self.sigmoid(self.channel_squeeze(x))\n        # # x, h => return: (bs, ch, h, w)\n        return x * h\n    \n    \nclass SpatialAttentionBlock(nn.Module):\n    \"\"\"Spatial Attention for (C, H, W) feature maps\"\"\"\n    \n    def __init__(\n        self, in_channels: int,\n        out_channels_list: tp.List[int],\n    ):\n        \"\"\"Initialize\"\"\"\n        super(SpatialAttentionBlock, self).__init__()\n        self.n_layers = len(out_channels_list)\n        channels_list = [in_channels] + out_channels_list\n        assert self.n_layers > 0\n        assert channels_list[-1] == 1\n        \n        for i in range(self.n_layers - 1):\n            in_chs, out_chs = channels_list[i: i + 2]\n            layer = Conv2dBNActiv(in_chs, out_chs, 3, 1, 1, activ=\"relu\")\n            setattr(self, f\"conv{i + 1}\", layer)\n            \n        in_chs, out_chs = channels_list[-2:]\n        layer = Conv2dBNActiv(in_chs, out_chs, 3, 1, 1, activ=\"sigmoid\")\n        setattr(self, f\"conv{self.n_layers}\", layer)\n    \n    def forward(self, x):\n        \"\"\"Forward\"\"\"\n        h = x\n        for i in range(self.n_layers):\n            h = getattr(self, f\"conv{i + 1}\")(h)\n            \n        h = h * x\n        return h","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class SingleHeadModel(nn.Module):\n    \n    def __init__(\n        self, base_name: str='resnext50_32x4d', out_dim: int=11, pretrained=False\n    ):\n        \"\"\"\"\"\"\n        self.base_name = base_name\n        super(SingleHeadModel, self).__init__()\n        \n        # # load base model\n        base_model = timm.create_model(base_name, pretrained=pretrained)\n        in_features = base_model.num_features\n        \n        # # remove global pooling and head classifier\n        # base_model.reset_classifier(0, '')\n        base_model.reset_classifier(0)\n        \n        # # Shared CNN Bacbone\n        self.backbone = base_model\n        \n        # # Single Heads.\n        self.head_fc = nn.Sequential(\n            nn.Linear(in_features, in_features),\n            nn.ReLU(inplace=True),\n            nn.Dropout(0.5),\n            nn.Linear(in_features, out_dim))\n\n    def forward(self, x):\n        \"\"\"\"\"\"\n        h = self.backbone(x)\n        h = self.head_fc(h)\n        return h\n        \n\nclass MultiHeadModel(nn.Module):\n    \n    def __init__(\n        self, base_name: str='resnext50_32x4d',\n        out_dims_head: tp.List[int]=[3, 4, 3, 1], pretrained=False):\n        \"\"\"\"\"\"\n        self.base_name = base_name\n        self.n_heads = len(out_dims_head)\n        super(MultiHeadModel, self).__init__()\n        \n        # # load base model\n        base_model = timm.create_model(base_name, pretrained=pretrained)\n        in_features = base_model.num_features\n        \n        # # remove global pooling and head classifier\n        base_model.reset_classifier(0, '')\n        \n        # # Shared CNN Bacbone\n        self.backbone = base_model\n        \n        # # Multi Heads.\n        for i, out_dim in enumerate(out_dims_head):\n            layer_name = f\"head_{i}\"\n            layer = nn.Sequential(\n                SpatialAttentionBlock(in_features, [64, 32, 16, 1]),\n                nn.AdaptiveAvgPool2d(output_size=1),\n                nn.Flatten(start_dim=1),\n                nn.Linear(in_features, in_features),\n                nn.ReLU(inplace=True),\n                nn.Dropout(0.5),\n                nn.Linear(in_features, out_dim))\n            setattr(self, layer_name, layer)\n\n    def forward(self, x):\n        \"\"\"\"\"\"\n        h = self.backbone(x)\n        hs = [\n            getattr(self, f\"head_{i}\")(h) for i in range(self.n_heads)]\n        y = torch.cat(hs, axis=1)\n        return y\n    \n    \nclass MultiHeadResNet200D(nn.Module):\n    \n    def __init__(\n        self, out_dims_head: tp.List[int]=[3, 4, 3, 1], pretrained=False\n    ):\n        \"\"\"\"\"\"\n        self.base_name = \"resnet200d_320\"\n        self.n_heads = len(out_dims_head)\n        super(MultiHeadResNet200D, self).__init__()\n        \n        # # load base model\n        base_model = timm.create_model(\n            self.base_name, num_classes=sum(out_dims_head), pretrained=False)\n        in_features = base_model.num_features\n        \n        if pretrained:\n            pretrained_model_path = '../input/startingpointschestx/resnet200d_320_chestx.pth'\n            state_dict = dict()\n            for k, v in torch.load(pretrained_model_path, map_location='cpu')[\"model\"].items():\n                if k[:6] == \"model.\":\n                    k = k.replace(\"model.\", \"\")\n                state_dict[k] = v\n            base_model.load_state_dict(state_dict)\n        \n        # # remove global pooling and head classifier\n        base_model.reset_classifier(0, '')\n        \n        # # Shared CNN Bacbone\n        self.backbone = base_model\n        \n        # # Multi Heads.\n        for i, out_dim in enumerate(out_dims_head):\n            layer_name = f\"head_{i}\"\n            layer = nn.Sequential(\n                SpatialAttentionBlock(in_features, [64, 32, 16, 1]),\n                nn.AdaptiveAvgPool2d(output_size=1),\n                nn.Flatten(start_dim=1),\n                nn.Linear(in_features, in_features),\n                nn.ReLU(inplace=True),\n                nn.Dropout(0.5),\n                nn.Linear(in_features, out_dim))\n            setattr(self, layer_name, layer)\n\n    def forward(self, x):\n        \"\"\"\"\"\"\n        h = self.backbone(x)\n        hs = [\n            getattr(self, f\"head_{i}\")(h) for i in range(self.n_heads)]\n        y = torch.cat(hs, axis=1)\n        return y\n    \n\n## forward test\n# m = SingleHeadModel(\"resnext50_32x4d\", 11, True)\nm = MultiHeadResNet200D([3, 4, 3, 1], True)\nm = m.eval()\n\nx = torch.rand(1, 3, 256, 256)\nwith torch.no_grad():\n    y = m(x)\nprint(\"[forward test]\")\nprint(\"input:\\t{}\\noutput:\\t{}\".format(x.shape, y.shape))\n\ndel m; del x; del y\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### dataset"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# class LabeledImageDataset(data.Dataset):\n#     \"\"\"\n#     Dataset class for (image, label) pairs\n\n#     reads images and applys transforms to them.\n\n#     Attributes\n#     ----------\n#     file_list : List[Tuple[tp.Union[str, Path], tp.Union[int, float, np.ndarray]]]\n#         list of (image file, label) pair\n#     transform_list : List[Dict]\n#         list of dict representing image transform \n#     \"\"\"\n\n#     def __init__(\n#         self,\n#         file_list: tp.List[\n#             tp.Tuple[tp.Union[str, Path], tp.Union[int, float, np.ndarray]]],\n#         transform_list: tp.List[tp.Dict],\n#     ):\n#         \"\"\"Initialize\"\"\"\n#         self.file_list = file_list\n#         self.transform = ImageTransformForCls(transform_list)\n\n#     def __len__(self):\n#         \"\"\"Return Num of Images.\"\"\"\n#         return len(self.file_list)\n\n#     def __getitem__(self, index):\n#         \"\"\"Return transformed image and mask for given index.\"\"\"\n#         img_path, label = self.file_list[index]\n#         img = self._read_image_as_array(img_path)\n        \n#         img, label = self.transform((img, label))\n#         return img, label\n\n#     def _read_image_as_array(self, path: str):\n#         \"\"\"Read image file and convert into numpy.ndarray\"\"\"\n#         img_arr = cv2.imread(str(path))\n#         img_arr = cv2.cvtColor(img_arr, cv2.COLOR_BGR2RGB)\n#         return img_arr\n\n\nclass LabeledImageDatasetNumpy(data.Dataset):\n    \"\"\"\n    Dataset class for (image, label) pairs\n\n    reads images and applys transforms to them.\n\n    Attributes\n    ----------\n    file_list : List[tp.Union[np.ndarray, np.ndarray]\n        list of (image, label) pair\n    transform : object\n        image transform object\n    \"\"\"\n\n    def __init__(\n        self,\n        file_list: tp.List[\n            tp.Tuple[np.ndarray, tp.Union[int, float, np.ndarray]]],\n        transform_list: tp.List[tp.Dict],\n        copy_in_channels=True, in_channels=3,\n    ):\n        \"\"\"Initialize\"\"\"\n        self.file_list = file_list\n        self.transform = ImageTransformForCls(transform_list)\n        self.copy_in_channels = copy_in_channels\n        self.in_channels = in_channels \n\n    def __len__(self):\n        \"\"\"Return Num of Images.\"\"\"\n        return len(self.file_list)\n\n    def __getitem__(self, index):\n        \"\"\"Return transformed image and mask for given index.\"\"\"\n        img, label = self.file_list[index]\n        if img.shape[-1] == 2:\n            img = img[..., None]\n\n        if self.copy_in_channels:\n            img = np.repeat(img, self.in_channels, axis=2)\n        \n        img, label = self.transform((img, label))\n        return img, label","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# def get_file_list(stgs, train_all, ext=\"jpg\"):\n#     \"\"\"Get file path and target info.\"\"\"\n#     # train_all = pd.read_csv(DATA / stgs[\"globals\"][\"meta_file\"])\n#     use_fold = stgs[\"globals\"][\"val_fold\"]\n    \n#     train_df = train_all[train_all[\"fold\"] != use_fold]\n#     val_df = train_all[train_all[\"fold\"] == use_fold]\n    \n#     # train_data_dir = DATA / stgs[\"globals\"][\"dataset_name\"]\n#     train_data_dir = TMP / stgs[\"globals\"][\"dataset_name\"]\n#     print(train_data_dir)\n\n#     train_file_list = list(zip(\n#         [train_data_dir / f\"{img_id}.{ext}\" for img_id in train_df[\"StudyInstanceUID\"].values],\n#         train_df[CLASSES].values.astype(\"f\")\n#     ))\n#     val_file_list = list(zip(\n#         [train_data_dir / f\"{img_id}.{ext}\" for img_id in val_df[\"StudyInstanceUID\"].values],\n#         val_df[CLASSES].values.astype(\"f\")\n#     ))\n\n#     return train_file_list, val_file_list\n\n\ndef get_file_list_with_array(stgs, train_all):\n    \"\"\"Get file path and target info.\"\"\"\n    # train_all = pd.read_csv(DATA / stgs[\"globals\"][\"meta_file\"])\n    use_fold = stgs[\"globals\"][\"val_fold\"]\n    \n    train_idx = train_all[train_all[\"fold\"] != use_fold].index.values\n    if stgs[\"globals\"][\"debug\"]:\n        train_idx = train_idx[:len(train_idx) // 20]\n    val_idx = train_all[train_all[\"fold\"] == use_fold].index.values\n    \n    train_data_path = TRAIN_NPY / \"{}.npy\".format(stgs[\"globals\"][\"dataset_name\"])\n    print(train_data_path)\n    # train_data_arr = np.load(train_data_path)\n    train_data_arr = np.load(train_data_path, mmap_mode=\"r\")\n    label_arr = train_all[CLASSES].values.astype(\"f\")\n    print(train_data_arr.shape, label_arr.shape)\n\n    train_file_list = [\n        (train_data_arr[idx][..., None], label_arr[idx])  for idx in train_idx]\n    val_file_list = [\n        (train_data_arr[idx][..., None], label_arr[idx])  for idx in val_idx]\n\n    return train_file_list, val_file_list\n\n\ndef get_dataloaders_cls(\n    stgs: tp.Dict,\n    train_file_list: tp.List[tp.List],\n    val_file_list: tp.List[tp.List],\n    dataset_class: data.Dataset\n):\n    \"\"\"Create DataLoader\"\"\"\n    train_loader = val_loader = None\n    if train_file_list is not None:\n        train_dataset = dataset_class(\n            train_file_list, **stgs[\"dataset\"][\"train\"])\n        train_loader = data.DataLoader(\n            train_dataset, **stgs[\"loader\"][\"train\"])\n\n    if val_file_list is not None:\n        val_dataset = dataset_class(\n            val_file_list, **stgs[\"dataset\"][\"val\"])\n        val_loader = data.DataLoader(\n            val_dataset, **stgs[\"loader\"][\"val\"])\n\n    return train_loader, val_loader","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### image transform"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"class ImageTransformBase:\n    \"\"\"\n    Base Image Transform class.\n\n    Args:\n        data_augmentations: List of tuple(method: str, params :dict), each elems pass to albumentations\n    \"\"\"\n\n    def __init__(self, data_augmentations: tp.List[tp.Tuple[str, tp.Dict]]):\n        \"\"\"Initialize.\"\"\"\n        augmentations_list = [\n            self._get_augmentation(aug_name)(**params)\n            for aug_name, params in data_augmentations]\n        self.data_aug = albumentations.Compose(augmentations_list)\n\n    def __call__(self, pair: tp.Tuple[np.ndarray]) -> tp.Tuple[np.ndarray]:\n        \"\"\"You have to implement this by task\"\"\"\n        raise NotImplementedError\n\n    def _get_augmentation(self, aug_name: str) -> tp.Tuple[ImageOnlyTransform, DualTransform]:\n        \"\"\"Get augmentations from albumentations\"\"\"\n        if hasattr(albumentations, aug_name):\n            return getattr(albumentations, aug_name)\n        else:\n            return eval(aug_name)\n\n\nclass ImageTransformForCls(ImageTransformBase):\n    \"\"\"Data Augmentor for Classification Task.\"\"\"\n\n    def __init__(self, data_augmentations: tp.List[tp.Tuple[str, tp.Dict]]):\n        \"\"\"Initialize.\"\"\"\n        super(ImageTransformForCls, self).__init__(data_augmentations)\n\n    def __call__(self, in_arrs: tp.Tuple[np.ndarray]) -> tp.Tuple[np.ndarray]:\n        \"\"\"Apply Transform.\"\"\"\n        img, label = in_arrs\n        augmented = self.data_aug(image=img)\n        img = augmented[\"image\"]\n\n        return img, label","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### metric"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"class EvalFuncManager(nn.Module):\n    \"\"\"Manager Class for evaluation at the end of epoch\"\"\"\n\n    def __init__(\n        self,\n        iters_per_epoch: int,\n        evalfunc_dict: tp.Dict[str, nn.Module],\n        prefix: str = \"val\"\n    ) -> None:\n        \"\"\"Initialize\"\"\"\n        self.tmp_iter = 0\n        self.iters_per_epoch = iters_per_epoch\n        self.prefix = prefix\n        self.metric_names = []\n        super(EvalFuncManager, self).__init__()\n        for k, v in evalfunc_dict.items():\n            setattr(self, k, v)\n            self.metric_names.append(k)\n        self.reset()\n\n    def reset(self) -> None:\n        \"\"\"Reset State.\"\"\"\n        self.tmp_iter = 0\n        for name in self.metric_names:\n            getattr(self, name).reset()\n\n    def __call__(self, y: torch.Tensor, t: torch.Tensor) -> None:\n        \"\"\"Forward.\"\"\"\n        for name in self.metric_names:\n            getattr(self, name).update(y, t)\n        self.tmp_iter += 1\n\n        if self.tmp_iter == self.iters_per_epoch:\n            ppe.reporting.report({\n                \"{}/{}\".format(self.prefix, name): getattr(self, name).compute()\n                for name in self.metric_names\n            })\n            self.reset()\n            \n            \nclass MeanLoss(nn.Module):\n    \n    def __init__(self):\n        super(MeanLoss, self).__init__()\n        self.loss_sum = 0\n        self.n_examples = 0\n        \n    def forward(self, y: torch.Tensor, t: torch.Tensor):\n        \"\"\"Compute metric at once\"\"\"\n        return self.loss_func(y, t)\n\n    def reset(self):\n        \"\"\"Reset state\"\"\"\n        self.loss_sum = 0\n        self.n_examples = 0\n    \n    def update(self, y: torch.Tensor, t: torch.Tensor):\n        \"\"\"Update metric by mini batch\"\"\"\n        self.loss_sum += self(y, t).item() * y.shape[0]\n        self.n_examples += y.shape[0]\n\n    def compute(self):\n        \"\"\"Compute metric for dataset\"\"\"\n        return self.loss_sum / self.n_examples\n    \n\nclass MyLogLoss(MeanLoss):\n    \n    def __init__(self, **params):\n        super(MyLogLoss, self).__init__()\n        self.loss_func = nn.BCEWithLogitsLoss(**params)\n\n\nclass MyROCAUC(nn.Module):\n    \"\"\"ROC AUC score\"\"\"\n\n    def __init__(self, average=\"macro\") -> None:\n        \"\"\"Initialize.\"\"\"\n        self.average = average\n        self._pred_list = []\n        self._true_list = []\n        super(MyROCAUC, self).__init__()\n\n    def reset(self) -> None:\n        \"\"\"Reset State.\"\"\"\n        self._pred_list = []\n        self._true_list = []\n\n    def update(self, y_pred: torch.Tensor, y_true: torch.Tensor) -> None:\n        \"\"\"Forward.\"\"\"\n        self._pred_list.append(y_pred.detach().cpu().numpy())\n        self._true_list.append(y_true.detach().cpu().numpy())\n\n    def compute(self) -> float:\n        \"\"\"Calc and return metric value.\"\"\"\n        y_pred = np.concatenate(self._pred_list, axis=0)\n        y_true = np.concatenate(self._true_list, axis=0)\n        score = roc_auc_score(y_true, y_pred, average=self.average)\n        return score\n\n    def forward(self, y_pred: torch.Tensor, y_true: torch.Tensor) -> float:\n        \"\"\"Forward.\"\"\"\n        self.reset()\n        self.update(y_pred, y_true)\n        return self.compute()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### training utils"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def set_random_seed(seed: int = 42, deterministic: bool = False):\n    \"\"\"Set seeds\"\"\"\n    random.seed(seed)\n    np.random.seed(seed)\n    os.environ[\"PYTHONHASHSEED\"] = str(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)  # type: ignore\n    torch.backends.cudnn.deterministic = deterministic  # type: ignore","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def get_stepper(manager, stgs, scheduler):\n    \"\"\"\"\"\"\n    def dummy_step():\n        pass\n    \n    def step():\n        scheduler.step()\n        \n    def step_with_epoch_detail():\n        scheduler.step(manager.epoch_detail)\n        \n    \n    if stgs[\"scheduler\"][\"name\"] == None:\n        return dummy_step, dummy_step\n    \n    elif stgs[\"scheduler\"][\"name\"] == \"CosineAnnealingWarmRestarts\":\n        return dummy_step, step_with_epoch_detail\n    \n    elif stgs[\"scheduler\"][\"name\"] == \"OneCycleLR\":\n        return dummy_step, step\n    \n    else:\n        return step, dummy_step\n\n\ndef run_train_loop(\n    manager, stgs, model, device, train_loader, optimizer, scheduler, loss_func\n):\n    \"\"\"Run minibatch training loop\"\"\"\n    step_scheduler_by_epoch, step_scheduler_by_iter = get_stepper(manager, stgs, scheduler)\n\n    if stgs[\"globals\"][\"use_amp\"]:     \n        while not manager.stop_trigger:\n            model.train()\n            scaler = torch.cuda.amp.GradScaler()\n            for x, t in train_loader:\n                with manager.run_iteration():\n                    x, t = x.to(device), t.to(device)\n                    optimizer.zero_grad()\n                    with torch.cuda.amp.autocast():\n                        y = model(x)\n                        loss = loss_func(y, t)\n                    ppe.reporting.report({'train/loss': loss.item()})\n                    scaler.scale(loss).backward()\n                    scaler.step(optimizer)\n                    scaler.update()\n                    step_scheduler_by_iter()\n            step_scheduler_by_epoch()\n    else:\n        while not manager.stop_trigger:\n            model.train()\n            for x, t in train_loader:\n                with manager.run_iteration():\n                    x, t = x.to(device), t.to(device)\n                    optimizer.zero_grad()\n                    y = model(x)\n                    loss = loss_func(y, t)\n                    ppe.reporting.report({'train/loss': loss.item()})\n                    loss.backward()\n                    optimizer.step()\n                    step_scheduler_by_iter()\n            step_scheduler_by_epoch()\n        \n        \ndef run_eval(stgs, model, device, batch, eval_manager):\n    \"\"\"Run evaliation for val or test. this function is applied to each batch.\"\"\"\n    model.eval()\n    x, t = batch\n    if stgs[\"globals\"][\"use_amp\"]:\n        with torch.cuda.amp.autocast(): \n            y = model(x.to(device))\n            eval_manager(y, t.to(device))\n    else:\n        y = model(x.to(device))\n        eval_manager(y, t.to(device))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def set_extensions(\n    manager, args, model, device,\n    val_loader, optimizer,\n    eval_manager, print_progress: bool = False,\n):\n    \"\"\"Set extensions for PPE\"\"\"\n    eval_names = [\"val/{}\".format(name) for name in eval_manager.metric_names]\n    \n    log_extentions = [\n        ppe_extensions.observe_lr(optimizer=optimizer),\n        ppe_extensions.LogReport(),\n        ppe_extensions.PlotReport([\"train/loss\", \"val/loss\"], 'epoch', filename='loss.png'),\n        ppe_extensions.PlotReport([\"lr\"], 'epoch', filename='lr.png'),\n        ppe_extensions.PrintReport([\n            \"epoch\", \"iteration\", \"lr\", \"train/loss\", *eval_names, \"elapsed_time\"])\n    ]\n    if print_progress:\n        log_extentions.append(ppe_extensions.ProgressBar(update_interval=20))\n\n    for ext in log_extentions:\n        manager.extend(ext)\n        \n    manager.extend( # evaluation\n        ppe_extensions.Evaluator(\n            val_loader, model,\n            eval_func=lambda *batch: run_eval(args, model, device, batch, eval_manager)),\n        trigger=(1, \"epoch\"))\n    \n    manager.extend(  # model snapshot\n        ppe_extensions.snapshot(target=model, filename=\"snapshot_epoch_{.epoch}.pth\"),\n        trigger=ppe.training.triggers.MaxValueTrigger(key=\"val/metric\", trigger=(1, 'epoch')))\n\n    return manager","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def train_one_fold(settings, train_all, output_path, print_progress=False):\n    \"\"\"train one fold\"\"\"\n    torch.backends.cudnn.benchmark = True\n    set_random_seed(settings[\"globals\"][\"seed\"])\n\n    # # prepare train, valid paths\n    # train_file_list, val_file_list = get_file_list(settings, train_all, \"png\")\n    train_file_list, val_file_list = get_file_list_with_array(settings, train_all)\n    print(\"train: {}, val: {}\".format(len(train_file_list), len(val_file_list)))\n\n    device = torch.device(settings[\"globals\"][\"device\"])\n    # # get data_loader\n    train_loader, val_loader = get_dataloaders_cls(\n        settings, train_file_list, val_file_list, LabeledImageDatasetNumpy)\n\n    # # get model\n    model = MultiHeadResNet200D(**settings[\"model\"][\"params\"])\n    model.to(device)\n\n    # # get optimizer\n    optimizer = getattr(\n        torch.optim, settings[\"optimizer\"][\"name\"]\n    )(model.parameters(), **settings[\"optimizer\"][\"params\"])\n\n    # # get scheduler\n    if settings[\"scheduler\"][\"name\"] == \"OneCycleLR\":\n        settings[\"scheduler\"][\"params\"][\"epochs\"] = settings[\"globals\"][\"max_epoch\"]\n        settings[\"scheduler\"][\"params\"][\"steps_per_epoch\"] = len(train_loader)\n    scheduler = getattr(\n        torch.optim.lr_scheduler, settings[\"scheduler\"][\"name\"]\n    )(optimizer, **settings[\"scheduler\"][\"params\"])\n\n    # # get loss\n    if hasattr(nn, settings[\"loss\"][\"name\"]):\n        loss_func = getattr(nn, settings[\"loss\"][\"name\"])(**settings[\"loss\"][\"params\"])\n    else:\n        loss_func = eval(settings[\"loss\"][\"name\"])(**settings[\"loss\"][\"params\"])\n    loss_func.to(device)\n\n    eval_manager = EvalFuncManager(\n        len(val_loader), {\n            metric[\"report_name\"]: eval(metric[\"name\"])(**metric[\"params\"])\n            for metric in settings[\"eval\"]\n        })\n    eval_manager.to(device)\n\n    # # get manager\n    # trigger = None\n    trigger = ppe.training.triggers.EarlyStoppingTrigger(\n        check_trigger=(1, 'epoch'),\n        # monitor='val/metric', mode=\"min\",\n        monitor='val/metric', mode=\"max\",\n        patience=settings[\"globals\"][\"patience\"], verbose=False,\n        max_trigger=(settings[\"globals\"][\"max_epoch\"], 'epoch'),\n    )\n    manager = ppe.training.ExtensionsManager(\n        model, optimizer, settings[\"globals\"][\"max_epoch\"],\n        iters_per_epoch=len(train_loader),\n        stop_trigger=trigger, out_dir=output_path\n    )\n    manager = set_extensions(\n        manager, settings, model, device, val_loader, optimizer, eval_manager, print_progress)\n\n    # # run training.\n    run_train_loop(\n        manager, settings, model, device, train_loader,\n        optimizer, scheduler, loss_func)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Train"},{"metadata":{"trusted":true},"cell_type":"code","source":"stgs_str = \"\"\"\nglobals:\n  seed: 1086\n  device: cuda\n  max_epoch: 15\n  patience: 3\n  dataset_name: train_640x640\n  use_amp: True\n  val_fold: 0\n  debug: False\n\ndataset:\n  name: LabeledImageDatasetNumpy\n  train:\n    transform_list:\n      - [HorizontalFlip, {p: 0.5}]\n      - [VerticalFlip, {p: 0.5}]\n      - [ShiftScaleRotate, {\n          p: 0.5, shift_limit: 0.2, scale_limit: 0.2,\n          rotate_limit: 20, border_mode: 0, value: 0, mask_value: 0}]\n      - [RandomResizedCrop, {height: 512, width: 512, scale: [0.9, 1.0]}]\n      - [Cutout, {max_h_size: 51, max_w_size: 51, num_holes: 5, p: 0.5}]\n      - [Normalize, {\n          always_apply: True, max_pixel_value: 255.0,\n          mean: [0.4887381077884414], std: [0.23064819430546407]}]\n      - [ToTensorV2, {always_apply: True}]\n  val:\n    transform_list:\n      - [Normalize, {\n          always_apply: True, max_pixel_value: 255.0,\n          mean: [0.4887381077884414], std: [0.23064819430546407]}]\n      - [ToTensorV2, {always_apply: True}]\n\nloader:\n  train: {batch_size: 16, shuffle: True, num_workers: 2, pin_memory: True, drop_last: True}\n  val: {batch_size: 32, shuffle: False, num_workers: 2, pin_memory: True, drop_last: False}\n\nmodel:\n  name: MultiHeadResNet200D\n  params:\n    # base_name: resnet200D_320\n    out_dims_head: [3, 4, 3, 1]\n    pretrained: True\n\nloss: {name: BCEWithLogitsLoss, params: {}}\n\neval:\n  - {name: MyLogLoss, report_name: loss, params: {}}\n  - {name: MyROCAUC, report_name: metric, params: {average: macro}}\n\noptimizer:\n    name: Adam\n    params:\n      lr: 2.5e-04\n\nscheduler:\n  name: CosineAnnealingWarmRestarts\n  params:\n    T_0: 16\n    T_mult: 1\n\"\"\"\nstgs = yaml.safe_load(stgs_str)\n\nif stgs[\"globals\"][\"debug\"]:\n    stgs[\"globals\"][\"max_epoch\"] = 1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"stgs_list = []\nfor fold_id in FOLDS:\n    tmp_stgs = copy.deepcopy(stgs)\n    tmp_stgs[\"globals\"][\"val_fold\"] = fold_id\n    stgs_list.append(tmp_stgs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"torch.cuda.empty_cache()\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for fold_id, tmp_stgs in zip(FOLDS, stgs_list):\n    train_one_fold(tmp_stgs, train, TMP / f\"fold{fold_id}\", False)\n    torch.cuda.empty_cache()\n    gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Inference OOF"},{"metadata":{},"cell_type":"markdown","source":"### copy best models"},{"metadata":{"trusted":true},"cell_type":"code","source":"ls ../tmp","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"best_log_list = []\nfor fold_id, tmp_stgs in zip(FOLDS, stgs_list):\n    exp_dir_path = TMP / f\"fold{fold_id}\"\n    log = pd.read_json(exp_dir_path / \"log\")\n    best_log = log.iloc[[log[\"val/metric\"].idxmax()],]\n    best_epoch = best_log.epoch.values[0]\n    best_log_list.append(best_log)\n    \n    best_model_path = exp_dir_path / f\"snapshot_epoch_{best_epoch}.pth\"\n    copy_to = f\"./best_model_fold{fold_id}.pth\"\n    shutil.copy(best_model_path, copy_to)\n    \n    for p in exp_dir_path.glob(\"*.pth\"):\n        p.unlink()\n    \n    shutil.copytree(exp_dir_path, f\"./fold{fold_id}\")\n    \n    with open(f\"./fold{fold_id}/settings.yml\", \"w\") as fw:\n        yaml.dump(tmp_stgs, fw)\n    \npd.concat(best_log_list, axis=0, ignore_index=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### inference"},{"metadata":{"trusted":true},"cell_type":"code","source":"# def run_inference_loop(stgs, model, loader, device):\n#     model.to(device)\n#     model.eval()\n#     pred_list = []\n#     with torch.no_grad():\n#         for x, t in tqdm(loader):\n#             y = model(x.to(device))\n#             pred_list.append(y.sigmoid().detach().cpu().numpy())\n#             # pred_list.append(y.detach().cpu().numpy())\n        \n#     pred_arr = np.concatenate(pred_list)\n#     del pred_list\n#     return pred_arr","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# oof_pred_arr = np.zeros((len(train), N_CLASSES))\n# label_arr = train[CLASSES].values\n# score_list = []\n\n# for fold_id in range(N_FOLD):\n#     tmp_dir = Path(f\"./fold{fold_id}\")\n#     with open(tmp_dir / \"settings.yml\", \"r\") as fr:\n#         tmp_stgs = yaml.safe_load(fr)\n#     device = torch.device(tmp_stgs[\"globals\"][\"device\"])\n#     val_idx = train.query(\"fold == @fold_id\").index.values\n    \n#     # # get data_loader\n#     _, val_file_list = get_file_list_with_array(tmp_stgs, train)\n#     _, val_loader = get_dataloaders_cls(\n#         tmp_stgs, None, val_file_list, LabeledImageDatasetNumpy)\n    \n#     # # get and load model\n#     model_path =f\"./best_model_fold{fold_id}.pth\"\n#     # model = SingleHeadModel(**tmp_stgs[\"model\"][\"params\"])\n#     model = MultiHeadModel(**tmp_stgs[\"model\"][\"params\"])\n#     model.load_state_dict(torch.load(model_path, map_location=device))\n\n#     val_pred = run_inference_loop(tmp_stgs, model, val_loader, device)\n#     val_score = roc_auc_score(label_arr[val_idx], val_pred, average=\"macro\")\n#     print(f\"[fold {fold_id}] val score: {val_score:.5f}\")\n#     oof_pred_arr[val_idx] = val_pred\n#     score_list.append([fold_id, val_score])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# oof_score = roc_auc_score(label_arr, oof_pred_arr)\n# score_list.append([\"oof\", oof_score])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# pd.DataFrame(score_list, columns=[\"fold\", \"metric\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# oof_df = train.copy()\n# oof_df[CLASSES] = oof_pred_arr\n# oof_df.to_csv(\"./oof_prediction.csv\", index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}