{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Training a Classifier to detect No-call\n\n\n* credits to : https://github.com/namakemono/kaggle-birdclef-2021/blob/master/share_solution/working/build_nocall_detector.ipynb","metadata":{}},{"cell_type":"markdown","source":"# About \n\n    After clipping the audio signals into smaller chunks, a lot of the chunks do not have any audio signal in them as a lot of the audios are sparsely populated with primary bird call. Hence this will harm the training as it will make the model learn incorrectly on signal that is not at all present in the spectrogram.\n    \n    \n    Hence to avoid this scenario, we can train a no-call classifier, which classifies if a spectrogram has a bird call signal present or not. We can use this classifier on the spectrograms of the birclef 22 data, and then construct labels depending upon if there is a bird call signal present in the spectrogram, and if the probablity of no call is high , we can train it with a \"no-call\" label. ","metadata":{}},{"cell_type":"markdown","source":"\n\n**Freefield Data**\n\n\n    This dataset contains 7690 10-second audio files in a standardised format, extracted from contributions on the Freesound archive which were labelled with the \"field-recording\" tag. Note that the original tagging (as well as the audio submission) is crowdsourced, so the dataset is not guaranteed to consist purely of \"field recordings\" as might be defined by practitioners. The intention is to represent the content of an archive collection on such a topic, rather than to represent a controlled definition of such a topic.\n\n    Each audio file has a corresponding text file, containing metadata such as author and tags. The dataset has been randomly split into 10 equal-size subsets. This is so that you can perform 10-fold crossvalidation in machine-learning experiments, or can use fixed subsets of the data (e.g. use one subset for development, and others for later validation). Each of the 10 subsets has about 128 minutes of audio; the dataset totals over 21 hours of audio.\n","metadata":{}},{"cell_type":"markdown","source":"# Imports","metadata":{}},{"cell_type":"code","source":"\nimport os\nimport math\nimport time\nimport random\nimport shutil\nfrom pathlib import Path\nfrom contextlib import contextmanager\nfrom collections import defaultdict, Counter\n\n\nimport scipy as sp\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n\nfrom sklearn import preprocessing\nfrom sklearn.metrics import accuracy_score, confusion_matrix\nfrom sklearn.model_selection import StratifiedKFold\n\nfrom tqdm.auto import tqdm\nfrom functools import partial\n\nimport cv2\nfrom PIL import Image\n\n#torch\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.optim import Adam, SGD\nimport torchvision.models as models\nfrom torch.nn.parameter import Parameter\nfrom torch.utils.data import DataLoader, Dataset\nfrom torch.optim.lr_scheduler import CosineAnnealingWarmRestarts\n\n\n#augmentations\nimport albumentations as A\nfrom albumentations.pytorch import ToTensorV2\nfrom albumentations import ImageOnlyTransform\n\n\n#getting models\n!pip install timm -q\nimport timm\n\nimport warnings \nwarnings.filterwarnings('ignore')","metadata":{"execution":{"iopub.status.busy":"2022-05-07T23:54:08.669195Z","iopub.execute_input":"2022-05-07T23:54:08.669724Z","iopub.status.idle":"2022-05-07T23:54:28.300373Z","shell.execute_reply.started":"2022-05-07T23:54:08.669637Z","shell.execute_reply":"2022-05-07T23:54:28.298327Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Config","metadata":{}},{"cell_type":"code","source":"\nclass CFG:\n    print_freq=50\n    num_workers=4\n    model_name= 'resnext50_32x4d'\n    dim=(128, 281)\n    scheduler='CosineAnnealingWarmRestarts'\n    epochs=15\n    lr=1e-4\n    T_0= 5 # for CosineAnnealingWarmRestarts\n    min_lr=5e-7 # for CosineAnnealingWarmRestarts\n    batch_size=32\n    weight_decay=1e-5\n    max_grad_norm=100\n    seed=7\n    target_size=2\n    target_col='hasbird'\n    n_fold = 5\n    pretrained = True\n    device  = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n\n\n\ndef seed_torch(seed=7):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\n\nseed_torch(seed=CFG.seed)","metadata":{"execution":{"iopub.status.busy":"2022-05-07T23:54:28.306190Z","iopub.execute_input":"2022-05-07T23:54:28.308606Z","iopub.status.idle":"2022-05-07T23:54:28.385167Z","shell.execute_reply.started":"2022-05-07T23:54:28.308502Z","shell.execute_reply":"2022-05-07T23:54:28.384355Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#what device? \nCFG.device","metadata":{"execution":{"iopub.status.busy":"2022-05-07T23:54:28.389981Z","iopub.execute_input":"2022-05-07T23:54:28.392115Z","iopub.status.idle":"2022-05-07T23:54:28.403137Z","shell.execute_reply.started":"2022-05-07T23:54:28.392077Z","shell.execute_reply":"2022-05-07T23:54:28.402275Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Loading csv**\n\n    Using the dataset and spectrograms by : https://www.kaggle.com/datasets/startjapan/ff1010bird-duration7","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv('../input/ff1010bird-duration7/rich_metadata.csv')\ntrain.loc[train['hasbird']==0, 'filepath'] = '../input/ff1010bird-duration7/nocall/' + train.query('hasbird==0')['filename'] + '.npy'\ntrain.loc[train['hasbird']==1, 'filepath'] = '../input/ff1010bird-duration7/bird/' + train.query('hasbird==1')['filename'] + '.npy'\n\ntrain = train.dropna().reset_index(drop=True)","metadata":{"execution":{"iopub.status.busy":"2022-05-07T23:54:28.405367Z","iopub.execute_input":"2022-05-07T23:54:28.406359Z","iopub.status.idle":"2022-05-07T23:54:28.498598Z","shell.execute_reply.started":"2022-05-07T23:54:28.406314Z","shell.execute_reply":"2022-05-07T23:54:28.497802Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# #freefield birdcall dir\n\n# train_dir = '../input/birdclef22-p1-extracting-spectograms/Freefield_Spectrograms'\n# OUTPUT_DIR = './'\n# train = pd.read_csv('../input/birdclef22-p1-extracting-spectograms/freefield_downsampled.csv',\n#                     usecols=['has_bird_call','id','filepath'])\n\n\n# #add a filepath to retrive files\n# train.filepath=train.filepath.apply(lambda x: train_dir +'/'+ x.split('/')[-1].replace('.wav','_0.jpg'))\n\n# train.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-07T23:54:28.499667Z","iopub.execute_input":"2022-05-07T23:54:28.500052Z","iopub.status.idle":"2022-05-07T23:54:28.509378Z","shell.execute_reply.started":"2022-05-07T23:54:28.500020Z","shell.execute_reply":"2022-05-07T23:54:28.507684Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train['hasbird'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-05-07T23:54:28.516326Z","iopub.execute_input":"2022-05-07T23:54:28.516834Z","iopub.status.idle":"2022-05-07T23:54:28.531667Z","shell.execute_reply.started":"2022-05-07T23:54:28.516797Z","shell.execute_reply":"2022-05-07T23:54:28.530560Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Split data into folds**","metadata":{}},{"cell_type":"code","source":"folds = train.copy()\nfolds.reset_index(drop=True,inplace=True)\n\nFold = StratifiedKFold(n_splits=CFG.n_fold, shuffle=True, random_state=CFG.seed)\nfor n, (train_index, val_index) in enumerate(Fold.split(folds, folds[CFG.target_col])):\n    folds.loc[val_index, 'fold'] = int(n)\n    \n    \nfolds['fold'] = folds['fold'].astype(int)\nprint(folds.groupby(['fold', CFG.target_col]).size())","metadata":{"execution":{"iopub.status.busy":"2022-05-07T23:54:28.532648Z","iopub.execute_input":"2022-05-07T23:54:28.532879Z","iopub.status.idle":"2022-05-07T23:54:28.563688Z","shell.execute_reply.started":"2022-05-07T23:54:28.532849Z","shell.execute_reply":"2022-05-07T23:54:28.562986Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Helper Functions","metadata":{}},{"cell_type":"code","source":"OUTPUT_DIR='./'","metadata":{"execution":{"iopub.status.busy":"2022-05-07T23:54:28.564690Z","iopub.execute_input":"2022-05-07T23:54:28.565069Z","iopub.status.idle":"2022-05-07T23:54:28.567894Z","shell.execute_reply.started":"2022-05-07T23:54:28.565039Z","shell.execute_reply":"2022-05-07T23:54:28.567143Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_score(y_true, y_pred):\n    return accuracy_score(y_true, y_pred)\n\ndef get_confusion_matrix(y_true, y_pred):\n    return confusion_matrix(y_true, y_pred)\n\n@contextmanager\ndef timer(name):\n    t0 = time.time()\n    LOGGER.info(f'[{name}] start')\n    yield\n    LOGGER.info(f'[{name}] done in {time.time() - t0:.0f} s.')\n\n\ndef init_logger(log_file=OUTPUT_DIR+'train.log'):\n    from logging import getLogger, INFO, FileHandler,  Formatter,  StreamHandler\n    logger = getLogger(__name__)\n    logger.setLevel(INFO)\n    handler1 = StreamHandler()\n    handler1.setFormatter(Formatter(\"%(message)s\"))\n    handler2 = FileHandler(filename=log_file)\n    handler2.setFormatter(Formatter(\"%(message)s\"))\n    logger.addHandler(handler1)\n    logger.addHandler(handler2)\n    return logger\n\n\nLOGGER = init_logger()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-05-07T23:54:28.569224Z","iopub.execute_input":"2022-05-07T23:54:28.569762Z","iopub.status.idle":"2022-05-07T23:54:28.590411Z","shell.execute_reply.started":"2022-05-07T23:54:28.569720Z","shell.execute_reply":"2022-05-07T23:54:28.589666Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class AverageMeter(object):\n    \"\"\"Computes and stores the average and current value\"\"\"\n    def __init__(self):\n        self.reset()\n\n    def reset(self):\n        self.val = 0\n        self.avg = 0\n        self.sum = 0\n        self.count = 0\n\n    def update(self, val, n=1):\n        self.val = val\n        self.sum += val * n\n        self.count += n\n        self.avg = self.sum / self.count\n\n\ndef asMinutes(s):\n    m = math.floor(s / 60)\n    s -= m * 60\n    return '%dm %ds' % (m, s)\n\ndef timeSince(since, percent):\n    now = time.time()\n    s = now - since\n    es = s / (percent)\n    rs = es - s\n    return '%s (remain %s)' % (asMinutes(s), asMinutes(rs))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-05-07T23:54:28.594141Z","iopub.execute_input":"2022-05-07T23:54:28.594670Z","iopub.status.idle":"2022-05-07T23:54:28.611510Z","shell.execute_reply.started":"2022-05-07T23:54:28.594632Z","shell.execute_reply":"2022-05-07T23:54:28.610512Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Training Dataset and Transformation**","metadata":{}},{"cell_type":"code","source":"class TrainDataset(Dataset):\n    def __init__(self, df, transform=None):\n        self.df = df\n        self.file_paths = df['filepath'].values\n        self.labels = df[CFG.target_col].values\n        self.transform = transform\n        \n    def __len__(self):\n        return len(self.df)\n\n    def __getitem__(self, idx):\n        file_path = self.file_paths[idx] \n        \n        image = np.load(file_path)\n        image = image.transpose(1,2,0)\n        image = np.squeeze(image)\n        image = np.stack((image,)*3, -1)\n        \n        \n        #read image\n#         image = cv2.imread(file_path)\n#         image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n        \n#         image = cv2.imread(file_path,cv2.IMREAD_GRAYSCALE)\n#         image = np.dstack((image,image,image)) # stack to form rgb image(so we can use pretrained weights)\n        \n        #apply transformations if any\n        if self.transform:\n            augmented = self.transform(image=image)\n            image = augmented['image']\n            \n        \n        # temp check \n#         fig,ax = plt.subplots(figsize=(10,6))\n#         plt.imshow(image.reshape((128,512,3)))\n#         plt.show()\n        \n        \n        #return as tensor    \n        label = torch.tensor(self.labels[idx]).long()\n        return image, label\n    \ndef get_transforms(*, data):\n    \n    if data == 'train':\n        return A.Compose([\n            A.Resize(CFG.dim[0], CFG.dim[1]),\n            A.HorizontalFlip(p=0.5),\n            A.VerticalFlip(p=0.5),\n            A.augmentations.transforms.JpegCompression(p=0.5),\n            A.augmentations.transforms.ImageCompression(p=0.5, compression_type=A.augmentations.transforms.ImageCompression.ImageCompressionType.WEBP),\n            A.Normalize(\n                mean=[0.485, 0.456, 0.406],\n                std=[0.229, 0.224, 0.225],\n            ),\n            ToTensorV2(),\n        ])\n\n    elif data == 'valid':\n        return A.Compose([\n            A.Resize(CFG.dim[0], CFG.dim[1]),\n            A.Normalize(\n                mean=[0.485, 0.456, 0.406],\n                std=[0.229, 0.224, 0.225],\n            ),\n            ToTensorV2(),\n        ])","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2022-05-07T23:54:28.612921Z","iopub.execute_input":"2022-05-07T23:54:28.613449Z","iopub.status.idle":"2022-05-07T23:54:28.639167Z","shell.execute_reply.started":"2022-05-07T23:54:28.613404Z","shell.execute_reply":"2022-05-07T23:54:28.638354Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Checking some sample spectrograms**","metadata":{}},{"cell_type":"code","source":"train_dataset = TrainDataset(train, \n                             transform=get_transforms(data='train'))\n\n\nfig,ax = plt.subplots(5,1,figsize=(15,30))\nfor i,ax1 in enumerate(ax):\n    i += np.random.randint(low=0,high = 1500)\n    image, label = train_dataset[i]\n    \n    \n    # reshape for visualization (from depth * height * width -> height*width * depth)\n    im = np.zeros(shape = (CFG.dim[0],CFG.dim[1],3))  \n    im[:,:,0] = image[0]\n    im[:,:,1] = image[1]\n    im[:,:,2] = image[2]\n    \n    ax1.imshow(im,cmap='jet')\n    ax1.set_title(f'label: {label}')\n\nplt.tight_layout()    \nplt.show() ","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-05-08T00:01:09.229719Z","iopub.execute_input":"2022-05-08T00:01:09.230448Z","iopub.status.idle":"2022-05-08T00:01:10.096802Z","shell.execute_reply.started":"2022-05-08T00:01:09.230398Z","shell.execute_reply":"2022-05-08T00:01:10.096126Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Model**","metadata":{}},{"cell_type":"code","source":"\n\nclass CustomResNext(nn.Module):\n    def __init__(self, model_name='resnext50_32x4d', pretrained=True):\n        super().__init__()\n        self.model = timm.create_model(model_name, pretrained=pretrained)\n        n_features = self.model.fc.in_features\n        self.model.fc = nn.Linear(n_features, CFG.target_size)\n\n    def forward(self, x):\n        x = self.model(x)\n        return x","metadata":{"execution":{"iopub.status.busy":"2022-05-08T00:01:16.046101Z","iopub.execute_input":"2022-05-08T00:01:16.046387Z","iopub.status.idle":"2022-05-08T00:01:16.052897Z","shell.execute_reply.started":"2022-05-08T00:01:16.046353Z","shell.execute_reply":"2022-05-08T00:01:16.052203Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_scheduler(optimizer):\n    '''cosine annealing scheduler'''\n    scheduler = CosineAnnealingWarmRestarts(optimizer, \n                                            T_0=CFG.T_0, \n                                            T_mult=1, \n                                            eta_min=CFG.min_lr, \n                                            last_epoch=-1)\n    return scheduler\n","metadata":{"execution":{"iopub.status.busy":"2022-05-08T00:01:16.186713Z","iopub.execute_input":"2022-05-08T00:01:16.187032Z","iopub.status.idle":"2022-05-08T00:01:16.191831Z","shell.execute_reply.started":"2022-05-08T00:01:16.187002Z","shell.execute_reply":"2022-05-08T00:01:16.191008Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Training and validation Functions**","metadata":{}},{"cell_type":"code","source":"\ndef train_fn(train_loader, model, criterion, optimizer, epoch, scheduler, device):\n    '''perform training on one epoch of data.'''\n    \n    \n    batch_time = AverageMeter()\n    data_time = AverageMeter()\n    losses = AverageMeter()\n    scores = AverageMeter()\n    \n        \n    # switch to train mode\n    model.train()\n    start = end = time.time()\n    global_step = 0\n    for step, (images, labels) in enumerate(train_loader):\n        # measure data loading time\n        data_time.update(time.time() - end)\n        \n        #load data\n        images = images.to(device)\n        labels = labels.to(device)\n        batch_size = labels.size(0)\n        \n        #forward pass\n        y_preds = model(images)\n        \n        #calculate loss\n        loss = criterion(y_preds, labels)\n        \n        \n        # record loss\n        losses.update(loss.item(), batch_size)\n        loss.backward()\n        \n        grad_norm = torch.nn.utils.clip_grad_norm_(model.parameters(),\n                                                   CFG.max_grad_norm)\n        \n        optimizer.step()\n        optimizer.zero_grad()\n        global_step += 1\n        # measure elapsed time\n        batch_time.update(time.time() - end)\n        end = time.time()\n        if step % CFG.print_freq == 0 or step == (len(train_loader)-1):\n            print('Epoch: [{0}][{1}/{2}] '\n                  'Data {data_time.val:.3f} ({data_time.avg:.3f}) '\n                  'Elapsed {remain:s} '\n                  'Loss: {loss.val:.4f}({loss.avg:.4f}) '\n                  'Grad: {grad_norm:.4f}  '\n                  .format(\n                   epoch+1, step+1, len(train_loader), batch_time=batch_time,\n                   data_time=data_time, loss=losses,\n                   remain=timeSince(start, float(step+1)/len(train_loader)),\n                   grad_norm=grad_norm,\n                   ))\n            \n    return losses.avg\n\ndef valid_fn(valid_loader, model, criterion, device):\n    '''perform validation'''\n    \n    batch_time = AverageMeter()\n    data_time = AverageMeter()\n    losses = AverageMeter()\n    scores = AverageMeter()\n    # switch to evaluation mode\n    model.eval()\n    preds = []\n    start = end = time.time()\n    for step, (images, labels) in enumerate(valid_loader):\n        # measure data loading time\n        data_time.update(time.time() - end)\n        images = images.to(device)\n        labels = labels.to(device)\n        batch_size = labels.size(0)\n        # compute loss\n        with torch.no_grad():\n            y_preds = model(images)\n        loss = criterion(y_preds, labels)\n        losses.update(loss.item(), batch_size)\n        \n        \n        # record accuracy\n        preds.append(y_preds.softmax(1).to('cpu').numpy())\n        \n        # measure elapsed time\n        batch_time.update(time.time() - end)\n        end = time.time()\n        \n        if step % CFG.print_freq == 0 or step == (len(valid_loader)-1):\n            print('EVAL: [{0}/{1}] '\n                  'Data {data_time.val:.3f} ({data_time.avg:.3f}) '\n                  'Elapsed {remain:s} '\n                  'Loss: {loss.val:.4f}({loss.avg:.4f}) '\n                  .format(\n                   step+1, len(valid_loader), batch_time=batch_time,\n                   data_time=data_time, loss=losses,\n                   remain=timeSince(start, float(step+1)/len(valid_loader)),\n                   ))\n    predictions = np.concatenate(preds)\n    return losses.avg, predictions\n\n\ndef inference(model, states, test_loader, device):\n    '''inference'''\n    model.to(device)\n    tk0 = tqdm(enumerate(test_loader), total=len(test_loader))\n    probs = []\n    for i, (images) in tk0:\n        images = images.to(device)\n        avg_preds = []\n        for state in states:\n            model.load_state_dict(state['model'])\n            model.eval()\n            with torch.no_grad():\n                y_preds = model(images)\n            avg_preds.append(y_preds.softmax(1).to('cpu').numpy())\n        avg_preds = np.mean(avg_preds, axis=0)\n        probs.append(avg_preds)\n    probs = np.concatenate(probs)\n    return probs","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-05-08T00:01:16.566743Z","iopub.execute_input":"2022-05-08T00:01:16.567008Z","iopub.status.idle":"2022-05-08T00:01:16.589387Z","shell.execute_reply.started":"2022-05-08T00:01:16.566978Z","shell.execute_reply":"2022-05-08T00:01:16.588493Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Training Procedure / loop**","metadata":{}},{"cell_type":"code","source":"\ndef train_loop(train_folds, valid_folds):\n\n    LOGGER.info(f\"========== training ==========\")\n\n    # ====================================================\n    # loader\n    # ====================================================\n    train_dataset = TrainDataset(train_folds, \n                                 transform=get_transforms(data='train'))\n    valid_dataset = TrainDataset(valid_folds, \n                                 transform=get_transforms(data='valid'))\n\n    train_loader = DataLoader(train_dataset, \n                              batch_size=CFG.batch_size, \n                              shuffle=False, \n                              num_workers=CFG.num_workers, pin_memory=True, drop_last=True)\n    valid_loader = DataLoader(valid_dataset, \n                              batch_size=CFG.batch_size, \n                              shuffle=False, \n                              num_workers=CFG.num_workers, pin_memory=True, drop_last=False)\n    \n    # ====================================================\n    # model & optimizer\n    # ====================================================\n    model = CustomResNext(CFG.model_name, pretrained=True)\n    model.to(CFG.device)\n    \n    optimizer = Adam(model.parameters(), lr=CFG.lr, weight_decay=CFG.weight_decay, amsgrad=False)\n    scheduler = get_scheduler(optimizer)\n\n    # ====================================================\n    # loop\n    # ====================================================\n    criterion = nn.CrossEntropyLoss()\n\n    best_score = 0.\n    best_loss = np.inf\n    \n    scores = []\n    \n    for epoch in range(CFG.epochs):\n        \n        start_time = time.time()\n        \n        # train\n        avg_loss = train_fn(train_loader, \n                            model, \n                            criterion, \n                            optimizer, \n                            epoch, \n                            scheduler, \n                            CFG.device)\n        \n        # eval\n        avg_val_loss, preds = valid_fn(valid_loader, model, criterion, CFG.device)\n        valid_labels = valid_folds[CFG.target_col].values\n        \n        scheduler.step()\n\n        # scoring\n        score = get_score(valid_labels, preds.argmax(1))\n\n        elapsed = time.time() - start_time\n\n        LOGGER.info(f'Epoch {epoch+1} - avg_train_loss: {avg_loss:.4f}  avg_val_loss: {avg_val_loss:.4f}  time: {elapsed:.0f}s')\n        LOGGER.info(f'Epoch {epoch+1} - Accuracy(validation): {score}')\n        \n        scores.append(score)\n        \n        \n        # save the model weights with the best score \n        if score > best_score:\n            best_score = score\n            LOGGER.info(f'Epoch {epoch+1} - Save Best Score: {best_score:.4f} Model')\n            torch.save({'model': model.state_dict(), \n                        'preds': preds},\n                        OUTPUT_DIR+f'{CFG.model_name}_best.pth')\n    \n    check_point = torch.load(OUTPUT_DIR+f'{CFG.model_name}_best.pth')\n    valid_folds[[str(c) for c in range(CFG.target_size)]] = check_point['preds']\n    valid_folds['preds'] = check_point['preds'].argmax(1)\n\n    return valid_folds, scores\n\n","metadata":{"execution":{"iopub.status.busy":"2022-05-08T00:01:16.913737Z","iopub.execute_input":"2022-05-08T00:01:16.914441Z","iopub.status.idle":"2022-05-08T00:01:16.928539Z","shell.execute_reply.started":"2022-05-08T00:01:16.914396Z","shell.execute_reply":"2022-05-08T00:01:16.927519Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_result(result_df):\n    \n    preds = result_df['preds'].values\n    labels = result_df[CFG.target_col].values\n    score = get_score(labels, preds)\n    LOGGER.info(f'Score: {score:<.5f}')\n\ndef get_confusion_mat(result_df):\n    preds = result_df['preds'].values\n    labels = result_df[CFG.target_col].values\n    matrix = get_confusion_matrix(labels, preds)\n    print('TN', matrix[0,0])\n    print('FP', matrix[0,1])\n    print('FN', matrix[1,0])\n    print('TP', matrix[1,1])\n","metadata":{"execution":{"iopub.status.busy":"2022-05-08T00:01:17.068652Z","iopub.execute_input":"2022-05-08T00:01:17.069199Z","iopub.status.idle":"2022-05-08T00:01:17.075709Z","shell.execute_reply.started":"2022-05-08T00:01:17.069161Z","shell.execute_reply":"2022-05-08T00:01:17.074708Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n\ndef main(fold):\n    '''run training on the dataset, with validation on the input fold'''\n    \n    # train \n    train_folds = folds.query(f'fold!={fold}').reset_index(drop=True)\n    valid_folds = folds.query(f'fold=={fold}').reset_index(drop=False)\n    oof_df, scores = train_loop(train_folds, valid_folds)\n    \n    \n    # CV result\n    LOGGER.info(f\"========== CV ==========\")\n    get_result(oof_df)\n    get_confusion_mat(oof_df)\n    \n    \n    \n    # save result\n    oof_df.to_csv(OUTPUT_DIR+'oof_df.csv', index=False)\n    plt.plot([i for i in range(CFG.epochs)], scores)\n    plt.title('valid score')\n    plt.show()\n\n","metadata":{"execution":{"iopub.status.busy":"2022-05-08T00:01:17.268671Z","iopub.execute_input":"2022-05-08T00:01:17.268986Z","iopub.status.idle":"2022-05-08T00:01:17.275232Z","shell.execute_reply.started":"2022-05-08T00:01:17.268956Z","shell.execute_reply":"2022-05-08T00:01:17.274331Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training","metadata":{}},{"cell_type":"markdown","source":"**Training with fold 5 as the validation set**","metadata":{}},{"cell_type":"code","source":"if __name__ == '__main__':\n    main(0)","metadata":{"execution":{"iopub.status.busy":"2022-05-08T00:01:18.467033Z","iopub.execute_input":"2022-05-08T00:01:18.467740Z","iopub.status.idle":"2022-05-08T00:04:35.010073Z","shell.execute_reply.started":"2022-05-08T00:01:18.467699Z","shell.execute_reply":"2022-05-08T00:04:35.009168Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**We can use the saved model on the birdclef spectrograms to filter out the no-bird calls for training.**","metadata":{}}]}