{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"!nvidia-smi","metadata":{"id":"C0vy_lvxVQf5","outputId":"0ad3c95b-49a6-4e37-c750-8b7f79e76f57","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#!pip install perceiver-pytorch","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Notes","metadata":{}},{"cell_type":"markdown","source":"Days back, I've shared this [infernece kernel](https://www.kaggle.com/kneroma/clean-fast-simple-bird-identifier-inference). But its weights are static as you can't retrain the model. In this work, I'm gonna release the training notebook which is almost my internal training pipeline. I removed some experimentation ideas to make things clearer and straightforward. Don't mind adding new ideas at your side as well.","metadata":{"id":"py6GyjyZVark"}},{"cell_type":"markdown","source":"To make the training faster, we cached the training set into RAM. The whole training records are already [converted into handy  melspecs images](https://www.kaggle.com/kneroma/kkiller-birdclef-2021). These images are from 7 seconds extracts (training on 7 seconds seems to be more effective than 5 seconds). Longer records are truncated into random 7x10 seconds.\n\n**If one is interessted in to the whole records' melspecs** (no truncation):\n* https://www.kaggle.com/kneroma/kkiller-birdclef-mels-computer-d7-part1\n* https://www.kaggle.com/kneroma/kkiller-birdclef-mels-computer-d7-part2\n* https://www.kaggle.com/kneroma/kkiller-birdclef-mels-computer-d7-part3\n* https://www.kaggle.com/kneroma/kkiller-birdclef-mels-computer-d7-part4","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Tips & suggestions\n* You can choose a wide set of models from the **get_model** interface : [\"resnest*\", \"resnet*\", \"resnext*\", \"efficientnet*\" ...]\n* You can change the learning rate scheduler: OneCycle ? ReduceOnPlateau ?\n* Adds secondary labels\n* Use train & test metadata (dates, positions (longitude, latitude), ...)\n* Add melspecs augmentation","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**For Colab training, you just have to uncomment the first cells**","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Versions","metadata":{}},{"cell_type":"markdown","source":"* **v1** : initial version\n* **v3** : enable training on whole (no truncation) record melspecs","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# from google.colab import drive\n# drive.mount('/content/drive')","metadata":{"id":"oYPb42V-Vaza","outputId":"f9846f5e-bb8b-407c-d15e-bab4b8604301","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ! pip install --upgrade --force-reinstall --no-deps  kaggle > /dev/null\n# ! mkdir ~/.kaggle\n# ! cp \"/content/drive/My Drive/Kaggle/kaggle.json\" ~/.kaggle/\n# ! chmod 600 ~/.kaggle/kaggle.json","metadata":{"id":"IRn4fOj5XDp6","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# %%time\n\n# import os\n# if not os.path.exists(\"/content/datasets/audio_images\"):\n#   !mkdir datasets\n#   !kaggle datasets download -d kneroma/kkiller-birdclef-2021\n#   !unzip /content//kkiller-birdclef-2021.zip -d datasets\nimport sys\nsys.path.append('../input/timm-latest')","metadata":{"id":"KAnAPecOXDtf","outputId":"fb981400-b8c5-45bc-9e7d-5d4a77f2550e","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install -q pysndfx SoundFile audiomentations pretrainedmodels efficientnet_pytorch resnest","metadata":{"id":"Yn1Ybf15VAqW","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\nimport librosa as lb\nimport librosa.display as lbd\nimport soundfile as sf\nfrom  soundfile import SoundFile\nimport pandas as pd\nfrom  IPython.display import Audio\nfrom pathlib import Path\n\nimport torch\nfrom torch import nn, optim\nfrom  torch.utils.data import Dataset, DataLoader\nimport timm\n\nfrom resnest.torch import resnest50\n\nfrom matplotlib import pyplot as plt\n\nimport os, random, gc\nimport re, time, json\nfrom  ast import literal_eval\n\n\nfrom IPython.display import Audio\nfrom sklearn.metrics import label_ranking_average_precision_score\n\nfrom tqdm.notebook import tqdm\nimport joblib","metadata":{"id":"2dt7oG43VAqc","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from efficientnet_pytorch import EfficientNet\n#from perceiver_pytorch import Perceiver\nimport pretrainedmodels\nimport resnest.torch as resnest_torch","metadata":{"id":"162Vl9uxe1Mj","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def seed_everything(seed=42):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\nseed_everything()","metadata":{"id":"Q39ZsGAhVAqe","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"NUM_CLASSES = 397\nSR = 32_000\nDURATION = 5\nN_MELS = 256\nEPSILON_FP16 = 0.005\nUSE_FOCAL_LOSS = False\nSECONDARY_FACTOR = 1\nSECONDARY_FACTORS = [1, 1]\nMODEL_NAMES = [\n      \"rexnet_200\",\n      \"rexnet_150\",\n] \nMAX_READ_SAMPLES = 9999999999 # Each record will have 10 melspecs at most, you can increase this on Colab with High Memory Enabled\n\n# # For colab\n# DATA_ROOT = Path(\"/content/datasets/\")\n# TRAIN_IMAGES_ROOT = Path(\"/content/datasets/audio_images\")\n# TRAIN_LABELS_FILE = Path(\"/content/datasets/rich_train_metadata.csv\")\n# MODEL_ROOT = Path(\"/content/drive/My Drive/Kaggle/BirdClef2021/models\")\nAUG_REPLACEMENT = ['birdclef-mels-computer-', \n                   ['birdclef-mels-audiomentation-', 'birdclef-mels-audiomentation2-']]\nAUG_RATE = 0.25\n\nDATA_ROOT = Path(\"../input/birdclef-2021\")\n# TRAIN_IMAGES_ROOT = Path(\"../input/kkiller-birdclef-2021/audio_images\")\n# TRAIN_LABELS_FILE = Path(\"../input/kkiller-birdclef-2021/rich_train_metadata.csv\")\n\nMEL_PATHS = sorted(Path(\"../input\").glob(\"birdclef-mels-computer-*/rich_train_metadata.csv\"))\nTRAIN_LABEL_PATHS = sorted(Path(\"../input\").glob(\"birdclef-mels-computer-*/LABEL_IDS.json\"))\n\nMODEL_ROOT = Path(\".\")","metadata":{"id":"2NfkUn9SCWs6","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"TRAIN_BATCH_SIZE = 32\nTRAIN_NUM_WORKERS = 4\n\nVAL_BATCH_SIZE = 32\nVAL_NUM_WORKERS = 4\n\nEPOCHS = 20\nFOLDS = [1]\n\nDEVICE = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n\nprint(\"Device:\", DEVICE)","metadata":{"id":"Iu56f-7VVAqf","outputId":"0f3fa344-0ed4-47d8-f3c0-218cf3bf5a78","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"id":"GuyjwJnACWs6","outputId":"cdaca87a-567c-4299-9840-7b3cb06be13f","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_df(mel_paths=MEL_PATHS, train_label_paths=TRAIN_LABEL_PATHS):\n  df = None\n  LABEL_IDS = {}\n    \n  for file_path in mel_paths:\n    temp = pd.read_csv(str(file_path), index_col=0)\n    temp[\"impath\"] = temp.apply(lambda row: file_path.parent/\"audio_images/{}/{}.npy\".format(row.primary_label, row.filename), axis=1) \n    df = temp if df is None else df.append(temp)\n    \n  df[\"secondary_labels\"] = df[\"secondary_labels\"].apply(literal_eval)\n\n  for file_path in train_label_paths:\n    with open(str(file_path)) as f:\n      LABEL_IDS.update(json.load(f))\n\n  return LABEL_IDS, df","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# df = pd.read_csv(TRAIN_LABELS_FILE, nrows=None)\n# df[\"secondary_labels\"] = df[\"secondary_labels\"].apply(literal_eval)\n# LABEL_IDS = {label: label_id for label_id,label in enumerate(sorted(df[\"primary_label\"].unique()))}\n\n# print(df.shape)\n# df.head()","metadata":{"id":"Kmh6xx5_NCjJ","outputId":"3e92e880-2e37-46c7-ffc0-5551e2641e7b","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"LABEL_IDS, df = get_df()\n\nprint(df.shape)\ndf.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"id":"W_Xf_natBGGL"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df[\"primary_label\"].value_counts()","metadata":{"id":"ZRz-DwbNVAqg","outputId":"42aa9ba1-7fc3-42b6-e818-c4a04277fc51","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df[\"label_id\"].min(), df[\"label_id\"].max()","metadata":{"id":"n68WeAa0VAqh","outputId":"31adfe31-5bfb-473c-e79d-272d5a8b8cf3","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class BCEFocalLoss(nn.Module):\n    def __init__(self, alpha=0.25, gamma=2.0):\n        super().__init__()\n        self.alpha = alpha\n        self.gamma = gamma\n\n    def forward(self, preds, targets):\n        bce_loss = nn.BCEWithLogitsLoss(reduction='none')(preds, targets)\n        probas = torch.sigmoid(preds)\n        loss = targets * self.alpha * \\\n            (1. - probas)**self.gamma * bce_loss + \\\n            (1. - targets) * probas**self.gamma * bce_loss\n        loss = loss.mean()\n        return loss\n    \nclass BCEFocal2WayLoss(nn.Module):\n    def __init__(self, weights=[1, 1], class_weights=None):\n        super().__init__()\n\n        self.focal = BCEFocalLoss()\n\n        self.weights = weights\n\n    def forward(self, input, target):\n        input_ = input[\"logit\"]\n        target = target.float()\n\n        framewise_output = input[\"framewise_logit\"]\n        clipwise_output_with_max, _ = framewise_output.max(dim=1)\n\n        loss = self.focal(input_, target)\n        aux_loss = self.focal(clipwise_output_with_max, target)\n\n        return self.weights[0] * loss + self.weights[1] * aux_loss","metadata":{"id":"xh4hfWZhuglm","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_model(name, num_classes=NUM_CLASSES):\n    \"\"\"\n    Loads a pretrained model. \n    Supports ResNest, ResNext-wsl, EfficientNet, ResNext and ResNet.\n\n    Arguments:\n        name {str} -- Name of the model to load\n\n    Keyword Arguments:\n        num_classes {int} -- Number of classes to use (default: {1})\n\n    Returns:\n        torch model -- Pretrained model\n    \"\"\"\n    if \"resnest\" in name:\n        #model = getattr(resnest_torch, name)(pretrained=True)\n        model = getattr(timm.models.resnest, name)(pretrained=True)\n    elif \"resnet\" in name or \"resnext\" in name:\n        model = getattr(timm.models.resnet, name)(pretrained=True)\n    elif \"densenet\" in name:\n        model = getattr(timm.models.densenet, name)(pretrained=True) \n    elif \"rexnet\" in name:\n        model = getattr(timm.models.rexnet, name)(pretrained=True, num_classes=NUM_CLASSES)\n    elif \"nf_\" in name:\n        model = getattr(timm.models.nfnet, name)(pretrained=True, num_classes=NUM_CLASSES)\n    elif \"vovnet\" in name:\n        model = getattr(timm.models.vovnet, name)(pretrained=True, num_classes=NUM_CLASSES)\n    elif name.startswith(\"dla\"):\n        model = getattr(timm.models.dla, name)(pretrained=True)\n    elif \"res2next\" in name:\n        model = getattr(timm.models.res2net, name)(pretrained=True)\n    elif \"regnet\" in name: \n        model = getattr(timm.models.regnet, name)(pretrained=True)\n    elif \"coat\" in name:\n        model = getattr(timm.models.coat, name)(pretrained=True)\n    elif \"wsl-image\" in name:\n        model = torch.hub.load(\"facebookresearch/WSL-Images\", name)\n    elif name.startswith(\"resnext\") or  name.startswith(\"resnet\"):\n        model = torch.hub.load(\"pytorch/vision:v0.6.0\", name, pretrained=True)\n    elif \"efficientnet_\" in name or \"mixnet\" in name:\n        model = getattr(timm.models.efficientnet, name)(pretrained=True)\n    elif \"efficientnet-b\" in name:\n        model = EfficientNet.from_pretrained(name)\n    else:\n        model = pretrainedmodels.__dict__[name](pretrained='imagenet')\n\n    if hasattr(model, \"fc\"):\n        nb_ft = model.fc.in_features\n        model.fc = nn.Linear(nb_ft, num_classes)\n    elif hasattr(model, \"_fc\"):\n        nb_ft = model._fc.in_features\n        model._fc = nn.Linear(nb_ft, num_classes)\n    elif hasattr(model, \"classifier\"):\n        nb_ft = model.classifier.in_features\n        model.classifier = nn.Linear(nb_ft, num_classes)\n    elif hasattr(model, \"last_linear\"):\n        nb_ft = model.last_linear.in_features\n        model.last_linear = nn.Linear(nb_ft, num_classes)\n\n    return model","metadata":{"id":"OGPDuihmVAqi","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def load_data(df):\n    def load_row(row):\n        # impath = TRAIN_IMAGES_ROOT/f\"{row.primary_label}/{row.filename}.npy\"\n        return row.filename, str(row.impath)\n    pool = joblib.Parallel(4)\n    mapper = joblib.delayed(load_row)\n    tasks = [mapper(row) for row in df.itertuples(False)]\n    res = pool(tqdm(tasks))\n    res = dict(res)\n    return res","metadata":{"id":"7HYQwAyBCWs8","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# We cache the train set to reduce training time\n\naudio_image_store = load_data(df)\nlen(audio_image_store)","metadata":{"id":"Vw19bB7mCWs9","outputId":"09a5e374-7e5c-4c92-91e4-b60313cbb1a9","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#print(\"shape:\", next(iter(audio_image_store.values())).shape)\n#lbd.specshow(next(iter(audio_image_store.values()))[0])","metadata":{"id":"4TNYmT7XCWs9","outputId":"b5052897-ca12-4c25-a49c-3799a88b9f9d","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"id":"_tSt9iC7CWs9"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.Series([len(x) for x in audio_image_store.values()]).value_counts()","metadata":{"id":"bUUqP5KMBZkc","outputId":"ab2ca6c4-8aed-4b02-ed52-485b7af70ae1","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"id":"rOTGe4dbBapl"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class BirdClefDataset(Dataset):\n\n    def __init__(self, audio_image_store, meta, sr=SR, is_train=True, num_classes=NUM_CLASSES, duration=DURATION, \n                 aug_replacement=AUG_REPLACEMENT, aug_rate=AUG_RATE):\n        \n        self.audio_image_store = audio_image_store\n        self.meta = meta.copy().reset_index(drop=True)\n        self.sr = sr\n        self.is_train = is_train\n        self.num_classes = num_classes\n        self.duration = duration\n        self.audio_length = self.duration*self.sr\n        self.aug_replacement = aug_replacement\n        self.aug_rate = aug_rate\n    \n    @staticmethod\n    def normalize(image):\n        image = image.astype(\"float32\", copy=False) / 255.0\n        image = np.stack([image, image, image])\n        return image\n\n    def __len__(self):\n        return len(self.meta)\n    \n    def __getitem__(self, idx):\n        row = self.meta.iloc[idx]\n        impath = self.audio_image_store[row.filename]\n        if random.random() > self.aug_rate:\n            impath = impath.replace(self.aug_replacement[0], random.choice(self.aug_replacement[1]))\n        image = np.load(impath)[:MAX_READ_SAMPLES]\n\n        image = image[np.random.choice(len(image))]\n        image = self.normalize(image)\n        \n        #if USE_1ST_LOSS:\n        #    secondary_labels = np.zeros(self.num_classes, dtype=np.float32)\n        #    for label in row.secondary_labels:\n        #        secondary_labels[LABEL_IDS[label]] = 1.0\n        #    all_labels = np.zeros(self.num_classes, dtype=np.float32)\n        #    all_labels[row.label_id] = 1.0\n        #    all_labels += secondary_labels\n        #    t = np.array([all_labels, secondary_labels], dtype=np.float32)\n        #else:\n        t = np.zeros(self.num_classes, dtype=np.float32) + 0.0025 # Label smoothing\n        if SECONDARY_FACTOR > 0:\n            for label in row.secondary_labels:\n                t[LABEL_IDS[label]] = SECONDARY_FACTOR\n        t[row.label_id] = 0.995\n        \n        return image, t\n    \ndef mono_to_color(X, eps=1e-6, mean=None, std=None):\n    mean = mean or X.mean()\n    std = std or X.std()\n    X = (X - mean) / (std + eps)\n    \n    _min, _max = X.min(), X.max()\n\n    if (_max - _min) > eps:\n        V = np.clip(X, _min, _max)\n        V = 255 * (V - _min) / (_max - _min)\n        V = V.astype(np.uint8)\n    else:\n        V = np.zeros_like(X, dtype=np.uint8)\n\n    return V\n\ndef crop_or_pad(y, length):\n    if len(y) < length:\n        y = np.concatenate([y, length - np.zeros(len(y))])\n    elif len(y) > length:\n        y = y[:length]\n    return y\n    \nclass MelSpecComputer:\n    def __init__(self, sr, n_mels, fmin, fmax, **kwargs):\n        self.sr = sr\n        self.n_mels = n_mels\n        self.fmin = fmin\n        self.fmax = fmax\n        kwargs[\"n_fft\"] = kwargs.get(\"n_fft\", self.sr//10)\n        kwargs[\"hop_length\"] = kwargs.get(\"hop_length\", self.sr//(10*4))\n        self.kwargs = kwargs\n\n    def __call__(self, y):\n\n        melspec = lb.feature.melspectrogram(\n            y, sr=self.sr, n_mels=self.n_mels, fmin=self.fmin, fmax=self.fmax, **self.kwargs,\n        )\n\n        melspec = lb.power_to_db(melspec).astype(np.float32)\n        return melspec\n    \nclass TestBirdCLEFDataset(Dataset):\n    def __init__(self, data, sr=SR, n_mels=N_MELS, fmin=0, fmax=None, duration=DURATION, step=None, res_type=\"kaiser_fast\", resample=True):\n        \n        self.data = data\n        \n        self.sr = sr\n        self.n_mels = n_mels\n        self.fmin = fmin\n        self.fmax = fmax or self.sr//2\n\n        self.duration = duration\n        self.audio_length = self.duration*self.sr\n        self.step = step or self.audio_length\n        \n        self.res_type = res_type\n        self.resample = resample\n\n        self.mel_spec_computer = MelSpecComputer(sr=self.sr, n_mels=self.n_mels, fmin=self.fmin,\n                                                 fmax=self.fmax)\n    def __len__(self):\n        return len(self.data)\n    \n    @staticmethod\n    def normalize(image):\n        image = image.astype(\"float32\", copy=False) / 255.0\n        image = np.stack([image, image, image])\n        return image\n    \n    def audio_to_image(self, audio):\n        melspec = self.mel_spec_computer(audio) \n        image = mono_to_color(melspec)\n        image = self.normalize(image)\n        return image\n\n    def read_file(self, filepath):\n        audio, orig_sr = sf.read(filepath, dtype=\"float32\")\n\n        if self.resample and orig_sr != self.sr:\n            audio = lb.resample(audio, orig_sr, self.sr, res_type=self.res_type)\n          \n        audios = []\n        for i in range(self.audio_length, len(audio) + self.step, self.step):\n            start = max(0, i - self.audio_length)\n            end = start + self.audio_length\n            audios.append(audio[start:end])\n            \n        if len(audios[-1]) < self.audio_length:\n            audios = audios[:-1]\n            \n        images = [self.audio_to_image(audio) for audio in audios]\n        images = np.stack(images)\n        \n        return images\n    \n        \n    def __getitem__(self, idx):\n        return self.read_file(self.data.loc[idx, \"filepath\"])","metadata":{"id":"OWSkCXyhCWs-","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ds = BirdClefDataset(audio_image_store, meta=df, sr=SR, duration=DURATION, is_train=True)\nlen(df)","metadata":{"id":"Np-56XrXVAqm","outputId":"84e45885-200f-4c98-a9a1-23ef459e5ade","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#x, y = ds[np.random.choice(len(ds))]\n# x, y = ds[0]\n#x.shape, y.shape, np.where(y >= 0.5)","metadata":{"id":"UNVUxIMpVAqm","outputId":"96e0930f-51f7-4268-9939-2d81999c002f","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lbd.specshow(x[0])","metadata":{"id":"bPwrCoRyCWs-","outputId":"388a8eaf-f373-4e05-c57d-be4f6975c890","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y[:5]","metadata":{"id":"GBr32Q9FCWs_","outputId":"141ba3fd-99b1-4878-8462-2a6eb3cb5e19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training the model","metadata":{"id":"F56zXq8CVAqn"}},{"cell_type":"code","source":"TEST_AUDIO_ROOT = Path(\"../input/birdclef-2021/train_soundscapes\")\nSAMPLE_SUB_PATH = None\n# SAMPLE_SUB_PATH = \"../input/birdclef-2021/sample_submission.csv\"\nTARGET_PATH = Path(\"../input/birdclef-2021/train_soundscape_labels.csv\")\nTHRESHS = [n/100 for n in range(15, 85)]\n\ndf_train = pd.read_csv(\"../input/birdclef-2021/train_metadata.csv\")\n\nLABEL_IDS = {label: label_id for label_id,label in enumerate(sorted(df_train[\"primary_label\"].unique()))}\nINV_LABEL_IDS = {val: key for key,val in LABEL_IDS.items()}\nLABEL_IDS['rocpig1'] = LABEL_IDS['rocpig']","metadata":{"id":"THm438BwMTeR","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def one_step( xb, yb, net, criterion, optimizer, scheduler=None):\n  try:\n      xb, yb = xb.to(DEVICE), yb.to(DEVICE)\n  except:\n      xb = xb.to(DEVICE)\n  optimizer.zero_grad()\n  o = net(xb)\n  loss = criterion(o, yb)\n  loss.backward()\n  optimizer.step()\n  \n  with torch.no_grad():\n      l = loss.item()\n        \n      o = o.sigmoid()\n      #if USE_1ST_LOSS:\n      #  yb = (yb[:, 0] > 0.5) * 1.0\n      #else:\n      yb = (yb > 0.5) * 1.0\n      lrap = label_ranking_average_precision_score(yb.cpu().numpy(), o.cpu().numpy())\n\n      o = (o > 0.5)*1.0\n\n      prec = (o*yb).sum()/(1e-6 + o.sum())\n      rec = (o*yb).sum()/(1e-6 + yb.sum())\n      f1 = 2*prec*rec/(1e-6+prec+rec)\n\n  if  scheduler is not None:\n    scheduler.step()\n\n  return l, lrap, f1.item(), rec.item(), prec.item()","metadata":{"id":"9Kjy1uquIGZw","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def predict(nets, test_data, names=True):\n    preds = []\n    with torch.no_grad():\n        for idx in  tqdm(list(range(len(test_data)))):\n            xb = torch.from_numpy(test_data[idx]).to(DEVICE)\n            pred = 0.\n            for i in range(len(nets)):\n                o = nets[i](xb)\n                o = torch.sigmoid(o)\n\n                pred += o\n\n            pred /= len(nets)\n            \n            if names:\n                pred = get_bird_names(get_thresh_preds(pred))\n\n            preds.append(pred)\n    return preds\n\ndef get_metrics(s_true, s_pred):\n    s_true = set(s_true.split())\n    s_pred = set(s_pred.split())\n    n, n_true, n_pred = len(s_true.intersection(s_pred)), len(s_true), len(s_pred)\n    \n    prec = n/n_pred\n    rec = n/n_true\n    f1 = 2*prec*rec/(prec + rec) if prec + rec else 0\n    \n    return {\"f1\": f1, \"prec\": prec, \"rec\": rec, \"n_true\": n_true, \"n_pred\": n_pred, \"n\": n}\n\ndata = pd.DataFrame(\n     [(path.stem, *path.stem.split(\"_\"), path) for path in Path(TEST_AUDIO_ROOT).glob(\"*.ogg\")],\n    columns = [\"filename\", \"id\", \"site\", \"date\", \"filepath\"]\n)\ntest_data = TestBirdCLEFDataset(data=data)\n\ndef preds_as_df(preds, data=data):\n    sub = {\n        \"row_id\": [],\n        \"birds\": [],\n    }\n    \n    for row, pred in zip(data.itertuples(False), preds):\n        row_id = [f\"{row.id}_{row.site}_{5*i}\" for i in range(1, len(pred)+1)]\n        sub[\"birds\"] += pred\n        sub[\"row_id\"] += row_id\n        \n    sub = pd.DataFrame(sub)\n    \n    if SAMPLE_SUB_PATH:\n        sample_sub = pd.read_csv(SAMPLE_SUB_PATH, usecols=[\"row_id\"])\n        sub = sample_sub.merge(sub, on=\"row_id\", how=\"left\")\n        sub[\"birds\"] = sub[\"birds\"].fillna(\"nocall\")\n    return sub\n\n@torch.no_grad()\ndef get_thresh_preds(out, thresh=None):\n    thresh = thresh or THRESH\n    o = (-out).argsort(1)\n    npreds = (out > thresh).sum(1)\n    preds = []\n    for oo, npred in zip(o, npreds):\n        preds.append(oo[:npred].cpu().numpy().tolist())\n    return preds\n\ndef get_bird_names(preds):\n    bird_names = []\n    for pred in preds:\n        if not pred:\n            bird_names.append(\"nocall\")\n        else:\n            bird_names.append(\" \".join([INV_LABEL_IDS[bird_id] for bird_id in pred]))\n    return bird_names\n\n@torch.no_grad()\ndef evaluate(net, test_data=test_data):\n    net.eval()\n    pred_probas = predict([net], test_data, names=False)\n\n    top_metrics = {\"f1\": 0}\n    for thresh in THRESHS:\n        preds = [get_bird_names(get_thresh_preds(pred, thresh=thresh)) for pred in pred_probas]\n        sub = preds_as_df(preds)\n        sub_target = pd.read_csv(TARGET_PATH)\n        sub_target = sub_target.merge(sub, how=\"left\", on=\"row_id\")\n\n        #print(sub_target[\"birds_x\"].notnull().sum(), sub_target[\"birds_x\"].notnull().sum())\n        assert sub_target[\"birds_x\"].notnull().all()\n        assert sub_target[\"birds_y\"].notnull().all()\n\n        df_metrics = pd.DataFrame([get_metrics(s_true, s_pred) for s_true, s_pred in zip(sub_target.birds_x, sub_target.birds_y)])\n        metrics = df_metrics.mean().to_dict()\n        if top_metrics[\"f1\"] < metrics[\"f1\"]:\n            top_metrics = metrics\n            top_metrics[\"thresh\"] = thresh\n    print(\"top_metrics:\", top_metrics)\n\n    return top_metrics[\"f1\"], top_metrics[\"prec\"], top_metrics[\"rec\"]","metadata":{"id":"q9v79J0pvXy1","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def one_epoch(net, criterion, optimizer, scheduler, train_laoder):\n  net.train()\n  l, lrap, prec, rec, f1, icount = 0.,0.,0.,0., 0., 0\n  train_laoder = tqdm(train_laoder, leave = False)\n  epoch_bar = train_laoder\n  \n  for (xb, yb) in  epoch_bar:\n      # epoch_bar.set_description(\"----|----|----|----|---->\")\n      _l, _lrap, _f1, _rec, _prec = one_step(xb, yb, net, criterion, optimizer)\n      l += _l\n      lrap += _lrap\n      f1 += _f1\n      rec += _rec\n      prec += _prec\n\n      icount += 1\n        \n      if hasattr(epoch_bar, \"set_postfix\") and not icount%10:\n          epoch_bar.set_postfix(\n            loss=\"{:.6f}\".format(l/icount),\n            lrap=\"{:.3f}\".format(lrap/icount),\n            prec=\"{:.3f}\".format(prec/icount),\n            rec=\"{:.3f}\".format(rec/icount),\n            f1=\"{:.3f}\".format(f1/icount),\n          )\n  \n  scheduler.step()\n\n  l /= icount\n  lrap /= icount\n  f1 /= icount\n  rec /= icount\n  prec /= icount\n  \n  f1_val, prec_val, rec_val = evaluate(net)\n  \n  return (f1, f1_val), (rec, rec_val), (prec, prec_val)","metadata":{"id":"qeDgf4LdLWGN","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"id":"IY4ET5V0RMJm"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class AutoSave:\n  def __init__(self, top_k=5, metric=\"f1\", mode=\"min\", root=None, name=\"ckpt\"):\n    self.top_k = top_k\n    self.logs = []\n    self.metric = metric\n    self.mode = mode\n    self.root = Path(root or MODEL_ROOT)\n    assert self.root.exists()\n    self.name = name\n\n    self.top_models = []\n    self.top_metrics = []\n\n  def log(self, model, metrics):\n    metric = metrics[self.metric]\n    rank = self.rank(metric)\n\n    self.top_metrics.insert(rank+1, metric)\n    if len(self.top_metrics) > self.top_k:\n      self.top_metrics.pop(0)\n\n    self.logs.append(metrics)\n    self.save(model, metric, rank, metrics[\"epoch\"])\n\n\n  def save(self, model, metric, rank, epoch):\n    t = time.strftime(\"%Y%m%d%H%M%S\")\n    name = \"{}_epoch_{:02d}_{}_{:.04f}_{}\".format(self.name, epoch, self.metric, metric, t)\n    name = re.sub(r\"[^\\w_-]\", \"\", name) + \".pth\"\n    path = self.root.joinpath(name)\n\n    old_model = None\n    self.top_models.insert(rank+1, name)\n    if len(self.top_models) > self.top_k:\n      old_model = self.root.joinpath(self.top_models[0])\n      self.top_models.pop(0)      \n\n    torch.save(model.state_dict(), path.as_posix())\n\n    if old_model is not None:\n      old_model.unlink()\n\n    self.to_json()\n\n\n  def rank(self, val):\n    r = -1\n    for top_val in self.top_metrics:\n      if val <= top_val:\n        return r\n      r += 1\n\n    return r\n  \n  def to_json(self):\n    # t = time.strftime(\"%Y%m%d%H%M%S\")\n    name = \"{}_logs\".format(self.name)\n    name = re.sub(r\"[^\\w_-]\", \"\", name) + \".json\"\n    path = self.root.joinpath(name)\n\n    with path.open(\"w\") as f:\n      json.dump(self.logs, f, indent=2)\n","metadata":{"id":"Cz7XPBvtPLO1","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def one_fold(model_name, fold, train_set, val_set, epochs=20, save=True, save_root=None):\n\n  save_root = Path(save_root) or MODEL_ROOT\n\n  saver = AutoSave(root=save_root, name=f\"birdclef_{model_name}_fold{fold}\", metric=\"f1_val\")\n\n  net = get_model(model_name).to(DEVICE)\n\n  if USE_FOCAL_LOSS:\n    criterion = BCEFocalLoss()\n  else:\n    criterion = nn.BCEWithLogitsLoss()\n\n  optimizer = optim.Adam(net.parameters(), lr=8e-4)\n  scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, eta_min=1e-5, T_max=epochs)\n\n  train_data = BirdClefDataset(audio_image_store, meta=df.iloc[train_set].reset_index(drop=True),\n                           sr=SR, duration=DURATION, is_train=True)\n  train_laoder = DataLoader(train_data, batch_size=TRAIN_BATCH_SIZE, num_workers=TRAIN_NUM_WORKERS, shuffle=True, pin_memory=True)\n\n  #val_data = BirdClefDataset(audio_image_store, meta=df.iloc[val_set].reset_index(drop=True),  sr=SR, duration=DURATION, is_train=False)\n  #val_laoder = DataLoader(val_data, batch_size=VAL_BATCH_SIZE, num_workers=VAL_NUM_WORKERS, shuffle=False)\n\n  epochs_bar = tqdm(list(range(epochs)), leave=False)\n  for epoch  in epochs_bar:\n    epochs_bar.set_description(f\"--> [EPOCH {epoch:02d}]\")\n    net.train()\n\n    (f1, f1_val), (rec, rec_val), (prec, prec_val) = one_epoch(\n        net=net,\n        criterion=criterion,\n        optimizer=optimizer,\n        scheduler=scheduler,\n        train_laoder=train_laoder,\n      )\n\n    epochs_bar.set_postfix(\n    #loss=\"({:.6f}, {:.6f})\".format(l, l_val),\n    prec=\"({:.3f}, {:.3f})\".format(prec, prec_val),\n    rec=\"({:.3f}, {:.3f})\".format(rec, rec_val),\n    f1=\"({:.3f}, {:.3f})\".format(f1, f1_val),\n    #lrap=\"({:.3f}, {:.3f})\".format(lrap, lrap_val),\n    )\n\n    print(\n        \"[{epoch:02d}] f1: {f1} rec: {rec} prec: {prec}\".format(\n            epoch=epoch,\n            #loss=\"({:.6f}, {:.6f})\".format(l, l_val),\n            prec=\"({:.3f}, {:.3f})\".format(prec, prec_val),\n            rec=\"({:.3f}, {:.3f})\".format(rec, rec_val),\n            f1=\"({:.3f}, {:.3f})\".format(f1, f1_val),\n            #lrap=\"({:.3f}, {:.3f})\".format(lrap, lrap_val),\n        )\n    )\n\n    if save:\n      metrics = {\n          \"f1\": f1, \"rec\": rec, \"prec\": prec,\n          \"f1_val\": f1_val, \"rec_val\": rec_val, \"prec_val\": prec_val,\n          \"epoch\": epoch,\n      }\n\n      saver.log(net, metrics)","metadata":{"id":"8X1dt_aWNi6F","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"id":"NiEiYTjaSCaH","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def train(model_name, epochs=20, save=True, n_splits=5, seed=177, save_root=None, suffix=\"\", folds=None):\n  gc.collect()\n  torch.cuda.empty_cache()\n\n  save_root = save_root or MODEL_ROOT/f\"{model_name}{suffix}\"\n  save_root.mkdir(exist_ok=True, parents=True)\n  \n  train_set = df.index\n\n  one_fold(model_name, fold=0, train_set=train_set , val_set=None , epochs=epochs, save=save, save_root=save_root)\n\n  gc.collect()\n  torch.cuda.empty_cache()","metadata":{"id":"ljqr4e2zQmzB","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"id":"aqN6xL7gVAqq","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in range(len(MODEL_NAMES)):\n  model_name = MODEL_NAMES[i]\n  SECONDARY_FACTOR = SECONDARY_FACTORS[i]\n  print(\"\\n\\n###########################################\", model_name.upper())\n  try:\n    train(model_name, epochs=EPOCHS, suffix=f\"_sr{SR}_d{DURATION}_v1_v1\", folds=FOLDS)\n  except Exception as e:\n    # print(f\"Error {model_name} : \\n{e}\")\n    raise ValueError() from  e","metadata":{"id":"WyFnAQGWELb_","outputId":"d676c647-65e4-43d1-a8f3-69f3d949e727","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"id":"Q46u71ImEL4E","trusted":true},"execution_count":null,"outputs":[]}]}