{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"!nvidia-smi","metadata":{"id":"C0vy_lvxVQf5","outputId":"0ad3c95b-49a6-4e37-c750-8b7f79e76f57","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Notes","metadata":{}},{"cell_type":"markdown","source":"Days back, I've shared this [infernece kernel](https://www.kaggle.com/kneroma/clean-fast-simple-bird-identifier-inference). But its weights are static as you can't retrain the model. In this work, I'm gonna release the training notebook which is almost my internal training pipeline. I removed some experimentation ideas to make things clearer and straightforward. Don't mind adding new ideas at your side as well.","metadata":{"id":"py6GyjyZVark"}},{"cell_type":"markdown","source":"To make the training faster, we cached the training set into RAM. The whole training records are already [converted into handy  melspecs images](https://www.kaggle.com/kneroma/kkiller-birdclef-2021). These images are from 7 seconds extracts (training on 7 seconds seems to be more effective than 5 seconds). Longer records are truncated into random 7x10 seconds.\n\n**If one is interessted in to the whole records' melspecs** (no truncation):\n* https://www.kaggle.com/kneroma/kkiller-birdclef-mels-computer-d7-part1\n* https://www.kaggle.com/kneroma/kkiller-birdclef-mels-computer-d7-part2\n* https://www.kaggle.com/kneroma/kkiller-birdclef-mels-computer-d7-part3\n* https://www.kaggle.com/kneroma/kkiller-birdclef-mels-computer-d7-part4","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Tips & suggestions\n* You can choose a wide set of models from the **get_model** interface : [\"resnest*\", \"resnet*\", \"resnext*\", \"efficientnet*\" ...]\n* You can change the learning rate scheduler: OneCycle ? ReduceOnPlateau ?\n* Adds secondary labels\n* Use train & test metadata (dates, positions (longitude, latitude), ...)\n* Add melspecs augmentation","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**For Colab training, you just have to uncomment the first cells**","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Versions","metadata":{}},{"cell_type":"markdown","source":"* **v1** : initial version\n* **v3** : enable training on whole (no truncation) record melspecs","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# from google.colab import drive\n# drive.mount('/content/drive')","metadata":{"id":"oYPb42V-Vaza","outputId":"f9846f5e-bb8b-407c-d15e-bab4b8604301","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ! pip install --upgrade --force-reinstall --no-deps  kaggle > /dev/null\n# ! mkdir ~/.kaggle\n# ! cp \"/content/drive/My Drive/Kaggle/kaggle.json\" ~/.kaggle/\n# ! chmod 600 ~/.kaggle/kaggle.json","metadata":{"id":"IRn4fOj5XDp6","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# %%time\n\n# import os\n# if not os.path.exists(\"/content/datasets/audio_images\"):\n#   !mkdir datasets\n#   !kaggle datasets download -d kneroma/kkiller-birdclef-2021\n#   !unzip /content//kkiller-birdclef-2021.zip -d datasets","metadata":{"id":"KAnAPecOXDtf","outputId":"fb981400-b8c5-45bc-9e7d-5d4a77f2550e","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install -q pysndfx SoundFile audiomentations pretrainedmodels efficientnet_pytorch resnest ","metadata":{"id":"Yn1Ybf15VAqW","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import sys\nsys.path.append('../input/timm-pytorch-image-models/pytorch-image-models-master')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\nimport librosa as lb\nimport librosa.display as lbd\nimport soundfile as sf\nfrom  soundfile import SoundFile\nimport pandas as pd\nfrom  IPython.display import Audio\nfrom pathlib import Path\n\nimport torch\nfrom torch import nn, optim\nfrom  torch.utils.data import Dataset, DataLoader\n\nfrom resnest.torch import resnest50\n\nimport timm\nfrom matplotlib import pyplot as plt\n\nimport os, random, gc\nimport re, time, json\nfrom  ast import literal_eval\n\n\nfrom IPython.display import Audio\nfrom sklearn.metrics import label_ranking_average_precision_score\n\nfrom tqdm.notebook import tqdm\nimport joblib","metadata":{"id":"2dt7oG43VAqc","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from efficientnet_pytorch import EfficientNet\nimport pretrainedmodels\nimport resnest.torch as resnest_torch","metadata":{"id":"162Vl9uxe1Mj","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def seed_everything(seed=42):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\nseed_everything()","metadata":{"id":"Q39ZsGAhVAqe","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"NUM_CLASSES = 397\nSR = 32_000\nDURATION = 7\n\nMAX_READ_SAMPLES = 5 # Each record will have 10 melspecs at most, you can increase this on Colab with High Memory Enabled\n\n# # For colab\n# DATA_ROOT = Path(\"/content/datasets/\")\n# TRAIN_IMAGES_ROOT = Path(\"/content/datasets/audio_images\")\n# TRAIN_LABELS_FILE = Path(\"/content/datasets/rich_train_metadata.csv\")\n# MODEL_ROOT = Path(\"/content/drive/My Drive/Kaggle/BirdClef2021/models\")\n\nDATA_ROOT = Path(\"../input/birdclef-2021\")\n# TRAIN_IMAGES_ROOT = Path(\"../input/kkiller-birdclef-2021/audio_images\")\n# TRAIN_LABELS_FILE = Path(\"../input/kkiller-birdclef-2021/rich_train_metadata.csv\")\n\nMEL_PATHS = sorted(Path(\"../input\").glob(\"kkiller-birdclef-mels-computer-d7-part?/rich_train_metadata.csv\"))\nTRAIN_LABEL_PATHS = sorted(Path(\"../input\").glob(\"kkiller-birdclef-mels-computer-d7-part?/LABEL_IDS.json\"))\n\nMODEL_ROOT = Path(\".\")","metadata":{"id":"2NfkUn9SCWs6","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"TRAIN_BATCH_SIZE = 100\nTRAIN_NUM_WORKERS = 2\n\nVAL_BATCH_SIZE = 128\nVAL_NUM_WORKERS = 2\n\nDEVICE = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n\nprint(\"Device:\", DEVICE)","metadata":{"id":"Iu56f-7VVAqf","outputId":"0f3fa344-0ed4-47d8-f3c0-218cf3bf5a78","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"id":"GuyjwJnACWs6","outputId":"cdaca87a-567c-4299-9840-7b3cb06be13f","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_df(mel_paths=MEL_PATHS, train_label_paths=TRAIN_LABEL_PATHS):\n  df = None\n  LABEL_IDS = {}\n    \n  for file_path in mel_paths:\n    temp = pd.read_csv(str(file_path), index_col=0)\n    temp[\"impath\"] = temp.apply(lambda row: file_path.parent/\"audio_images/{}/{}.npy\".format(row.primary_label, row.filename), axis=1) \n    df = temp if df is None else df.append(temp)\n    \n  df[\"secondary_labels\"] = df[\"secondary_labels\"].apply(literal_eval)\n\n  for file_path in train_label_paths:\n    with open(str(file_path)) as f:\n      LABEL_IDS.update(json.load(f))\n\n  return LABEL_IDS, df","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# df = pd.read_csv(TRAIN_LABELS_FILE, nrows=None)\n# df[\"secondary_labels\"] = df[\"secondary_labels\"].apply(literal_eval)\n# LABEL_IDS = {label: label_id for label_id,label in enumerate(sorted(df[\"primary_label\"].unique()))}\n\n# print(df.shape)\n# df.head()","metadata":{"id":"Kmh6xx5_NCjJ","outputId":"3e92e880-2e37-46c7-ffc0-5551e2641e7b","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"LABEL_IDS, df = get_df()\n\nprint(df.shape)\ndf.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"id":"W_Xf_natBGGL"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df[\"primary_label\"].value_counts()","metadata":{"id":"ZRz-DwbNVAqg","outputId":"42aa9ba1-7fc3-42b6-e818-c4a04277fc51","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df[\"label_id\"].min(), df[\"label_id\"].max()","metadata":{"id":"n68WeAa0VAqh","outputId":"31adfe31-5bfb-473c-e79d-272d5a8b8cf3","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"id":"xh4hfWZhuglm"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_model(name, num_classes=NUM_CLASSES):\n    \"\"\"\n    Loads a pretrained model. \n    Supports ResNest, ResNext-wsl, EfficientNet, ResNext and ResNet.\n\n    Arguments:\n        name {str} -- Name of the model to load\n\n    Keyword Arguments:\n        num_classes {int} -- Number of classes to use (default: {1})\n\n    Returns:\n        torch model -- Pretrained model\n    \"\"\"\n    if \"resnest\" in name:\n        model = getattr(resnest_torch, name)(pretrained=True)\n    elif \"wsl\" in name:\n        model = torch.hub.load(\"facebookresearch/WSL-Images\", name)\n    elif name.startswith(\"resnext\") or  name.startswith(\"resnet\"):\n        model = torch.hub.load(\"pytorch/vision:v0.6.0\", name, pretrained=True)\n    elif name.startswith(\"tf_efficientnet_b\"):\n        model = getattr(timm.models.efficientnet, name)(pretrained=True)\n    elif \"efficientnet-b\" in name:\n        model = EfficientNet.from_pretrained(name)\n    else:\n        model = pretrainedmodels.__dict__[name](pretrained='imagenet')\n\n    if hasattr(model, \"fc\"):\n        nb_ft = model.fc.in_features\n        model.fc = nn.Linear(nb_ft, num_classes)\n    elif hasattr(model, \"_fc\"):\n        nb_ft = model._fc.in_features\n        model._fc = nn.Linear(nb_ft, num_classes)\n    elif hasattr(model, \"classifier\"):\n        nb_ft = model.classifier.in_features\n        model.classifier = nn.Linear(nb_ft, num_classes)\n    elif hasattr(model, \"last_linear\"):\n        nb_ft = model.last_linear.in_features\n        model.last_linear = nn.Linear(nb_ft, num_classes)\n\n    return model","metadata":{"id":"OGPDuihmVAqi","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def load_data(df):\n    def load_row(row):\n        # impath = TRAIN_IMAGES_ROOT/f\"{row.primary_label}/{row.filename}.npy\"\n        return row.filename, np.load(str(row.impath))[:MAX_READ_SAMPLES]\n    pool = joblib.Parallel(4)\n    mapper = joblib.delayed(load_row)\n    tasks = [mapper(row) for row in df.itertuples(False)]\n    res = pool(tqdm(tasks))\n    res = dict(res)\n    return res","metadata":{"id":"7HYQwAyBCWs8","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# We cache the train set to reduce training time\n\naudio_image_store = load_data(df)\nlen(audio_image_store)","metadata":{"id":"Vw19bB7mCWs9","outputId":"09a5e374-7e5c-4c92-91e4-b60313cbb1a9","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"shape:\", next(iter(audio_image_store.values())).shape)\nlbd.specshow(next(iter(audio_image_store.values()))[0])","metadata":{"id":"4TNYmT7XCWs9","outputId":"b5052897-ca12-4c25-a49c-3799a88b9f9d","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"id":"_tSt9iC7CWs9"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.Series([len(x) for x in audio_image_store.values()]).value_counts()","metadata":{"id":"bUUqP5KMBZkc","outputId":"ab2ca6c4-8aed-4b02-ed52-485b7af70ae1","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"id":"rOTGe4dbBapl"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class BirdClefDataset(Dataset):\n\n    def __init__(self, audio_image_store, meta, sr=SR, is_train=True, num_classes=NUM_CLASSES, duration=DURATION):\n        \n        self.audio_image_store = audio_image_store\n        self.meta = meta.copy().reset_index(drop=True)\n        self.sr = sr\n        self.is_train = is_train\n        self.num_classes = num_classes\n        self.duration = duration\n        self.audio_length = self.duration*self.sr\n    \n    @staticmethod\n    def normalize(image):\n        image = image.astype(\"float32\", copy=False) / 255.0\n        image = np.stack([image, image, image])\n        return image\n\n    def __len__(self):\n        return len(self.meta)\n    \n    def __getitem__(self, idx):\n        row = self.meta.iloc[idx]\n        image = self.audio_image_store[row.filename]\n\n        image = image[np.random.choice(len(image))]\n        image = self.normalize(image)\n        \n        \n        t = np.zeros(self.num_classes, dtype=np.float32) + 0.0025 # Label smoothing\n        t[row.label_id] = 0.995\n        \n        return image, t","metadata":{"id":"OWSkCXyhCWs-","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ds = BirdClefDataset(audio_image_store, meta=df, sr=SR, duration=DURATION, is_train=True)\nlen(df)","metadata":{"id":"Np-56XrXVAqm","outputId":"84e45885-200f-4c98-a9a1-23ef459e5ade","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x, y = ds[np.random.choice(len(ds))]\n# x, y = ds[0]\nx.shape, y.shape, np.where(y >= 0.5)","metadata":{"id":"UNVUxIMpVAqm","outputId":"96e0930f-51f7-4268-9939-2d81999c002f","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lbd.specshow(x[0])","metadata":{"id":"bPwrCoRyCWs-","outputId":"388a8eaf-f373-4e05-c57d-be4f6975c890","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y[:5]","metadata":{"id":"GBr32Q9FCWs_","outputId":"141ba3fd-99b1-4878-8462-2a6eb3cb5e19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training the model","metadata":{"id":"F56zXq8CVAqn"}},{"cell_type":"code","source":"","metadata":{"id":"THm438BwMTeR"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def one_step( xb, yb, net, criterion, optimizer, scheduler=None):\n  xb, yb = xb.to(DEVICE), yb.to(DEVICE)\n        \n  optimizer.zero_grad()\n  o = net(xb)\n  loss = criterion(o, yb)\n  loss.backward()\n  optimizer.step()\n  \n  with torch.no_grad():\n      l = loss.item()\n\n      o = o.sigmoid()\n      yb = (yb > 0.5 )*1.0\n      lrap = label_ranking_average_precision_score(yb.cpu().numpy(), o.cpu().numpy())\n\n      o = (o > 0.5)*1.0\n\n      prec = (o*yb).sum()/(1e-6 + o.sum())\n      rec = (o*yb).sum()/(1e-6 + yb.sum())\n      f1 = 2*prec*rec/(1e-6+prec+rec)\n\n  if  scheduler is not None:\n    scheduler.step()\n\n  return l, lrap, f1.item(), rec.item(), prec.item()","metadata":{"id":"9Kjy1uquIGZw","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"@torch.no_grad()\ndef evaluate(net, criterion, val_laoder):\n    net.eval()\n\n    os, y = [], []\n    val_laoder = tqdm(val_laoder, leave = False, total=len(val_laoder))\n\n    for icount, (xb, yb) in  enumerate(val_laoder):\n\n        y.append(yb.to(DEVICE))\n\n        xb = xb.to(DEVICE)\n        o = net(xb)\n\n        os.append(o)\n\n    y = torch.cat(y)\n    o = torch.cat(os)\n\n    l = criterion(o, y).item()\n    \n    o = o.sigmoid()\n    y = (y > 0.5)*1.0\n\n    lrap = label_ranking_average_precision_score(y.cpu().numpy(), o.cpu().numpy())\n\n    o = (o > 0.5)*1.0\n\n    prec = ((o*y).sum()/(1e-6 + o.sum())).item()\n    rec = ((o*y).sum()/(1e-6 + y.sum())).item()\n    f1 = 2*prec*rec/(1e-6+prec+rec)\n\n    return l, lrap, f1, rec, prec, \n    ","metadata":{"id":"q9v79J0pvXy1","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def one_epoch(net, criterion, optimizer, scheduler, train_laoder, val_laoder):\n  net.train()\n  l, lrap, prec, rec, f1, icount = 0.,0.,0.,0., 0., 0\n  train_laoder = tqdm(train_laoder, leave = False)\n  epoch_bar = train_laoder\n  \n  for (xb, yb) in  epoch_bar:\n      # epoch_bar.set_description(\"----|----|----|----|---->\")\n      _l, _lrap, _f1, _rec, _prec = one_step(xb, yb, net, criterion, optimizer)\n      l += _l\n      lrap += _lrap\n      f1 += _f1\n      rec += _rec\n      prec += _prec\n\n      icount += 1\n        \n      if hasattr(epoch_bar, \"set_postfix\") and not icount%10:\n          epoch_bar.set_postfix(\n            loss=\"{:.6f}\".format(l/icount),\n            lrap=\"{:.3f}\".format(lrap/icount),\n            prec=\"{:.3f}\".format(prec/icount),\n            rec=\"{:.3f}\".format(rec/icount),\n            f1=\"{:.3f}\".format(f1/icount),\n          )\n  \n  scheduler.step()\n\n  l /= icount\n  lrap /= icount\n  f1 /= icount\n  rec /= icount\n  prec /= icount\n  \n  l_val, lrap_val, f1_val, rec_val, prec_val = evaluate(net, criterion, val_laoder)\n  \n  return (l, l_val), (lrap, lrap_val), (f1, f1_val), (rec, rec_val), (prec, prec_val)","metadata":{"id":"qeDgf4LdLWGN","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"id":"IY4ET5V0RMJm"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class AutoSave:\n  def __init__(self, top_k=2, metric=\"f1\", mode=\"min\", root=None, name=\"ckpt\"):\n    self.top_k = top_k\n    self.logs = []\n    self.metric = metric\n    self.mode = mode\n    self.root = Path(root or MODEL_ROOT)\n    assert self.root.exists()\n    self.name = name\n\n    self.top_models = []\n    self.top_metrics = []\n\n  def log(self, model, metrics):\n    metric = metrics[self.metric]\n    rank = self.rank(metric)\n\n    self.top_metrics.insert(rank+1, metric)\n    if len(self.top_metrics) > self.top_k:\n      self.top_metrics.pop(0)\n\n    self.logs.append(metrics)\n    self.save(model, metric, rank, metrics[\"epoch\"])\n\n\n  def save(self, model, metric, rank, epoch):\n    t = time.strftime(\"%Y%m%d%H%M%S\")\n    name = \"{}_epoch_{:02d}_{}_{:.04f}_{}\".format(self.name, epoch, self.metric, metric, t)\n    name = re.sub(r\"[^\\w_-]\", \"\", name) + \".pth\"\n    path = self.root.joinpath(name)\n\n    old_model = None\n    self.top_models.insert(rank+1, name)\n    if len(self.top_models) > self.top_k:\n      old_model = self.root.joinpath(self.top_models[0])\n      self.top_models.pop(0)      \n\n    torch.save(model.state_dict(), path.as_posix())\n\n    if old_model is not None:\n      old_model.unlink()\n\n    self.to_json()\n\n\n  def rank(self, val):\n    r = -1\n    for top_val in self.top_metrics:\n      if val <= top_val:\n        return r\n      r += 1\n\n    return r\n  \n  def to_json(self):\n    # t = time.strftime(\"%Y%m%d%H%M%S\")\n    name = \"{}_logs\".format(self.name)\n    name = re.sub(r\"[^\\w_-]\", \"\", name) + \".json\"\n    path = self.root.joinpath(name)\n\n    with path.open(\"w\") as f:\n      json.dump(self.logs, f, indent=2)\n","metadata":{"id":"Cz7XPBvtPLO1","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def one_fold(model_name, fold, train_set, val_set, epochs=20, save=True, save_root=None):\n\n  save_root = Path(save_root) or MODEL_ROOT\n\n  saver = AutoSave(root=save_root, name=f\"birdclef_{model_name}_fold{fold}\", metric=\"f1_val\")\n\n  net = get_model(model_name).to(DEVICE)\n\n  criterion = nn.BCEWithLogitsLoss()\n\n  optimizer = optim.Adam(net.parameters(), lr=8e-4)\n  scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, eta_min=1e-5, T_max=epochs)\n\n  train_data = BirdClefDataset(audio_image_store, meta=df.iloc[train_set].reset_index(drop=True),\n                           sr=SR, duration=DURATION, is_train=True)\n  train_laoder = DataLoader(train_data, batch_size=TRAIN_BATCH_SIZE, num_workers=TRAIN_NUM_WORKERS, shuffle=True, pin_memory=True)\n\n  val_data = BirdClefDataset(audio_image_store, meta=df.iloc[val_set].reset_index(drop=True),  sr=SR, duration=DURATION, is_train=False)\n  val_laoder = DataLoader(val_data, batch_size=VAL_BATCH_SIZE, num_workers=VAL_NUM_WORKERS, shuffle=False)\n\n  epochs_bar = tqdm(list(range(epochs)), leave=False)\n  for epoch  in epochs_bar:\n    epochs_bar.set_description(f\"--> [EPOCH {epoch:02d}]\")\n    net.train()\n\n    (l, l_val), (lrap, lrap_val), (f1, f1_val), (rec, rec_val), (prec, prec_val) = one_epoch(\n        net=net,\n        criterion=criterion,\n        optimizer=optimizer,\n        scheduler=scheduler,\n        train_laoder=train_laoder,\n        val_laoder=val_laoder,\n      )\n\n    epochs_bar.set_postfix(\n    loss=\"({:.6f}, {:.6f})\".format(l, l_val),\n    prec=\"({:.3f}, {:.3f})\".format(prec, prec_val),\n    rec=\"({:.3f}, {:.3f})\".format(rec, rec_val),\n    f1=\"({:.3f}, {:.3f})\".format(f1, f1_val),\n    lrap=\"({:.3f}, {:.3f})\".format(lrap, lrap_val),\n    )\n\n    print(\n        \"[{epoch:02d}] loss: {loss} lrap: {lrap} f1: {f1} rec: {rec} prec: {prec}\".format(\n            epoch=epoch,\n            loss=\"({:.6f}, {:.6f})\".format(l, l_val),\n            prec=\"({:.3f}, {:.3f})\".format(prec, prec_val),\n            rec=\"({:.3f}, {:.3f})\".format(rec, rec_val),\n            f1=\"({:.3f}, {:.3f})\".format(f1, f1_val),\n            lrap=\"({:.3f}, {:.3f})\".format(lrap, lrap_val),\n        )\n    )\n\n    if save:\n      metrics = {\n          \"loss\": l, \"lrap\": lrap, \"f1\": f1, \"rec\": rec, \"prec\": prec,\n          \"loss_val\": l_val, \"lrap_val\": lrap_val, \"f1_val\": f1_val, \"rec_val\": rec_val, \"prec_val\": prec_val,\n          \"epoch\": epoch,\n      }\n\n      saver.log(net, metrics)","metadata":{"id":"8X1dt_aWNi6F","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"id":"NiEiYTjaSCaH","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def train(model_name, epochs=20, save=True, n_splits=5, seed=177, save_root=None, suffix=\"\", folds=None):\n  gc.collect()\n  torch.cuda.empty_cache()\n\n  save_root = save_root or MODEL_ROOT/f\"{model_name}{suffix}\"\n  save_root.mkdir(exist_ok=True, parents=True)\n  \n  fold_bar = tqdm(df.reset_index().groupby(\"fold\").index.apply(list).items(), total=df.fold.max()+1)\n  \n  for fold, val_set in fold_bar:\n      if folds and not fold in folds:\n        continue\n      \n      print(f\"\\n############################### [FOLD {fold}]\")\n      fold_bar.set_description(f\"[FOLD {fold}]\")\n      train_set = np.setdiff1d(df.index, val_set)\n        \n      one_fold(model_name, fold=fold, train_set=train_set , val_set=val_set , epochs=epochs, save=save, save_root=save_root)\n    \n      gc.collect()\n      torch.cuda.empty_cache()","metadata":{"id":"ljqr4e2zQmzB","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"MODEL_NAMES = [\n      \"resnest50\",\n] ","metadata":{"id":"aqN6xL7gVAqq","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for model_name in MODEL_NAMES:\n  print(\"\\n\\n###########################################\", model_name.upper())\n  try:\n    train(model_name, epochs=20, suffix=f\"_sr{SR}_d{DURATION}_v1_v1\", folds=[0])\n  except Exception as e:\n    # print(f\"Error {model_name} : \\n{e}\")\n    raise ValueError() from  e","metadata":{"id":"WyFnAQGWELb_","outputId":"d676c647-65e4-43d1-a8f3-69f3d949e727","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"id":"Q46u71ImEL4E","trusted":true},"execution_count":null,"outputs":[]}]}