{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Birdsong Pytorch Baseline: ResNeSt50-fast (Training)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## About\n\nIn this notebook, I try ResNeSt, which is the one of state of the art in image recognition.  \n\nFor the fair comparison with @hidehisaarai1213 's [great baseline](https://www.kaggle.com/hidehisaarai1213/inference-pytorch-birdcall-resnet-baseline), I used a model with the same depth and as the same experimental settings as possible. But There are some differences mainly because of the GPU resource limitation.\n\nThe experimental settings are as follows:\n\n* Randomly crop 5 seconds for each train audio clip each epoch.\n* No augmentation.\n* Used pretrained weight of _`ResNeSt50-fast-1s1x64d`_ provided by the authors at [their repository](https://github.com/zhanghang1989/ResNeSt).\n* Used `BCELoss`\n* Trained **_50_** epoch and saved the weight which got best **_loss_** (this is because f1 score relies on thresholds.)\n* `Adam` optimizer (`lr=0.001`) with `CosineAnnealingLR` (`T_max=10`).\n* Used `StratifiedKFold(n_splits=5)` to split dataset and used only first fold\n* `batch_size`: **_50_**\n* melspectrogram parameters\n  - `n_mels`: 128\n  - `fmin`: 20\n  - `fmax`: 16000\n* image size: 224x547\n\nI forked a lot of codes such as preprocessing from @hidehisaarai1213 's [notebook](https://www.kaggle.com/hidehisaarai1213/inference-pytorch-birdcall-resnet-baseline) and [GitHub repository](https://github.com/koukyo1994/kaggle-birdcall-resnet-baseline-training). Many thanks!!!\n\n\n### Note\n\n#### about dataset\nI prepared resmpaled train dataset for this notebook, see more details in:\nhttps://www.kaggle.com/c/birdsong-recognition/discussion/164197\n\n\n#### about custom packages\nIn this **_training notebook_**, I used two custom packages, `pytorch-pfn-extras` for training and the authors' official implementation of `ResNeSt` for building model.  \nOn the other hand, as stated in [code requirements](https://www.kaggle.com/c/birdsong-recognition/overview/code-requirements), participants are **not allowed** to use custom packages in **_submission notebook_**.\n\nIf you fork this notebook, keep the above things in mind.\n\n\n### Reference\n\n#### ResNeSt: Split-Attention Networks\n* author: Hang Zhang, Chongruo Wu, Zhongyue Zhang, Yi Zhu, Zhi Zhang, Haibin Lin, Yue Sun, Tong He, Jonas Muller, R. Manmatha, Mu Li and Alex Smola \n* paper: [arXiv 2004.08955](https://arxiv.org/abs/2004.08955)\n* code: [GitHub](https://github.com/koukyo1994/kaggle-birdcall-resnet-baseline-training)\n\n#### pytorch-pfn-extras\n* author: Preferred Networks, Inc.\n* code: [GitHub](https://github.com/pfnet/pytorch-pfn-extras)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Prepare","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### import libraries","execution_count":null},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"%%bash\npip install ../input/pytorch-pfn-extras/pytorch-pfn-extras-0.2.1/","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"%%bash\npip install ../input/resnest50-fast-package/resnest-0.0.6b20200701/resnest/","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport gc\nimport time\nimport shutil\nimport random\nimport warnings\nimport typing as tp\nfrom pathlib import Path\nfrom contextlib import contextmanager\n\nimport yaml\nfrom joblib import delayed, Parallel\n\nimport cv2\nimport librosa\nimport audioread\nimport soundfile as sf\n\nimport numpy as np\nimport pandas as pd\n\nfrom sklearn.metrics import f1_score\nfrom sklearn.model_selection import StratifiedKFold\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.utils.data as data\nimport resnest.torch as resnest_torch\n\nimport pytorch_pfn_extras as ppe\nfrom pytorch_pfn_extras.training import extensions as ppe_extensions\n\npd.options.display.max_rows = 500\npd.options.display.max_columns = 500","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Path(\"/root/.cache/torch/checkpoints\").mkdir(parents=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!cp ../input/resnest50-fast-package/resnest50_fast_*.pth /root/.cache/torch/checkpoints/","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"### define utilities","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def set_seed(seed: int = 42):\n    random.seed(seed)\n    np.random.seed(seed)\n    os.environ[\"PYTHONHASHSEED\"] = str(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)  # type: ignore\n#     torch.backends.cudnn.deterministic = True  # type: ignore\n#     torch.backends.cudnn.benchmark = True  # type: ignore\n    \n\n@contextmanager\ndef timer(name: str) -> None:\n    \"\"\"Timer Util\"\"\"\n    t0 = time.time()\n    print(\"[{}] start\".format(name))\n    yield\n    print(\"[{}] done in {:.0f} s\".format(name, time.time() - t0))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### read data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ROOT = Path.cwd().parent\nINPUT_ROOT = ROOT / \"input\"\nRAW_DATA = INPUT_ROOT / \"birdsong-recognition\"\nTRAIN_AUDIO_DIR = RAW_DATA / \"train_audio\"\nTRAIN_RESAMPLED_AUDIO_DIRS = [\n  INPUT_ROOT / \"birdsong-resampled-train-audio-{:0>2}\".format(i)  for i in range(5)\n]\nTEST_AUDIO_DIR = RAW_DATA / \"test_audio\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# train = pd.read_csv(RAW_DATA / \"train.csv\")\ntrain = pd.read_csv(TRAIN_RESAMPLED_AUDIO_DIRS[0] / \"train_mod.csv\")","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"train.head().T","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if not TEST_AUDIO_DIR.exists():\n    TEST_AUDIO_DIR = INPUT_ROOT / \"birdcall-check\" / \"test_audio\"\n    test = pd.read_csv(INPUT_ROOT / \"birdcall-check\" / \"test.csv\")\nelse:\n    test = pd.read_csv(RAW_DATA / \"test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"test.head().T","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### settings","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"settings_str = \"\"\"\nglobals:\n  seed: 1213\n  device: cuda\n  num_epochs: 50\n  output_dir: /kaggle/training_output/\n  use_fold: 0\n  target_sr: 32000\n\ndataset:\n  name: SpectrogramDataset\n  params:\n    img_size: 224\n    melspectrogram_parameters:\n      n_mels: 128\n      fmin: 20\n      fmax: 16000\n    \nsplit:\n  name: StratifiedKFold\n  params:\n    n_splits: 5\n    random_state: 42\n    shuffle: True\n\nloader:\n  train:\n    batch_size: 50\n    shuffle: True\n    num_workers: 2\n    pin_memory: True\n    drop_last: True\n  val:\n    batch_size: 100\n    shuffle: False\n    num_workers: 2\n    pin_memory: True\n    drop_last: False\n\nmodel:\n  name: resnest50_fast_1s1x64d\n  params:\n    pretrained: True\n    n_classes: 264\n\nloss:\n  name: BCEWithLogitsLoss\n  params: {}\n\noptimizer:\n  name: Adam\n  params:\n    lr: 0.001\n\nscheduler:\n  name: CosineAnnealingLR\n  params:\n    T_max: 10\n\"\"\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"settings = yaml.safe_load(settings_str)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# if not torch.cuda.is_available():\n#     settings[\"globals\"][\"device\"] = \"cpu\"","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":false,"trusted":true},"cell_type":"code","source":"for k, v in settings.items():\n    print(\"[{}]\".format(k))\n    print(v)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### preprocess audio data\n\nCode is forked from: https://github.com/koukyo1994/kaggle-birdcall-resnet-baseline-training/blob/master/input/birdsong-recognition/prepare.py\n\nI modified this partially. \n\nHowever, in this notebook, I used uploaded resampled audio because this preprocessing is too heavy for kaggle notebook.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def resample(ebird_code: str,filename: str, target_sr: int):    \n    audio_dir = TRAIN_AUDIO_DIR\n    resample_dir = TRAIN_RESAMPLED_DIR\n    ebird_dir = resample_dir / ebird_code\n    \n    try:\n        y, _ = librosa.load(\n            audio_dir / ebird_code / filename,\n            sr=target_sr, mono=True, res_type=\"kaiser_fast\")\n\n        filename = filename.replace(\".mp3\", \".wav\")\n        sf.write(ebird_dir / filename, y, samplerate=target_sr)\n    except Exception as e:\n        print(e)\n        with open(\"skipped.txt\", \"a\") as f:\n            file_path = str(audio_dir / ebird_code / filename)\n            f.write(file_path + \"\\n\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# train_org = train.copy()\n# TRAIN_RESAMPLED_DIR = Path(\"/kaggle/processed_data/train_audio_resampled\")\n# TRAIN_RESAMPLED_DIR.mkdir(parents=True)\n\n# for ebird_code in train.ebird_code.unique():\n#     ebird_dir = TRAIN_RESAMPLED_DIR / ebird_code\n#     ebird_dir.mkdir()\n\n# warnings.simplefilter(\"ignore\")\n# train_audio_infos = train[[\"ebird_code\", \"filename\"]].values.tolist()\n# Parallel(n_jobs=NUM_THREAD, verbose=10)(\n#     delayed(resample)(ebird_code, file_name, TARGET_SR) for ebird_code, file_name in train_audio_infos)\n\n# train[\"resampled_sampling_rate\"] = TARGET_SR\n# train[\"resampled_filename\"] = train[\"filename\"].map(\n#     lambda x: x.replace(\".mp3\", \".wav\"))\n# train[\"resampled_channels\"] = \"1 (mono)\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Definition","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Dataset\n* forked from: https://github.com/koukyo1994/kaggle-birdcall-resnet-baseline-training/blob/master/src/dataset.py\n* modified partialy\n","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"BIRD_CODE = {\n    'aldfly': 0, 'ameavo': 1, 'amebit': 2, 'amecro': 3, 'amegfi': 4,\n    'amekes': 5, 'amepip': 6, 'amered': 7, 'amerob': 8, 'amewig': 9,\n    'amewoo': 10, 'amtspa': 11, 'annhum': 12, 'astfly': 13, 'baisan': 14,\n    'baleag': 15, 'balori': 16, 'banswa': 17, 'barswa': 18, 'bawwar': 19,\n    'belkin1': 20, 'belspa2': 21, 'bewwre': 22, 'bkbcuc': 23, 'bkbmag1': 24,\n    'bkbwar': 25, 'bkcchi': 26, 'bkchum': 27, 'bkhgro': 28, 'bkpwar': 29,\n    'bktspa': 30, 'blkpho': 31, 'blugrb1': 32, 'blujay': 33, 'bnhcow': 34,\n    'boboli': 35, 'bongul': 36, 'brdowl': 37, 'brebla': 38, 'brespa': 39,\n    'brncre': 40, 'brnthr': 41, 'brthum': 42, 'brwhaw': 43, 'btbwar': 44,\n    'btnwar': 45, 'btywar': 46, 'buffle': 47, 'buggna': 48, 'buhvir': 49,\n    'bulori': 50, 'bushti': 51, 'buwtea': 52, 'buwwar': 53, 'cacwre': 54,\n    'calgul': 55, 'calqua': 56, 'camwar': 57, 'cangoo': 58, 'canwar': 59,\n    'canwre': 60, 'carwre': 61, 'casfin': 62, 'caster1': 63, 'casvir': 64,\n    'cedwax': 65, 'chispa': 66, 'chiswi': 67, 'chswar': 68, 'chukar': 69,\n    'clanut': 70, 'cliswa': 71, 'comgol': 72, 'comgra': 73, 'comloo': 74,\n    'commer': 75, 'comnig': 76, 'comrav': 77, 'comred': 78, 'comter': 79,\n    'comyel': 80, 'coohaw': 81, 'coshum': 82, 'cowscj1': 83, 'daejun': 84,\n    'doccor': 85, 'dowwoo': 86, 'dusfly': 87, 'eargre': 88, 'easblu': 89,\n    'easkin': 90, 'easmea': 91, 'easpho': 92, 'eastow': 93, 'eawpew': 94,\n    'eucdov': 95, 'eursta': 96, 'evegro': 97, 'fiespa': 98, 'fiscro': 99,\n    'foxspa': 100, 'gadwal': 101, 'gcrfin': 102, 'gnttow': 103, 'gnwtea': 104,\n    'gockin': 105, 'gocspa': 106, 'goleag': 107, 'grbher3': 108, 'grcfly': 109,\n    'greegr': 110, 'greroa': 111, 'greyel': 112, 'grhowl': 113, 'grnher': 114,\n    'grtgra': 115, 'grycat': 116, 'gryfly': 117, 'haiwoo': 118, 'hamfly': 119,\n    'hergul': 120, 'herthr': 121, 'hoomer': 122, 'hoowar': 123, 'horgre': 124,\n    'horlar': 125, 'houfin': 126, 'houspa': 127, 'houwre': 128, 'indbun': 129,\n    'juntit1': 130, 'killde': 131, 'labwoo': 132, 'larspa': 133, 'lazbun': 134,\n    'leabit': 135, 'leafly': 136, 'leasan': 137, 'lecthr': 138, 'lesgol': 139,\n    'lesnig': 140, 'lesyel': 141, 'lewwoo': 142, 'linspa': 143, 'lobcur': 144,\n    'lobdow': 145, 'logshr': 146, 'lotduc': 147, 'louwat': 148, 'macwar': 149,\n    'magwar': 150, 'mallar3': 151, 'marwre': 152, 'merlin': 153, 'moublu': 154,\n    'mouchi': 155, 'moudov': 156, 'norcar': 157, 'norfli': 158, 'norhar2': 159,\n    'normoc': 160, 'norpar': 161, 'norpin': 162, 'norsho': 163, 'norwat': 164,\n    'nrwswa': 165, 'nutwoo': 166, 'olsfly': 167, 'orcwar': 168, 'osprey': 169,\n    'ovenbi1': 170, 'palwar': 171, 'pasfly': 172, 'pecsan': 173, 'perfal': 174,\n    'phaino': 175, 'pibgre': 176, 'pilwoo': 177, 'pingro': 178, 'pinjay': 179,\n    'pinsis': 180, 'pinwar': 181, 'plsvir': 182, 'prawar': 183, 'purfin': 184,\n    'pygnut': 185, 'rebmer': 186, 'rebnut': 187, 'rebsap': 188, 'rebwoo': 189,\n    'redcro': 190, 'redhea': 191, 'reevir1': 192, 'renpha': 193, 'reshaw': 194,\n    'rethaw': 195, 'rewbla': 196, 'ribgul': 197, 'rinduc': 198, 'robgro': 199,\n    'rocpig': 200, 'rocwre': 201, 'rthhum': 202, 'ruckin': 203, 'rudduc': 204,\n    'rufgro': 205, 'rufhum': 206, 'rusbla': 207, 'sagspa1': 208, 'sagthr': 209,\n    'savspa': 210, 'saypho': 211, 'scatan': 212, 'scoori': 213, 'semplo': 214,\n    'semsan': 215, 'sheowl': 216, 'shshaw': 217, 'snobun': 218, 'snogoo': 219,\n    'solsan': 220, 'sonspa': 221, 'sora': 222, 'sposan': 223, 'spotow': 224,\n    'stejay': 225, 'swahaw': 226, 'swaspa': 227, 'swathr': 228, 'treswa': 229,\n    'truswa': 230, 'tuftit': 231, 'tunswa': 232, 'veery': 233, 'vesspa': 234,\n    'vigswa': 235, 'warvir': 236, 'wesblu': 237, 'wesgre': 238, 'weskin': 239,\n    'wesmea': 240, 'wessan': 241, 'westan': 242, 'wewpew': 243, 'whbnut': 244,\n    'whcspa': 245, 'whfibi': 246, 'whtspa': 247, 'whtswi': 248, 'wilfly': 249,\n    'wilsni1': 250, 'wiltur': 251, 'winwre3': 252, 'wlswar': 253, 'wooduc': 254,\n    'wooscj2': 255, 'woothr': 256, 'y00475': 257, 'yebfly': 258, 'yebsap': 259,\n    'yehbla': 260, 'yelwar': 261, 'yerwar': 262, 'yetvir': 263\n}\n\nINV_BIRD_CODE = {v: k for k, v in BIRD_CODE.items()}","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"PERIOD = 5\n\ndef mono_to_color(\n    X: np.ndarray, mean=None, std=None,\n    norm_max=None, norm_min=None, eps=1e-6\n):\n    # Stack X as [X,X,X]\n    X = np.stack([X, X, X], axis=-1)\n\n    # Standardize\n    mean = mean or X.mean()\n    X = X - mean\n    std = std or X.std()\n    Xstd = X / (std + eps)\n    _min, _max = Xstd.min(), Xstd.max()\n    norm_max = norm_max or _max\n    norm_min = norm_min or _min\n    if (_max - _min) > eps:\n        # Normalize to [0, 255]\n        V = Xstd\n        V[V < norm_min] = norm_min\n        V[V > norm_max] = norm_max\n        V = 255 * (V - norm_min) / (norm_max - norm_min)\n        V = V.astype(np.uint8)\n    else:\n        # Just zero\n        V = np.zeros_like(Xstd, dtype=np.uint8)\n    return V\n\nclass SpectrogramDataset(data.Dataset):\n    def __init__(\n        self,\n        file_list: tp.List[tp.List[str]], img_size=224,\n        waveform_transforms=None, spectrogram_transforms=None, melspectrogram_parameters={}\n    ):\n        self.file_list = file_list  # list of list: [file_path, ebird_code]\n        self.img_size = img_size\n        self.waveform_transforms = waveform_transforms\n        self.spectrogram_transforms = spectrogram_transforms\n        self.melspectrogram_parameters = melspectrogram_parameters\n\n    def __len__(self):\n        return len(self.file_list)\n\n    def __getitem__(self, idx: int):\n        wav_path, ebird_code = self.file_list[idx]\n\n        y, sr = sf.read(wav_path)\n\n        if self.waveform_transforms:\n            y = self.waveform_transforms(y)\n        else:\n            len_y = len(y)\n            effective_length = sr * PERIOD\n            if len_y < effective_length:\n                new_y = np.zeros(effective_length, dtype=y.dtype)\n                start = np.random.randint(effective_length - len_y)\n                new_y[start:start + len_y] = y\n                y = new_y.astype(np.float32)\n            elif len_y > effective_length:\n                start = np.random.randint(len_y - effective_length)\n                y = y[start:start + effective_length].astype(np.float32)\n            else:\n                y = y.astype(np.float32)\n\n        melspec = librosa.feature.melspectrogram(y, sr=sr, **self.melspectrogram_parameters)\n        melspec = librosa.power_to_db(melspec).astype(np.float32)\n\n        if self.spectrogram_transforms:\n            melspec = self.spectrogram_transforms(melspec)\n        else:\n            pass\n\n        image = mono_to_color(melspec)\n        height, width, _ = image.shape\n        image = cv2.resize(image, (int(width * self.img_size / height), self.img_size))\n        image = np.moveaxis(image, 2, 0)\n        image = (image / 255.0).astype(np.float32)\n\n#         labels = np.zeros(len(BIRD_CODE), dtype=\"i\")\n        labels = np.zeros(len(BIRD_CODE), dtype=\"f\")\n        labels[BIRD_CODE[ebird_code]] = 1\n\n        return image, labels","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Training Utility","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def get_loaders_for_training(\n    args_dataset: tp.Dict, args_loader: tp.Dict,\n    train_file_list: tp.List[str], val_file_list: tp.List[str]\n):\n    # # make dataset\n    train_dataset = SpectrogramDataset(train_file_list, **args_dataset)\n    val_dataset = SpectrogramDataset(val_file_list, **args_dataset)\n    # # make dataloader\n    train_loader = data.DataLoader(train_dataset, **args_loader[\"train\"])\n    val_loader = data.DataLoader(val_dataset, **args_loader[\"val\"])\n    \n    return train_loader, val_loader","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def get_model(args: tp.Dict):\n    model =getattr(resnest_torch, args[\"name\"])(pretrained=args[\"params\"][\"pretrained\"])\n    del model.fc\n    # # use the same head as the baseline notebook.\n    model.fc = nn.Sequential(\n        nn.Linear(2048, 1024), nn.ReLU(), nn.Dropout(p=0.2),\n        nn.Linear(1024, 1024), nn.ReLU(), nn.Dropout(p=0.2),\n        nn.Linear(1024, args[\"params\"][\"n_classes\"]))\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def train_loop(\n    manager, args, model, device,\n    train_loader, optimizer, scheduler, loss_func\n):\n    \"\"\"Run minibatch training loop\"\"\"\n    while not manager.stop_trigger:\n        model.train()\n        for batch_idx, (data, target) in enumerate(train_loader):\n            with manager.run_iteration():\n                data, target = data.to(device), target.to(device)\n                optimizer.zero_grad()\n                output = model(data)\n                loss = loss_func(output, target)\n                ppe.reporting.report({'train/loss': loss.item()})\n                loss.backward()\n                optimizer.step()\n                scheduler.step()\n\ndef eval_for_batch(\n    args, model, device,\n    data, target, loss_func, eval_func_dict={}\n):\n    \"\"\"\n    Run evaliation for valid\n    \n    This function is applied to each batch of val loader.\n    \"\"\"\n    model.eval()\n    data, target = data.to(device), target.to(device)\n    output = model(data)\n    # Final result will be average of averages of the same size\n    val_loss = loss_func(output, target).item()\n    ppe.reporting.report({'val/loss': val_loss})\n    \n    for eval_name, eval_func in eval_func_dict.items():\n        eval_value = eval_func(output, target).item()\n        ppe.reporting.report({\"val/{}\".format(eval_aame): eval_value})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def set_extensions(\n    manager, args, model, device, test_loader, optimizer,\n    loss_func, eval_func_dict={}\n):\n    \"\"\"set extensions for PPE\"\"\"\n        \n    my_extensions = [\n        # # observe, report\n        ppe_extensions.observe_lr(optimizer=optimizer),\n        # ppe_extensions.ParameterStatistics(model, prefix='model'),\n        # ppe_extensions.VariableStatisticsPlot(model),\n        ppe_extensions.LogReport(),\n        ppe_extensions.PlotReport(['train/loss', 'val/loss'], 'epoch', filename='loss.png'),\n        ppe_extensions.PlotReport(['lr',], 'epoch', filename='lr.png'),\n        ppe_extensions.PrintReport([\n            'epoch', 'iteration', 'lr', 'train/loss', 'val/loss', \"elapsed_time\"]),\n#         ppe_extensions.ProgressBar(update_interval=100),\n\n        # # evaluation\n        (\n            ppe_extensions.Evaluator(\n                test_loader, model,\n                eval_func=lambda data, target:\n                    eval_for_batch(args, model, device, data, target, loss_func, eval_func_dict),\n                progress_bar=True),\n            (1, \"epoch\"),\n        ),\n        # # save model snapshot.\n        (\n            ppe_extensions.snapshot(\n                target=model, filename=\"snapshot_epoch_{.updater.epoch}.pth\"),\n            ppe.training.triggers.MinValueTrigger(key=\"val/loss\", trigger=(1, 'epoch'))\n        ),\n    ]\n           \n    # # set extensions to manager\n    for ext in my_extensions:\n        if isinstance(ext, tuple):\n            manager.extend(ext[0], trigger=ext[1])\n        else:\n            manager.extend(ext)\n        \n    return manager","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Training","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### prepare data","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"#### get wav file path","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"tmp_list = []\nfor audio_d in TRAIN_RESAMPLED_AUDIO_DIRS:\n    if not audio_d.exists():\n        continue\n    for ebird_d in audio_d.iterdir():\n        if ebird_d.is_file():\n            continue\n        for wav_f in ebird_d.iterdir():\n            tmp_list.append([ebird_d.name, wav_f.name, wav_f.as_posix()])\n            \ntrain_wav_path_exist = pd.DataFrame(\n    tmp_list, columns=[\"ebird_code\", \"resampled_filename\", \"file_path\"])\n\ndel tmp_list\n\ntrain_all = pd.merge(\n    train, train_wav_path_exist, on=[\"ebird_code\", \"resampled_filename\"], how=\"inner\")\n\nprint(train.shape)\nprint(train_wav_path_exist.shape)\nprint(train_all.shape)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"train_all.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### split data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# # for test run\n# test_run_idx = sorted(np.random.choice(len(train_all), len(train_all) // 10, replace=False))\n# train_all = train_all.iloc[test_run_idx, :].reset_index(drop=True)\n# settings[\"globals\"][\"num_epochs\"] = 20\n# settings[\"scheduler\"][\"params\"][\"T_max\"] = 4","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":false,"trusted":true},"cell_type":"code","source":"skf = StratifiedKFold(**settings[\"split\"][\"params\"])\n\ntrain_all[\"fold\"] = -1\nfor fold_id, (train_index, val_index) in enumerate(skf.split(train_all, train_all[\"ebird_code\"])):\n    train_all.iloc[val_index, -1] = fold_id\n    \n# # check the propotion\nfold_proportion = pd.pivot_table(train_all, index=\"ebird_code\", columns=\"fold\", values=\"xc_id\", aggfunc=len)\nprint(fold_proportion.shape)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"fold_proportion","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"use_fold = settings[\"globals\"][\"use_fold\"]\ntrain_file_list = train_all.query(\"fold != @use_fold\")[[\"file_path\", \"ebird_code\"]].values.tolist()\nval_file_list = train_all.query(\"fold == @use_fold\")[[\"file_path\", \"ebird_code\"]].values.tolist()\n\nprint(\"[fold {}] train: {}, val: {}\".format(use_fold, len(train_file_list), len(val_file_list)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## run training","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"set_seed(settings[\"globals\"][\"seed\"])\ndevice = torch.device(settings[\"globals\"][\"device\"])\noutput_dir = Path(settings[\"globals\"][\"output_dir\"])\n\n# # # get loader\ntrain_loader, val_loader = get_loaders_for_training(\n    settings[\"dataset\"][\"params\"], settings[\"loader\"], train_file_list, val_file_list)\n\n# # # get model\nmodel = get_model(settings[\"model\"])\nmodel = model.to(device)\n\n# # # get optimizer\noptimizer = getattr(\n    torch.optim, settings[\"optimizer\"][\"name\"]\n)(model.parameters(), **settings[\"optimizer\"][\"params\"])\n\n# # # get scheduler\nscheduler = getattr(\n    torch.optim.lr_scheduler, settings[\"scheduler\"][\"name\"]\n)(optimizer, **settings[\"scheduler\"][\"params\"])\n\n# # # get loss\nloss_func = getattr(nn, settings[\"loss\"][\"name\"])(**settings[\"loss\"][\"params\"])\n\n# # # create training manager\ntrigger = None\n\nmanager = ppe.training.ExtensionsManager(\n    model, optimizer, settings[\"globals\"][\"num_epochs\"],\n    iters_per_epoch=len(train_loader),\n    stop_trigger=trigger,\n    out_dir=output_dir\n)\n\n# # # set manager extensions\nmanager = set_extensions(\n    manager, settings, model, device,\n    val_loader, optimizer, loss_func,\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# # runtraining\ntrain_loop(\n    manager, settings, model, device,\n    train_loader, optimizer, scheduler, loss_func)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del train_loader\ndel val_loader\ndel model\ndel optimizer\ndel scheduler\ndel loss_func\ndel manager\n\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## save results","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%bash\nls /kaggle/training_output","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for f_name in [\"log\",\"loss.png\", \"lr.png\"]:\n    shutil.copy(output_dir / f_name, f_name)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"log = pd.read_json(\"log\")\nbest_epoch = log[\"val/loss\"].idxmin() + 1\nlog.iloc[[best_epoch - 1],]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"shutil.copy(output_dir / \"snapshot_epoch_{}.pth\".format(best_epoch), \"best_model.pth\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"m = get_model({\n    'name': settings[\"model\"][\"name\"],\n    'params': {'pretrained': False, 'n_classes': 264}})\nstate_dict = torch.load('best_model.pth')\nm.load_state_dict(state_dict)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}