{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Birdsong Pytorch Baseline: ResNeSt50-fast (Inference)"},{"metadata":{},"cell_type":"markdown","source":"## About\n\nIn this notebook, I try ResNeSt, which is the one of state of the art in image recognition.  \n\nThis is a notebook for **_inference & submission_**. I shared training process as another one:  \nhttps://www.kaggle.com/ttahara/training-birdsong-baseline-resnest50-fast  \nIf you want to know experimental details, see it.\n\nMost of this notebook consists of [great baseline](https://www.kaggle.com/hidehisaarai1213/inference-pytorch-birdcall-resnet-baseline) shared by @hidehisaarai1213 .  \nThank you for sharing !"},{"metadata":{},"cell_type":"markdown","source":"## Prepare"},{"metadata":{},"cell_type":"markdown","source":"### import libraries"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport gc\nimport time\nimport math\nimport shutil\nimport random\nimport warnings\nimport typing as tp\nfrom pathlib import Path\nfrom contextlib import contextmanager\nimport torchvision.models as models\nimport yaml\nfrom joblib import delayed, Parallel\n\nimport cv2\nimport librosa\nimport audioread\nimport soundfile as sf\n\nimport numpy as np\nimport pandas as pd\n\nfrom fastprogress import progress_bar\nfrom sklearn.metrics import f1_score\nfrom sklearn.model_selection import StratifiedKFold\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.nn import Conv2d, Module, Linear, BatchNorm2d, ReLU\nfrom torch.nn.modules.utils import _pair\nimport torch.utils.data as data\n\npd.options.display.max_rows = 500\npd.options.display.max_columns = 500","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"### define utilities"},{"metadata":{"trusted":true},"cell_type":"code","source":"def set_seed(seed: int = 42):\n    random.seed(seed)\n    np.random.seed(seed)\n    os.environ[\"PYTHONHASHSEED\"] = str(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)  # type: ignore\n#     torch.backends.cudnn.deterministic = True  # type: ignore\n#     torch.backends.cudnn.benchmark = True  # type: ignore\n    \n\n@contextmanager\ndef timer(name: str) -> None:\n    \"\"\"Timer Util\"\"\"\n    t0 = time.time()\n    #print(\"[{}] start\".format(name))\n    yield\n    #print(\"[{}] done in {:.0f} s\".format(name, time.time() - t0))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# logger = get_logger(\"main.log\")\nset_seed(1213)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### read data"},{"metadata":{"trusted":true},"cell_type":"code","source":"ROOT = Path.cwd().parent\nINPUT_ROOT = ROOT / \"input\"\nRAW_DATA = INPUT_ROOT / \"birdsong-recognition\"\nTRAIN_AUDIO_DIR = RAW_DATA / \"train_audio\"\n# TRAIN_RESAMPLED_AUDIO_DIRS = [\n#   INPUT_ROOT / \"birdsong-resampled-train-audio-{:0>2}\".format(i)  for i in range(5)\n# ]\nTEST_AUDIO_DIR = RAW_DATA / \"test_audio\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv(RAW_DATA / \"train.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if not TEST_AUDIO_DIR.exists():\n    TEST_AUDIO_DIR = INPUT_ROOT / \"birdcall-check\" / \"test_audio\"\n    test = pd.read_csv(INPUT_ROOT / \"birdcall-check\" / \"test.csv\")\nelse:\n    test = pd.read_csv(RAW_DATA / \"test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub = pd.read_csv(\"../input/birdsong-recognition/sample_submission.csv\")\nsub.to_csv(\"submission.csv\", index=False)  # this will be overwritten if everything goes well","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### set parameters"},{"metadata":{"trusted":true},"cell_type":"code","source":"TARGET_SR = 32000\nmodel_config = {\n    \"base_model_name\": \"resnest50_fast_1s1x64d\",\n    \"pretrained\": False,\n    \"num_classes\": 264,\n    \"trained_weights\": \"../input/bird-seed-v2/bird.pth\"\n}\n\nmelspectrogram_parameters = {\n    \"n_mels\": 155,\n    \"fmin\": 0,\n    \"fmax\": 16000,\n    \"n_fft\": 1024,\n    \"hop_length\": 256\n    \n\n}\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Definition"},{"metadata":{},"cell_type":"markdown","source":"### Dataset\n\nFor `site_3`, I decided to use the same procedure as I did for `site_1` and `site_2`, which is, crop 5 seconds out of the clip and provide prediction on that short clip.\nThe only difference is that I crop 5 seconds short clip from start to the end of the `site_3` clip and aggeregate predictions for each short clip after I did prediction for all those short clips."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"BIRD_CODE = {\n    'aldfly': 0, 'ameavo': 1, 'amebit': 2, 'amecro': 3, 'amegfi': 4,\n    'amekes': 5, 'amepip': 6, 'amered': 7, 'amerob': 8, 'amewig': 9,\n    'amewoo': 10, 'amtspa': 11, 'annhum': 12, 'astfly': 13, 'baisan': 14,\n    'baleag': 15, 'balori': 16, 'banswa': 17, 'barswa': 18, 'bawwar': 19,\n    'belkin1': 20, 'belspa2': 21, 'bewwre': 22, 'bkbcuc': 23, 'bkbmag1': 24,\n    'bkbwar': 25, 'bkcchi': 26, 'bkchum': 27, 'bkhgro': 28, 'bkpwar': 29,\n    'bktspa': 30, 'blkpho': 31, 'blugrb1': 32, 'blujay': 33, 'bnhcow': 34,\n    'boboli': 35, 'bongul': 36, 'brdowl': 37, 'brebla': 38, 'brespa': 39,\n    'brncre': 40, 'brnthr': 41, 'brthum': 42, 'brwhaw': 43, 'btbwar': 44,\n    'btnwar': 45, 'btywar': 46, 'buffle': 47, 'buggna': 48, 'buhvir': 49,\n    'bulori': 50, 'bushti': 51, 'buwtea': 52, 'buwwar': 53, 'cacwre': 54,\n    'calgul': 55, 'calqua': 56, 'camwar': 57, 'cangoo': 58, 'canwar': 59,\n    'canwre': 60, 'carwre': 61, 'casfin': 62, 'caster1': 63, 'casvir': 64,\n    'cedwax': 65, 'chispa': 66, 'chiswi': 67, 'chswar': 68, 'chukar': 69,\n    'clanut': 70, 'cliswa': 71, 'comgol': 72, 'comgra': 73, 'comloo': 74,\n    'commer': 75, 'comnig': 76, 'comrav': 77, 'comred': 78, 'comter': 79,\n    'comyel': 80, 'coohaw': 81, 'coshum': 82, 'cowscj1': 83, 'daejun': 84,\n    'doccor': 85, 'dowwoo': 86, 'dusfly': 87, 'eargre': 88, 'easblu': 89,\n    'easkin': 90, 'easmea': 91, 'easpho': 92, 'eastow': 93, 'eawpew': 94,\n    'eucdov': 95, 'eursta': 96, 'evegro': 97, 'fiespa': 98, 'fiscro': 99,\n    'foxspa': 100, 'gadwal': 101, 'gcrfin': 102, 'gnttow': 103, 'gnwtea': 104,\n    'gockin': 105, 'gocspa': 106, 'goleag': 107, 'grbher3': 108, 'grcfly': 109,\n    'greegr': 110, 'greroa': 111, 'greyel': 112, 'grhowl': 113, 'grnher': 114,\n    'grtgra': 115, 'grycat': 116, 'gryfly': 117, 'haiwoo': 118, 'hamfly': 119,\n    'hergul': 120, 'herthr': 121, 'hoomer': 122, 'hoowar': 123, 'horgre': 124,\n    'horlar': 125, 'houfin': 126, 'houspa': 127, 'houwre': 128, 'indbun': 129,\n    'juntit1': 130, 'killde': 131, 'labwoo': 132, 'larspa': 133, 'lazbun': 134,\n    'leabit': 135, 'leafly': 136, 'leasan': 137, 'lecthr': 138, 'lesgol': 139,\n    'lesnig': 140, 'lesyel': 141, 'lewwoo': 142, 'linspa': 143, 'lobcur': 144,\n    'lobdow': 145, 'logshr': 146, 'lotduc': 147, 'louwat': 148, 'macwar': 149,\n    'magwar': 150, 'mallar3': 151, 'marwre': 152, 'merlin': 153, 'moublu': 154,\n    'mouchi': 155, 'moudov': 156, 'norcar': 157, 'norfli': 158, 'norhar2': 159,\n    'normoc': 160, 'norpar': 161, 'norpin': 162, 'norsho': 163, 'norwat': 164,\n    'nrwswa': 165, 'nutwoo': 166, 'olsfly': 167, 'orcwar': 168, 'osprey': 169,\n    'ovenbi1': 170, 'palwar': 171, 'pasfly': 172, 'pecsan': 173, 'perfal': 174,\n    'phaino': 175, 'pibgre': 176, 'pilwoo': 177, 'pingro': 178, 'pinjay': 179,\n    'pinsis': 180, 'pinwar': 181, 'plsvir': 182, 'prawar': 183, 'purfin': 184,\n    'pygnut': 185, 'rebmer': 186, 'rebnut': 187, 'rebsap': 188, 'rebwoo': 189,\n    'redcro': 190, 'redhea': 191, 'reevir1': 192, 'renpha': 193, 'reshaw': 194,\n    'rethaw': 195, 'rewbla': 196, 'ribgul': 197, 'rinduc': 198, 'robgro': 199,\n    'rocpig': 200, 'rocwre': 201, 'rthhum': 202, 'ruckin': 203, 'rudduc': 204,\n    'rufgro': 205, 'rufhum': 206, 'rusbla': 207, 'sagspa1': 208, 'sagthr': 209,\n    'savspa': 210, 'saypho': 211, 'scatan': 212, 'scoori': 213, 'semplo': 214,\n    'semsan': 215, 'sheowl': 216, 'shshaw': 217, 'snobun': 218, 'snogoo': 219,\n    'solsan': 220, 'sonspa': 221, 'sora': 222, 'sposan': 223, 'spotow': 224,\n    'stejay': 225, 'swahaw': 226, 'swaspa': 227, 'swathr': 228, 'treswa': 229,\n    'truswa': 230, 'tuftit': 231, 'tunswa': 232, 'veery': 233, 'vesspa': 234,\n    'vigswa': 235, 'warvir': 236, 'wesblu': 237, 'wesgre': 238, 'weskin': 239,\n    'wesmea': 240, 'wessan': 241, 'westan': 242, 'wewpew': 243, 'whbnut': 244,\n    'whcspa': 245, 'whfibi': 246, 'whtspa': 247, 'whtswi': 248, 'wilfly': 249,\n    'wilsni1': 250, 'wiltur': 251, 'winwre3': 252, 'wlswar': 253, 'wooduc': 254,\n    'wooscj2': 255, 'woothr': 256, 'y00475': 257, 'yebfly': 258, 'yebsap': 259,\n    'yehbla': 260, 'yelwar': 261, 'yerwar': 262, 'yetvir': 263\n}\n\nINV_BIRD_CODE = {v: k for k, v in BIRD_CODE.items()}\n'''\n\nBIRD_CODE = {\n    'amegfi': 0, 'blujay': 1, 'horlar': 2, 'norcar': 3, 'mallar3': 4,\n    'bkhgro': 5, 'yerwar': 6, 'orcwar': 7, 'norwat': 8, 'carwre': 9,\n    'normoc': 10, 'marwre': 11, 'houwre': 12, 'barswa': 13, 'eastow': 14,\n    'easmea': 15, 'winwre3': 16, 'foxspa': 17, 'sonspa': 18, 'amered': 19,\n    'scoori': 20, 'boboli': 21, 'tuftit': 22, 'bkcchi': 23, 'bulori': 24,\n    'comred': 25, 'houspa': 26, 'brespa': 27, 'linspa': 28, 'swathr': 29,\n    'wesmea': 30, 'woothr': 31, 'chswar': 32, 'eucdov': 33, 'brncre': 34,\n    'norfli': 35, 'comyel': 36, 'wewpew': 37, 'cangoo': 38, 'indbun': 39,\n    'redcro': 40, 'haiwoo': 41, 'ruckin': 42, 'houfin': 43, 'spotow': 44,\n    'stejay': 45, 'hoowar': 46, 'chispa': 47, 'astfly': 48, 'amecro': 49,\n\n}\n\nINV_BIRD_CODE = {v: k for k, v in BIRD_CODE.items()}\n'''","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"'''def mono_to_color(X: np.ndarray,\n                  mean=None,\n                  std=None,\n                  norm_max=None,\n                  norm_min=None,\n                  eps=1e-6):\n    \"\"\"\n    Code from https://www.kaggle.com/daisukelab/creating-fat2019-preprocessed-data\n    \"\"\"\n    # Stack X as [X,X,X]\n    X = np.stack([X, X, X], axis=-1)\n\n    # Standardize\n    mean = mean or X.mean()\n    X = X - mean\n    std = std or X.std()\n    Xstd = X / (std + eps)\n    _min, _max = Xstd.min(), Xstd.max()\n    norm_max = norm_max or _max\n    norm_min = norm_min or _min\n    if (_max - _min) > eps:\n        # Normalize to [0, 255]\n        V = Xstd\n        V[V < norm_min] = norm_min\n        V[V > norm_max] = norm_max\n        V = 255 * (V - norm_min) / (norm_max - norm_min)\n        V = V.astype(np.uint8)\n    else:\n        # Just zero\n        V = np.zeros_like(Xstd, dtype=np.uint8)\n    return V\n\n\nclass TestDataset(data.Dataset):\n    def __init__(self, df: pd.DataFrame, clip: np.ndarray,\n                 img_size=224, melspectrogram_parameters={}):\n        self.df = df\n        self.clip = clip\n        self.img_size = img_size\n        self.melspectrogram_parameters = melspectrogram_parameters\n        \n    def __len__(self):\n        return len(self.df)\n    \n    def __getitem__(self, idx: int):\n        SR = 32000\n        sample = self.df.loc[idx, :]\n        site = sample.site\n        row_id = sample.row_id\n        \n        if site == \"site_3\":\n            y = self.clip.astype(np.float32)\n            len_y = len(y)\n            start = 0\n            end = SR * 5\n            images = []\n            while len_y > start:\n                y_batch = y[start:end].astype(np.float32)\n                if len(y_batch) != (SR * 5):\n                    break\n                start = end\n                end = end + SR * 5\n                \n                melspec = librosa.feature.melspectrogram(y_batch,\n                                                         sr=SR,\n                                                         **self.melspectrogram_parameters)\n                melspec = librosa.power_to_db(melspec).astype(np.float32)\n                \n                #melspec = librosa.pcen(melspec, sr=32000, hop_length=melspectrogram_parameters['hop_length'])\n                \n                print('this is the melspec shape ',melspec.shape)\n                image = mono_to_color(melspec)\n                height, width, _ = image.shape\n                #image = cv2.resize(image,(self.img_size, self.img_size))\n                image = cv2.resize(image, (int(width * self.img_size / height), self.img_size))\n                image = np.flipud(image)\n                image = np.moveaxis(image, 2, 0)\n                image = (image / 255.0).astype(np.float32)\n                images.append(image)\n            images = np.asarray(images)\n            return images, row_id, site\n        else:\n            end_seconds = int(sample.seconds)\n            start_seconds = int(end_seconds - 5)\n            \n            start_index = SR * start_seconds\n            end_index = SR * end_seconds\n            \n            y = self.clip[start_index:end_index].astype(np.float32)\n\n            melspec = librosa.feature.melspectrogram(y, sr=SR, **self.melspectrogram_parameters)\n            melspec = librosa.power_to_db(melspec).astype(np.float32)\n            \n            #melspec = librosa.pcen(melspec, sr=32000, hop_length=melspectrogram_parameters['hop_length'])\n\n            image = mono_to_color(melspec)\n            height, width, _ = image.shape\n            #image = cv2.resize(image,(self.img_size, self.img_size))\n            image = cv2.resize(image, (int(width * self.img_size / height), self.img_size))\n            image = np.flipud(image)\n            image = np.moveaxis(image, 2, 0)\n            image = (image / 255.0).astype(np.float32)\n\n            return image, row_id, site'''","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''def mono_to_color(X: np.ndarray,\n                  mean=None,\n                  std=None,\n                  norm_max=None,\n                  norm_min=None,\n                  eps=1e-6):\n    \"\"\"\n    Code from https://www.kaggle.com/daisukelab/creating-fat2019-preprocessed-data\n    \"\"\"\n    # Stack X as [X,X,X]\n    X = np.stack([X, X, X], axis=-1)\n\n    # Standardize\n    mean = mean or X.mean()\n    X = X - mean\n    std = std or X.std()\n    Xstd = X / (std + eps)\n    _min, _max = Xstd.min(), Xstd.max()\n    norm_max = norm_max or _max\n    norm_min = norm_min or _min\n    if (_max - _min) > eps:\n        # Normalize to [0, 255]\n        V = Xstd\n        V[V < norm_min] = norm_min\n        V[V > norm_max] = norm_max\n        V = 255 * (V - norm_min) / (norm_max - norm_min)\n        V = V.astype(np.uint8)\n    else:\n        # Just zero\n        V = np.zeros_like(Xstd, dtype=np.uint8)\n    return V\n\n\nclass TestDataset(data.Dataset):\n    def __init__(self, df: pd.DataFrame, clip: np.ndarray,\n                 img_size=224, melspectrogram_parameters={}):\n        self.df = df\n        self.clip = clip\n        self.img_size = img_size\n        self.melspectrogram_parameters = melspectrogram_parameters\n        \n    def __len__(self):\n        return len(self.df)\n    \n    def __getitem__(self, idx: int):\n        SR = 32000\n        sample = self.df.loc[idx, :]\n        site = sample.site\n        row_id = sample.row_id\n        \n        if site == \"site_3\":\n            y = self.clip.astype(np.float32)\n            len_y = len(y)\n            start = 0\n            end   =  start + 32000 * 1 # 1 sec\n            images = []\n            \n            melspec = librosa.feature.melspectrogram(y,\n                                                         sr=SR,\n                                                         **self.melspectrogram_parameters)\n            melspec_base = librosa.power_to_db(melspec).astype(np.float32)\n            print('site 3 mel ',melspec_base.shape)\n            \n            \n            \n            while len_y > start:\n                y_batch = y[start:end].astype(np.float32)\n                if len(y_batch) != 32000*1:# to break at the last if the time exceeds 5 sec\n                    #print('put break')\n                    break\n                start = end - (32000*1)//2\n                end = start + 32000 * 1\n                \n                melspec = librosa.feature.melspectrogram(y_batch,\n                                                         sr=SR,\n                                                         **self.melspectrogram_parameters)\n                melspec_base = librosa.power_to_db(melspec).astype(np.float32)\n                \n                #melspec = librosa.pcen(melspec, sr=32000, hop_length=melspectrogram_parameters['hop_length'])\n                \n\n                #print('start ',start)\n                #print('end ',end)\n\n                #melspec = melspec_base[:, start : end ]\n                #print('this is the melspec shape site 3',melspec.shape)\n\n                image = mono_to_color(melspec_base)\n                height, width, _ = image.shape\n                #image = cv2.resize(image,(self.img_size, self.img_size))\n                image = cv2.resize(image, (int(width * self.img_size / height), self.img_size))\n                #print(image.shape)\n                image = np.flipud(image)\n                image = np.moveaxis(image, 2, 0)\n                image = (image / 255.0).astype(np.float32)\n                images.append(image)\n            images = np.asarray(images)\n            #print('this is site 3 images ',images.shape)\n            return images, row_id, site\n        else:\n            end_seconds = int(sample.seconds)\n            start_seconds = int(end_seconds - 5)\n            \n            start_index = SR * start_seconds\n            end_index = SR * end_seconds\n            \n            y = self.clip[start_index:end_index].astype(np.float32)\n\n            melspec = librosa.feature.melspectrogram(y, sr=SR, **self.melspectrogram_parameters)\n            melspec_base = librosa.power_to_db(melspec).astype(np.float32)\n            #print('this is the melspec shape site 1',melspec.shape)\n            images = []\n            for iii in range(9):\n                    start =  iii * (626//5)//2\n                    end   =  start + (626//5)\n                    if end > 626:\n                        #print(\"mistake on site 1\")\n                        break\n                    #print('start ',start)\n                    #print('end ',end)\n\n                    melspec = melspec_base[:, start : end ]\n                    #print('this is the melspec shape 1',melspec.shape)\n\n                    image = mono_to_color(melspec)\n                    height, width, _ = image.shape\n                    #image = cv2.resize(image,(self.img_size, self.img_size))\n                    image = cv2.resize(image, (int(width * self.img_size / height), self.img_size))\n                    image = np.flipud(image)\n                    image = np.moveaxis(image, 2, 0)\n                    image = (image / 255.0).astype(np.float32)\n                    images.append(image)\n            #print('this is site images shape ',np.array(images).shape)\n\n            return np.array(images), row_id, site'''","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def mono_to_color(X: np.ndarray,\n                  mean=None,\n                  std=None,\n                  norm_max=None,\n                  norm_min=None,\n                  eps=1e-6):\n    \"\"\"\n    Code from https://www.kaggle.com/daisukelab/creating-fat2019-preprocessed-data\n    \"\"\"\n    # Stack X as [X,X,X]\n    X = np.stack([X, X, X], axis=-1)\n\n    # Standardize\n    mean = mean or X.mean()\n    X = X - mean\n    std = std or X.std()\n    Xstd = X / (std + eps)\n    _min, _max = Xstd.min(), Xstd.max()\n    norm_max = norm_max or _max\n    norm_min = norm_min or _min\n    if (_max - _min) > eps:\n        # Normalize to [0, 255]\n        V = Xstd\n        V[V < norm_min] = norm_min\n        V[V > norm_max] = norm_max\n        V = 255 * (V - norm_min) / (norm_max - norm_min)\n        V = V.astype(np.uint8)\n    else:\n        # Just zero\n        V = np.zeros_like(Xstd, dtype=np.uint8)\n    return V\n\n\nclass TestDataset(data.Dataset):\n    def __init__(self, df: pd.DataFrame, clip: np.ndarray,\n                 img_size=224, melspectrogram_parameters={}):\n        self.df = df\n        self.clip = clip\n        self.img_size = img_size\n        self.melspectrogram_parameters = melspectrogram_parameters\n        \n    def __len__(self):\n        return len(self.df)\n    \n    def __getitem__(self, idx: int):\n        SR = 32000\n        sample = self.df.loc[idx, :]\n        site = sample.site\n        row_id = sample.row_id\n        \n        if site == \"site_3\":\n            y = self.clip.astype(np.float32)\n            len_y = len(y)\n            start = 0\n            end   =  start + 32000 * 1 # 1 sec\n            images = []\n            \n            melspec = librosa.feature.melspectrogram(y,\n                                                         sr=SR,\n                                                         **self.melspectrogram_parameters)\n            melspec_base = librosa.power_to_db(melspec).astype(np.float32)\n            #print('site 3 mel ',melspec_base.shape)\n            images = []\n            for iii in range((melspec_base.shape[-1]//125) + (melspec_base.shape[-1]//125) -1):\n                    start =  iii * 125//2\n                    end   =  start + 125\n                    if end > melspec_base.shape[-1]:\n                        #print(\"mistake on site 1\")\n                        break\n                    #print('start ',start)\n                    #print('end ',end)\n\n                    melspec = melspec_base[:, start : end ]\n                    #print('this is the melspec shape 1',melspec.shape)\n\n                    image = mono_to_color(melspec)\n                    height, width, _ = image.shape\n                    #image = cv2.resize(image,(self.img_size, self.img_size))\n                    image = cv2.resize(image, (int(width * self.img_size / height), self.img_size))\n                    image = np.flipud(image)\n                    image = np.moveaxis(image, 2, 0)\n                    image = (image / 255.0).astype(np.float32)\n                    images.append(image)\n            \n            \n            \n            images = np.asarray(images)\n            #print('this is site 3 images ',images.shape)\n            return images, row_id, site\n        else:\n            end_seconds = int(sample.seconds)\n            start_seconds = int(end_seconds - 5)\n            \n            start_index = SR * start_seconds\n            end_index = SR * end_seconds\n            \n            y = self.clip[start_index:end_index].astype(np.float32)\n\n            melspec = librosa.feature.melspectrogram(y, sr=SR, **self.melspectrogram_parameters)\n            melspec_base = librosa.power_to_db(melspec).astype(np.float32)\n            #print('this is the melspec shape site 1',melspec.shape)\n            images = []\n            for iii in range(9):\n                    start =  iii * (626//5)//2\n                    end   =  start + (626//5)\n                    if end > 626:\n                        #print(\"mistake on site 1\")\n                        break\n                    #print('start ',start)\n                    #print('end ',end)\n\n                    melspec = melspec_base[:, start : end ]\n                    #print('this is the melspec shape 1',melspec.shape)\n\n                    image = mono_to_color(melspec)\n                    height, width, _ = image.shape\n                    #image = cv2.resize(image,(self.img_size, self.img_size))\n                    image = cv2.resize(image, (int(width * self.img_size / height), self.img_size))\n                    image = np.flipud(image)\n                    image = np.moveaxis(image, 2, 0)\n                    image = (image / 255.0).astype(np.float32)\n                    images.append(image)\n            #print('this is site images shape ',np.array(images).shape)\n\n            return np.array(images), row_id, site","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### model\n\n* I forked this code from authors' original implementation. [GitHub](https://github.com/zhanghang1989/ResNeSt)"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\"\"\"\nclass Autopool(nn.Module):\n    def __init__(self, input_size, ):\n        super(Autopool, self).__init__()\n        self.alpha = nn.Parameter(requires_grad= True)\n        self.alpha.data = torch.ones([input_size], dtype=torch.float32, requires_grad= True, device=device)\n        self.sigmoid_layer = nn.Sigmoid()\n        #self.softmax_layer = nn.Softmax(dim=2)\n        \n        \n    def forward(self,x):\n        \n        sigmoid_output = self.sigmoid_layer(x)\n        alpa_mult_out = torch.mul(sigmoid_output, self.alpha)\n        \n        \n        max_tensor = torch.max(alpa_mult_out,dim = 1)\n        max_tensor_unsqueezed = max_tensor.values.unsqueeze(dim=1)\n        #print('alpa_mult_out shape ',alpa_mult_out.shape)\n        #print('max_tensor shape ',max_tensor.values.shape)\n        \n        softmax_numerator = torch.exp(alpa_mult_out.sub(max_tensor_unsqueezed))\n        \n        softmax_den =torch.sum(softmax_numerator, dim = 1)\n        softmax_den = softmax_den.unsqueeze(dim=1)\n        \n        weights  = softmax_numerator/softmax_den\n        \n        final_out = torch.sum(torch.mul(sigmoid_output, weights),dim = 1)\n        return final_out, sigmoid_output\n\n        \n\n\nclass Yuvsub(nn.Module):   \n    def __init__(self, args):\n        super(Yuvsub, self).__init__()\n        self.species = nn.Sequential(\n            nn.Linear(2048, 1024), nn.ReLU(), nn.Dropout(p=0.2),\n            nn.Linear(1024, 1024), nn.ReLU(), nn.Dropout(p=0.2),\n            nn.Linear(1024, 264))\n        \n        \n        \n    def forward(self, GAP):\n        \n        #print('gap shape ',GAP.shape)\n        #GAP = torch.flatten(GAP, 1)\n        #print('gap shape ',GAP.shape)\n        spe = self.species(GAP)\n        \n        #cnt = self.counter(GAP)\n        return spe\n\nclass Yuvgru(nn.Module):\n    def __init__(self, args):\n        super(Yuvgru, self).__init__()\n        self.gru_layer = torch.nn.GRU(input_size = 1024, hidden_size=args[\"gru_hidden_size\"],num_layers = args[\"gru_layers\"],\n                                      dropout=0,bidirectional = args[\"gru_bidirectional\"])\n    def forward(self,x):\n        Routput, hn = self.gru_layer(x)## not passing the hiddenstate and it will be default to 0\n        return Routput, hn\n\nclass YuvNet(nn.Module):   \n    def __init__(self, args):\n        super(YuvNet, self).__init__()\n\n        #print(args[\"name\"] , args[\"params\"][\"pretrained\"])\n        self.model =ResNet(\n                        Bottleneck, [3, 4, 6, 3],\n                        radix=1, groups=1, bottleneck_width=64,\n                        deep_stem=True, stem_width=32, avg_down=True,\n                        avd=True, avd_first=True)\n        del self.model.fc\n        self.model.fc = Yuvsub(args)\n        #self.Gru = Yuvgru(args)\n        \n        \n        self.autopool = Autopool(264)\n           \n    def forward(self, x):\n        batch_size, time_steps, C, H, W = x.size()\n        c_in = x.view(batch_size * time_steps, C, H, W)\n        \n        #print('c_in shape ',c_in.shape)\n        #print('c_in type ',c_in.dtype)\n        spe = self.model(c_in)\n        #print('shape of spe ',spe.shape)\n        spe = spe.view(batch_size, time_steps, -1)\n        final_output, sigmoid_output = self.autopool(spe)\n        \n        '''print('gap shape ',GAP.shape)\n        GAP = torch.flatten(GAP, 1)\n        print('gap shape ',GAP.shape)\n        spe = self.species(GAP)\n        cnt = self.counter(GAP)'''\n        #return final_output, sigmoid_output\n        return final_output, sigmoid_output\n\n\n\"\"\"\n\nclass Autopool(nn.Module):\n    def __init__(self, input_size, ):\n        super(Autopool, self).__init__()\n        self.alpha = nn.Parameter(requires_grad= True)\n        self.alpha.data = torch.ones([input_size], dtype=torch.float32, requires_grad= True, device=device)\n        self.sigmoid_layer = nn.Sigmoid()        \n    def forward(self,x):       \n        sigmoid_output = self.sigmoid_layer(x)\n        alpa_mult_out = torch.mul(sigmoid_output, self.alpha)        \n        max_tensor = torch.max(alpa_mult_out,dim = 1)\n        max_tensor_unsqueezed = max_tensor.values.unsqueeze(dim=1)        \n        softmax_numerator = torch.exp(alpa_mult_out.sub(max_tensor_unsqueezed))        \n        softmax_den =torch.sum(softmax_numerator, dim = 1)\n        softmax_den = softmax_den.unsqueeze(dim=1)       \n        weights  = softmax_numerator/softmax_den       \n        final_out = torch.sum(torch.mul(sigmoid_output, weights),dim = 1)\n        return final_out, sigmoid_output\n\nclass Yuvsub(nn.Module):   \n    def __init__(self, args):\n        super(Yuvsub, self).__init__()\n        self.species = nn.Sequential(\n            nn.Linear(512, 512), nn.ReLU(), nn.Dropout(p=0.2),\n            nn.Linear(512, 264))\n        \n        \n        \n    def forward(self, GAP):\n        \n        #print('gap shape ',GAP.shape)\n        #GAP = torch.flatten(GAP, 1)\n        #print('gap shape ',GAP.shape)\n        spe = self.species(GAP)\n        \n        #cnt = self.counter(GAP)\n        return spe\n\nclass Yuvgru(nn.Module):\n    def __init__(self, args):\n        super(Yuvgru, self).__init__()\n        self.gru_layer = torch.nn.GRU(input_size = 1024, hidden_size=args[\"gru_hidden_size\"],num_layers = args[\"gru_layers\"],\n                                      dropout=0,bidirectional = args[\"gru_bidirectional\"])\n    def forward(self,x):\n        Routput, hn = self.gru_layer(x)## not passing the hiddenstate and it will be default to 0\n        return Routput, hn\n\nclass YuvNet(nn.Module):   \n    def __init__(self, args):\n        super(YuvNet, self).__init__()\n\n        #print(args[\"name\"] , args[\"params\"][\"pretrained\"])\n        #self.model =getattr(resnest_torch, args[\"name\"])(pretrained=args[\"params\"][\"pretrained\"])\n        self.model = models.resnet18(pretrained = False)\n        del self.model.fc\n        self.model.fc = Yuvsub(args)\n        #self.Gru = Yuvgru(args)\n        \n        \n        self.autopool = Autopool(264)\n           \n    def forward(self, x):\n        batch_size, time_steps, C, H, W = x.size()\n        c_in = x.view(batch_size * time_steps, C, H, W)\n        \n        #print('c_in shape ',c_in.shape)\n        #print('c_in type ',c_in.dtype)\n        spe = self.model(c_in)\n        #print('shape of spe ',spe.shape)\n        spe = spe.view(batch_size, time_steps, -1)\n        final_output, sigmoid_output = self.autopool(spe)\n        \n        '''print('gap shape ',GAP.shape)\n        GAP = torch.flatten(GAP, 1)\n        print('gap shape ',GAP.shape)\n        spe = self.species(GAP)\n        cnt = self.counter(GAP)'''\n        #return final_output, sigmoid_output\n        return final_output, sigmoid_output\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"device = torch.device(\"cuda\")\ndef get_model(args: tp.Dict):\n    # # get resnest50_fast_1s1x64d\n    \n    model_1 = YuvNet(args)\n    state_dict = torch.load(\"../input/bird-fold1-150-v1/bird.pth\")\n    model_1.load_state_dict(state_dict)\n    model_1.to(device)\n    model_1.eval()\n    \n    model_2 = YuvNet(args)\n    state_dict = torch.load(\"../input/bird-label-v1/bird.pth\")\n    model_2.load_state_dict(state_dict)\n    model_2.to(device)\n    model_2.eval()\n    \n    model_3 = YuvNet(args)\n    state_dict = torch.load(\"../input/bird-fold-22/bird.pth\")\n    model_3.load_state_dict(state_dict)\n    model_3.to(device)\n    model_3.eval()\n    \n    model_4 = YuvNet(args)\n    state_dict = torch.load(\"../input/bird-light/bird.pth\")\n    model_4.load_state_dict(state_dict)\n    model_4.to(device)\n    model_4.eval()\n    \n    return model_1, model_2 , model_3, model_4","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_1, model_2, model_3, model_4 = get_model(model_config)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Prediction loop"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def prediction_for_clip(test_df: pd.DataFrame, \n                        clip: np.ndarray, \n                        model: nn.Module, \n                        mel_params: dict, \n                        threshold=0.5):\n    #print(test_df)\n    dataset = TestDataset(df=test_df, \n                          clip=clip,\n                          img_size=224,\n                          melspectrogram_parameters=mel_params)\n    loader = data.DataLoader(dataset, batch_size=1, shuffle=False)\n    device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n    \n    model_1.eval()\n    model_2.eval()\n    model_3.eval()\n    model_4.eval()\n    prediction_dict = {}\n    for image, row_id, site in progress_bar(loader):\n        site = site[0]\n        row_id = row_id[0]\n        if site in {\"site_1\", \"site_2\"}:\n            #image = image.unsqueeze(0)\n            image = image.to(device)\n            #print('site 1 image ',image.shape)\n            with torch.no_grad():\n                prediction1,_ = model_1(image)\n                prediction2,_ = model_2(image)\n                prediction3,_ = model_3(image)\n                prediction4,_ = model_4(image)\n                proba1 = prediction1.detach().cpu().numpy().reshape(-1)\n                proba2 = prediction2.detach().cpu().numpy().reshape(-1)\n                proba3 = prediction3.detach().cpu().numpy().reshape(-1)\n                proba4 = prediction4.detach().cpu().numpy().reshape(-1)\n            #print('proba1 ',proba1.shape)\n            #print('proba2 ',proba2.shape)\n            proba = (proba1 + proba2 + proba3 + proba4)/4.\n            #print('proba ',proba.shape)\n            events = proba >= threshold\n            labels = np.argwhere(events).reshape(-1).tolist()\n            #print('this is site 1 labels ',labels)\n\n        else:\n            # to avoid prediction on large batch\n            #image = image.squeeze(0)\n            batch_size = 32\n            image = image.squeeze(0)\n            whole_size = image.size(0)\n            #print('site 3 image sha ', image.size())\n            if whole_size % batch_size == 0:\n                n_iter = whole_size // batch_size\n            else:\n                n_iter = whole_size // batch_size + 1\n                \n            all_events = set()\n            \n            for batch_i in range(n_iter):\n                batch = image[batch_i * batch_size:(batch_i + 1) * batch_size]\n                #print('site 3 batch ',batch.shape)\n                #if batch.ndim == 3:\n                #    batch = batch.unsqueeze(0)\n                #batch = batch.unsqueeze(0)\n                batch = batch.unsqueeze(0)\n                batch = batch.to(device)\n                #print('site 3 batch ',batch.shape)\n                '''with torch.no_grad():\n                    prediction,_ = model(batch)\n                    proba = prediction.detach().cpu().numpy()'''\n                with torch.no_grad():\n                    prediction1,_ = model_1(batch)\n                    prediction2,_ = model_2(batch)\n                    proba1 = prediction1.detach().cpu().numpy().reshape(-1)\n                    proba2 = prediction2.detach().cpu().numpy().reshape(-1)\n                #print('this is site 1 prediciton shape ',proba)\n                #print('proba1 ',proba1.shape)\n                #print('proba2 ',proba2.shape)\n                proba = (proba1 + proba2)/2.\n                #print('proba ',proba.shape)\n                #print(proba)   \n                events = proba >= threshold\n                #print('event ',events)\n                labels = np.argwhere(events).reshape(-1).tolist()\n                for label in labels:\n                        all_events.add(label)\n                '''\n                for i in range(len(events)):\n                    event = events[i, :]\n                    labels = np.argwhere(event).reshape(-1).tolist()\n                    for label in labels:\n                        all_events.add(label)'''\n                        \n            labels = list(all_events)\n            #print('this is site 3 labels ',labels)\n        if len(labels) == 0:\n            prediction_dict[row_id] = \"nocall\"\n        else:\n            labels_str_list = list(map(lambda x: INV_BIRD_CODE[x], labels))\n            label_string = \" \".join(labels_str_list)\n            prediction_dict[row_id] = label_string\n    return prediction_dict","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.seconds.max()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def prediction(test_df: pd.DataFrame,\n               test_audio: Path,\n               model_config: dict,\n               mel_params: dict,\n               target_sr: int,\n               threshold=0.5):\n    model = get_model(model_config)\n    unique_audio_id = test_df.audio_id.unique()\n\n    warnings.filterwarnings(\"ignore\")\n    prediction_dfs = []\n    for audio_id in unique_audio_id:\n        with timer(f\"Loading {audio_id}\"):\n            clip, _ = librosa.load(test_audio / (audio_id + \".mp3\"),\n                                   sr=target_sr,\n                                   mono=True,\n                                   res_type=\"kaiser_fast\")\n        \n        test_df_for_audio_id = test_df.query(\n            f\"audio_id == '{audio_id}'\").reset_index(drop=True)\n        with timer(f\"Prediction on {audio_id}\"):\n            prediction_dict = prediction_for_clip(test_df_for_audio_id,\n                                                  clip=clip,\n                                                  model=model,\n                                                  mel_params=mel_params,\n                                                  threshold=threshold)\n        row_id = list(prediction_dict.keys())\n        birds = list(prediction_dict.values())\n        prediction_df = pd.DataFrame({\n            \"row_id\": row_id,\n            \"birds\": birds\n        })\n        prediction_dfs.append(prediction_df)\n    \n    prediction_df = pd.concat(prediction_dfs, axis=0, sort=False).reset_index(drop=True)\n    return prediction_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Prediction"},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"submission = prediction(test_df=test,\n                        test_audio=TEST_AUDIO_DIR,\n                        model_config=model_config,\n                        mel_params=melspectrogram_parameters,\n                        target_sr=TARGET_SR,\n                        threshold=0.4)\nsubmission.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"## EOF"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}