{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Birdsong Pytorch Baseline: ResNeSt50-fast (Inference)"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Info\n\n

\n This is the same\n classical 0.568 public notebook\n with some post-processing during inference time: same code, same pretrained weights !\n

\n\nHere, I aim to find out how far could we go with just good post-processing !?"},{"metadata":{},"cell_type":"markdown","source":"![aa](https://i.ibb.co/JrZT22k/sliding-windows.png)"},{"metadata":{},"cell_type":"markdown","source":"There are two versions of our main post-processing idea: the first one is based on birds occurrence counts and the seconds on the probabilities sum. The last one seems to perform better. The main idea consists in 3 sliding windows whom probabilities are thresholded and summed, the *n*-first birds with higher score are then recommended. If there is no birds above the threshold, we predict **nocall**"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## About\n\nIn this notebook, I try ResNeSt, which is the one of state of the art in image recognition. \n\nThis is a notebook for **_inference & submission_**. I shared training process as another one: \nhttps://www.kaggle.com/ttahara/training-birdsong-baseline-resnest50-fast \nIf you want to know experimental details, see it.\n\nMost of this notebook consists of [great baseline](https://www.kaggle.com/hidehisaarai1213/inference-pytorch-birdcall-resnet-baseline) shared by @hidehisaarai1213 . \nThank you for sharing !"},{"metadata":{},"cell_type":"markdown","source":"## Prepare"},{"metadata":{},"cell_type":"markdown","source":"### import libraries"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"import os\nimport gc\nimport time\nimport math\nimport shutil\nimport random\nimport warnings\nimport typing as tp\nfrom pathlib import Path\nfrom contextlib import contextmanager\n\nimport yaml\nfrom joblib import delayed, Parallel\n\nimport cv2\nimport librosa\nimport audioread\nimport soundfile as sf\n\nimport numpy as np\nimport pandas as pd\n\nfrom fastprogress import progress_bar\nfrom sklearn.metrics import f1_score\nfrom sklearn.model_selection import StratifiedKFold\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.nn import Conv2d, Module, Linear, BatchNorm2d, ReLU\nfrom torch.nn.modules.utils import _pair\nimport torch.utils.data as data\n\npd.options.display.max_rows = 500\npd.options.display.max_columns = 500","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true},"cell_type":"markdown","source":"### define utilities"},{"metadata":{"trusted":true},"cell_type":"code","source":"def set_seed(seed: int = 42):\n random.seed(seed)\n np.random.seed(seed)\n os.environ[\"PYTHONHASHSEED\"] = str(seed)\n torch.manual_seed(seed)\n torch.cuda.manual_seed(seed) # type: ignore\n# torch.backends.cudnn.deterministic = True # type: ignore\n# torch.backends.cudnn.benchmark = True # type: ignore\n \n\n@contextmanager\ndef timer(name: str) -> None:\n \"\"\"Timer Util\"\"\"\n t0 = time.time()\n print(\"[{}] start\".format(name))\n yield\n print(\"[{}] done in {:.0f} s\".format(name, time.time() - t0))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# logger = get_logger(\"main.log\")\nset_seed(1213)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### read data"},{"metadata":{"trusted":true},"cell_type":"code","source":"ROOT = Path.cwd().parent\nINPUT_ROOT = ROOT / \"input\"\nRAW_DATA = INPUT_ROOT / \"birdsong-recognition\"\nTRAIN_AUDIO_DIR = RAW_DATA / \"train_audio\"\n# TRAIN_RESAMPLED_AUDIO_DIRS = [\n# INPUT_ROOT / \"birdsong-resampled-train-audio-{:0>2}\".format(i) for i in range(5)\n# ]\nTEST_AUDIO_DIR = RAW_DATA / \"test_audio\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv(RAW_DATA / \"train.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if not TEST_AUDIO_DIR.exists():\n TEST_AUDIO_DIR = INPUT_ROOT / \"birdcall-check\" / \"test_audio\"\n test = pd.read_csv(INPUT_ROOT / \"birdcall-check\" / \"test.csv\")\nelse:\n test = pd.read_csv(RAW_DATA / \"test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub = pd.read_csv(\"../input/birdsong-recognition/sample_submission.csv\")\nsub.to_csv(\"submission.csv\", index=False) # this will be overwritten if everything goes well","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### set parameters"},{"metadata":{"trusted":true},"cell_type":"code","source":"TARGET_SR = 32000\nmodel_config = {\n \"base_model_name\": \"resnest50_fast_1s1x64d\",\n \"pretrained\": False,\n \"num_classes\": 264,\n \"trained_weights\": \"../input/training-birdsong-baseline-resnest50-fast/best_model.pth\"\n}\n\nmelspectrogram_parameters = {\n \"n_mels\": 128,\n \"fmin\": 20,\n \"fmax\": 16000\n}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Definition"},{"metadata":{},"cell_type":"markdown","source":"### Dataset\n\nFor `site_3`, I decided to use the same procedure as I did for `site_1` and `site_2`, which is, crop 5 seconds out of the clip and provide prediction on that short clip.\nThe only difference is that I crop 5 seconds short clip from start to the end of the `site_3` clip and aggeregate predictions for each short clip after I did prediction for all those short clips."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"BIRD_CODE = {\n 'aldfly': 0, 'ameavo': 1, 'amebit': 2, 'amecro': 3, 'amegfi': 4,\n 'amekes': 5, 'amepip': 6, 'amered': 7, 'amerob': 8, 'amewig': 9,\n 'amewoo': 10, 'amtspa': 11, 'annhum': 12, 'astfly': 13, 'baisan': 14,\n 'baleag': 15, 'balori': 16, 'banswa': 17, 'barswa': 18, 'bawwar': 19,\n 'belkin1': 20, 'belspa2': 21, 'bewwre': 22, 'bkbcuc': 23, 'bkbmag1': 24,\n 'bkbwar': 25, 'bkcchi': 26, 'bkchum': 27, 'bkhgro': 28, 'bkpwar': 29,\n 'bktspa': 30, 'blkpho': 31, 'blugrb1': 32, 'blujay': 33, 'bnhcow': 34,\n 'boboli': 35, 'bongul': 36, 'brdowl': 37, 'brebla': 38, 'brespa': 39,\n 'brncre': 40, 'brnthr': 41, 'brthum': 42, 'brwhaw': 43, 'btbwar': 44,\n 'btnwar': 45, 'btywar': 46, 'buffle': 47, 'buggna': 48, 'buhvir': 49,\n 'bulori': 50, 'bushti': 51, 'buwtea': 52, 'buwwar': 53, 'cacwre': 54,\n 'calgul': 55, 'calqua': 56, 'camwar': 57, 'cangoo': 58, 'canwar': 59,\n 'canwre': 60, 'carwre': 61, 'casfin': 62, 'caster1': 63, 'casvir': 64,\n 'cedwax': 65, 'chispa': 66, 'chiswi': 67, 'chswar': 68, 'chukar': 69,\n 'clanut': 70, 'cliswa': 71, 'comgol': 72, 'comgra': 73, 'comloo': 74,\n 'commer': 75, 'comnig': 76, 'comrav': 77, 'comred': 78, 'comter': 79,\n 'comyel': 80, 'coohaw': 81, 'coshum': 82, 'cowscj1': 83, 'daejun': 84,\n 'doccor': 85, 'dowwoo': 86, 'dusfly': 87, 'eargre': 88, 'easblu': 89,\n 'easkin': 90, 'easmea': 91, 'easpho': 92, 'eastow': 93, 'eawpew': 94,\n 'eucdov': 95, 'eursta': 96, 'evegro': 97, 'fiespa': 98, 'fiscro': 99,\n 'foxspa': 100, 'gadwal': 101, 'gcrfin': 102, 'gnttow': 103, 'gnwtea': 104,\n 'gockin': 105, 'gocspa': 106, 'goleag': 107, 'grbher3': 108, 'grcfly': 109,\n 'greegr': 110, 'greroa': 111, 'greyel': 112, 'grhowl': 113, 'grnher': 114,\n 'grtgra': 115, 'grycat': 116, 'gryfly': 117, 'haiwoo': 118, 'hamfly': 119,\n 'hergul': 120, 'herthr': 121, 'hoomer': 122, 'hoowar': 123, 'horgre': 124,\n 'horlar': 125, 'houfin': 126, 'houspa': 127, 'houwre': 128, 'indbun': 129,\n 'juntit1': 130, 'killde': 131, 'labwoo': 132, 'larspa': 133, 'lazbun': 134,\n 'leabit': 135, 'leafly': 136, 'leasan': 137, 'lecthr': 138, 'lesgol': 139,\n 'lesnig': 140, 'lesyel': 141, 'lewwoo': 142, 'linspa': 143, 'lobcur': 144,\n 'lobdow': 145, 'logshr': 146, 'lotduc': 147, 'louwat': 148, 'macwar': 149,\n 'magwar': 150, 'mallar3': 151, 'marwre': 152, 'merlin': 153, 'moublu': 154,\n 'mouchi': 155, 'moudov': 156, 'norcar': 157, 'norfli': 158, 'norhar2': 159,\n 'normoc': 160, 'norpar': 161, 'norpin': 162, 'norsho': 163, 'norwat': 164,\n 'nrwswa': 165, 'nutwoo': 166, 'olsfly': 167, 'orcwar': 168, 'osprey': 169,\n 'ovenbi1': 170, 'palwar': 171, 'pasfly': 172, 'pecsan': 173, 'perfal': 174,\n 'phaino': 175, 'pibgre': 176, 'pilwoo': 177, 'pingro': 178, 'pinjay': 179,\n 'pinsis': 180, 'pinwar': 181, 'plsvir': 182, 'prawar': 183, 'purfin': 184,\n 'pygnut': 185, 'rebmer': 186, 'rebnut': 187, 'rebsap': 188, 'rebwoo': 189,\n 'redcro': 190, 'redhea': 191, 'reevir1': 192, 'renpha': 193, 'reshaw': 194,\n 'rethaw': 195, 'rewbla': 196, 'ribgul': 197, 'rinduc': 198, 'robgro': 199,\n 'rocpig': 200, 'rocwre': 201, 'rthhum': 202, 'ruckin': 203, 'rudduc': 204,\n 'rufgro': 205, 'rufhum': 206, 'rusbla': 207, 'sagspa1': 208, 'sagthr': 209,\n 'savspa': 210, 'saypho': 211, 'scatan': 212, 'scoori': 213, 'semplo': 214,\n 'semsan': 215, 'sheowl': 216, 'shshaw': 217, 'snobun': 218, 'snogoo': 219,\n 'solsan': 220, 'sonspa': 221, 'sora': 222, 'sposan': 223, 'spotow': 224,\n 'stejay': 225, 'swahaw': 226, 'swaspa': 227, 'swathr': 228, 'treswa': 229,\n 'truswa': 230, 'tuftit': 231, 'tunswa': 232, 'veery': 233, 'vesspa': 234,\n 'vigswa': 235, 'warvir': 236, 'wesblu': 237, 'wesgre': 238, 'weskin': 239,\n 'wesmea': 240, 'wessan': 241, 'westan': 242, 'wewpew': 243, 'whbnut': 244,\n 'whcspa': 245, 'whfibi': 246, 'whtspa': 247, 'whtswi': 248, 'wilfly': 249,\n 'wilsni1': 250, 'wiltur': 251, 'winwre3': 252, 'wlswar': 253, 'wooduc': 254,\n 'wooscj2': 255, 'woothr': 256, 'y00475': 257, 'yebfly': 258, 'yebsap': 259,\n 'yehbla': 260, 'yelwar': 261, 'yerwar': 262, 'yetvir': 263\n}\n\nINV_BIRD_CODE = {v: k for k, v in BIRD_CODE.items()}","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def mono_to_color(X: np.ndarray,\n mean=None,\n std=None,\n norm_max=None,\n norm_min=None,\n eps=1e-6):\n \"\"\"\n Code from https://www.kaggle.com/daisukelab/creating-fat2019-preprocessed-data\n \"\"\"\n # Stack X as [X,X,X]\n X = np.stack([X, X, X], axis=-1)\n\n # Standardize\n mean = mean or X.mean()\n X = X - mean\n std = std or X.std()\n Xstd = X / (std + eps)\n _min, _max = Xstd.min(), Xstd.max()\n norm_max = norm_max or _max\n norm_min = norm_min or _min\n if (_max - _min) > eps:\n # Normalize to [0, 255]\n V = Xstd\n V[V < norm_min] = norm_min\n V[V > norm_max] = norm_max\n V = 255 * (V - norm_min) / (norm_max - norm_min)\n V = V.astype(np.uint8)\n else:\n # Just zero\n V = np.zeros_like(Xstd, dtype=np.uint8)\n return V\n\nclass TestDataset(data.Dataset):\n def __init__(self, df: pd.DataFrame, clip: np.ndarray,\n img_size=224, melspectrogram_parameters={}):\n self.df = df\n self.clip = clip\n self.img_size = img_size\n self.melspectrogram_parameters = melspectrogram_parameters\n \n def __len__(self):\n return len(self.df)\n \n def __getitem__(self, idx: int):\n try:\n return self.read(idx)\n except Exception as e:\n print(\"Exception: {}\".format(e))\n sample = self.df.loc[idx, :]\n return np.zeros((1,3, 224,547), dtype=\"float32\"),sample.row_id, sample.site\n \n \n def read(self, idx: int):\n SR = 32000\n sample = self.df.loc[idx, :]\n site = sample.site\n row_id = sample.row_id\n STRIDE = 5\n TRUE_KERNEL_SIZE = 5\n LARGE_KERNEL_SIZE = 15\n \n if site == \"site_3\":\n y = self.clip.astype(np.float32)\n len_y = len(y)\n start = 0\n end = SR * 5\n images = []\n while len_y > start:\n y_batch = y[start:end].astype(np.float32)\n if len(y_batch) != (SR * 5):\n break\n start = end\n end = end + SR * 5\n \n melspec = librosa.feature.melspectrogram(y_batch,\n sr=SR,\n **self.melspectrogram_parameters)\n melspec = librosa.power_to_db(melspec).astype(np.float32)\n image = mono_to_color(melspec)\n height, width, _ = image.shape\n image = cv2.resize(image, (int(width * self.img_size / height), self.img_size))\n image = np.moveaxis(image, 2, 0)\n image = (image / 255.0).astype(np.float32)\n images.append(image)\n images = np.asarray(images)\n return images, row_id, site\n else:\n # v11 : change here to make audio\n TRESH_SIZE = LARGE_KERNEL_SIZE*SR\n \n if len(self.clip) < TRESH_SIZE:\n nrepeats = TRESH_SIZE//len(self.clip)\n npads = TRESH_SIZE%len(self.clip)\n self.clip = np.concatenate([self.clip]*nrepeats + [self.clip[-npads:]])\n start_seconds0 = 0\n assert len(self.clip) == TRESH_SIZE\n \n elif len(self.clip) > 10*SR:\n delta = len(self.clip)/SR - sample.seconds\n start_seconds0 = sample.seconds - (LARGE_KERNEL_SIZE - STRIDE) if delta > STRIDE else len(self.clip)/SR - LARGE_KERNEL_SIZE\n start_seconds0 = max(0, start_seconds0)\n else:\n start_seconds0 = 0\n \n assert len(self.clip)>= TRESH_SIZE, (len(self.clip)/SR, SR)\n assert start_seconds0>=0, start_seconds0\n assert (start_seconds0 + LARGE_KERNEL_SIZE)*SR <= len(self.clip) , (start_seconds0, len(self.clip)/SR, SR, sample.seconds)\n# assert start_seconds0>=0, start_seconds0\n \n \n# start_index = int(round(SR * start_seconds))\n# end_index = int(round(SR * end_seconds))\n \n images = []\n for start_seconds in np.arange(start_seconds0, start_seconds0 + LARGE_KERNEL_SIZE - TRUE_KERNEL_SIZE + STRIDE, STRIDE):\n start_index = int(round(start_seconds*SR))\n end_index = int(round((start_seconds+TRUE_KERNEL_SIZE)*SR))\n \n \n y = self.clip[start_index:end_index].astype(np.float32)\n \n# print(len(y)/SR,start_seconds)\n\n melspec = librosa.feature.melspectrogram(y, sr=SR, **self.melspectrogram_parameters)\n melspec = librosa.power_to_db(melspec).astype(np.float32)\n\n image = mono_to_color(melspec)\n height, width, _ = image.shape\n image = cv2.resize(image, (int(width * self.img_size / height), self.img_size))\n image = np.moveaxis(image, 2, 0)\n image = (image / 255.0).astype(np.float32)\n \n images.append(image)\n try:\n images = np.asarray(images)\n# print(\"images.shape\", images.shape)\n except ValueError as e:\n msg = [im.shape for im in images],start_seconds\n raise ValueError(msg) from e\n return images, row_id, site","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# submission = prediction(test_df=test,\n# test_audio=TEST_AUDIO_DIR,\n# model_config=model_config,\n# mel_params=melspectrogram_parameters,\n# target_sr=TARGET_SR,\n# threshold=.6,\n# maxpreds = 3,\n# )\n# submission.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### model\n\n* I forked this code from authors' original implementation. [GitHub](https://github.com/zhanghang1989/ResNeSt)"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"class SplAtConv2d(Module):\n \"\"\"Split-Attention Conv2d\n \"\"\"\n def __init__(self, in_channels, channels, kernel_size, stride=(1, 1), padding=(0, 0),\n dilation=(1, 1), groups=1, bias=True,\n radix=2, reduction_factor=4,\n rectify=False, rectify_avg=False, norm_layer=None,\n dropblock_prob=0.0, **kwargs):\n super(SplAtConv2d, self).__init__()\n padding = _pair(padding)\n self.rectify = rectify and (padding[0] > 0 or padding[1] > 0)\n self.rectify_avg = rectify_avg\n inter_channels = max(in_channels*radix//reduction_factor, 32)\n self.radix = radix\n self.cardinality = groups\n self.channels = channels\n self.dropblock_prob = dropblock_prob\n if self.rectify:\n from rfconv import RFConv2d\n self.conv = RFConv2d(in_channels, channels*radix, kernel_size, stride, padding, dilation,\n groups=groups*radix, bias=bias, average_mode=rectify_avg, **kwargs)\n else:\n self.conv = Conv2d(in_channels, channels*radix, kernel_size, stride, padding, dilation,\n groups=groups*radix, bias=bias, **kwargs)\n self.use_bn = norm_layer is not None\n if self.use_bn:\n self.bn0 = norm_layer(channels*radix)\n self.relu = ReLU(inplace=True)\n self.fc1 = Conv2d(channels, inter_channels, 1, groups=self.cardinality)\n if self.use_bn:\n self.bn1 = norm_layer(inter_channels)\n self.fc2 = Conv2d(inter_channels, channels*radix, 1, groups=self.cardinality)\n if dropblock_prob > 0.0:\n self.dropblock = DropBlock2D(dropblock_prob, 3)\n self.rsoftmax = rSoftMax(radix, groups)\n\n def forward(self, x):\n x = self.conv(x)\n if self.use_bn:\n x = self.bn0(x)\n if self.dropblock_prob > 0.0:\n x = self.dropblock(x)\n x = self.relu(x)\n\n batch, rchannel = x.shape[:2]\n if self.radix > 1:\n if torch.__version__ < '1.5':\n splited = torch.split(x, int(rchannel//self.radix), dim=1)\n else:\n splited = torch.split(x, rchannel//self.radix, dim=1)\n gap = sum(splited) \n else:\n gap = x\n gap = F.adaptive_avg_pool2d(gap, 1)\n gap = self.fc1(gap)\n\n if self.use_bn:\n gap = self.bn1(gap)\n gap = self.relu(gap)\n\n atten = self.fc2(gap)\n atten = self.rsoftmax(atten).view(batch, -1, 1, 1)\n\n if self.radix > 1:\n if torch.__version__ < '1.5':\n attens = torch.split(atten, int(rchannel//self.radix), dim=1)\n else:\n attens = torch.split(atten, rchannel//self.radix, dim=1)\n out = sum([att*split for (att, split) in zip(attens, splited)])\n else:\n out = atten * x\n return out.contiguous()\n\nclass rSoftMax(nn.Module):\n def __init__(self, radix, cardinality):\n super().__init__()\n self.radix = radix\n self.cardinality = cardinality\n\n def forward(self, x):\n batch = x.size(0)\n if self.radix > 1:\n x = x.view(batch, self.cardinality, self.radix, -1).transpose(1, 2)\n x = F.softmax(x, dim=1)\n x = x.reshape(batch, -1)\n else:\n x = torch.sigmoid(x)\n return x","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"class DropBlock2D(object):\n def __init__(self, *args, **kwargs):\n raise NotImplementedError\n\nclass GlobalAvgPool2d(nn.Module):\n def __init__(self):\n \"\"\"Global average pooling over the input's spatial dimensions\"\"\"\n super(GlobalAvgPool2d, self).__init__()\n\n def forward(self, inputs):\n return nn.functional.adaptive_avg_pool2d(inputs, 1).view(inputs.size(0), -1)\n\nclass Bottleneck(nn.Module):\n \"\"\"ResNet Bottleneck\n \"\"\"\n # pylint: disable=unused-argument\n expansion = 4\n def __init__(self, inplanes, planes, stride=1, downsample=None,\n radix=1, cardinality=1, bottleneck_width=64,\n avd=False, avd_first=False, dilation=1, is_first=False,\n rectified_conv=False, rectify_avg=False,\n norm_layer=None, dropblock_prob=0.0, last_gamma=False):\n super(Bottleneck, self).__init__()\n group_width = int(planes * (bottleneck_width / 64.)) * cardinality\n self.conv1 = nn.Conv2d(inplanes, group_width, kernel_size=1, bias=False)\n self.bn1 = norm_layer(group_width)\n self.dropblock_prob = dropblock_prob\n self.radix = radix\n self.avd = avd and (stride > 1 or is_first)\n self.avd_first = avd_first\n\n if self.avd:\n self.avd_layer = nn.AvgPool2d(3, stride, padding=1)\n stride = 1\n\n if dropblock_prob > 0.0:\n self.dropblock1 = DropBlock2D(dropblock_prob, 3)\n if radix == 1:\n self.dropblock2 = DropBlock2D(dropblock_prob, 3)\n self.dropblock3 = DropBlock2D(dropblock_prob, 3)\n\n if radix >= 1:\n self.conv2 = SplAtConv2d(\n group_width, group_width, kernel_size=3,\n stride=stride, padding=dilation,\n dilation=dilation, groups=cardinality, bias=False,\n radix=radix, rectify=rectified_conv,\n rectify_avg=rectify_avg,\n norm_layer=norm_layer,\n dropblock_prob=dropblock_prob)\n elif rectified_conv:\n from rfconv import RFConv2d\n self.conv2 = RFConv2d(\n group_width, group_width, kernel_size=3, stride=stride,\n padding=dilation, dilation=dilation,\n groups=cardinality, bias=False,\n average_mode=rectify_avg)\n self.bn2 = norm_layer(group_width)\n else:\n self.conv2 = nn.Conv2d(\n group_width, group_width, kernel_size=3, stride=stride,\n padding=dilation, dilation=dilation,\n groups=cardinality, bias=False)\n self.bn2 = norm_layer(group_width)\n\n self.conv3 = nn.Conv2d(\n group_width, planes * 4, kernel_size=1, bias=False)\n self.bn3 = norm_layer(planes*4)\n\n if last_gamma:\n from torch.nn.init import zeros_\n zeros_(self.bn3.weight)\n self.relu = nn.ReLU(inplace=True)\n self.downsample = downsample\n self.dilation = dilation\n self.stride = stride\n\n def forward(self, x):\n residual = x\n\n out = self.conv1(x)\n out = self.bn1(out)\n if self.dropblock_prob > 0.0:\n out = self.dropblock1(out)\n out = self.relu(out)\n\n if self.avd and self.avd_first:\n out = self.avd_layer(out)\n\n out = self.conv2(out)\n if self.radix == 0:\n out = self.bn2(out)\n if self.dropblock_prob > 0.0:\n out = self.dropblock2(out)\n out = self.relu(out)\n\n if self.avd and not self.avd_first:\n out = self.avd_layer(out)\n\n out = self.conv3(out)\n out = self.bn3(out)\n if self.dropblock_prob > 0.0:\n out = self.dropblock3(out)\n\n if self.downsample is not None:\n residual = self.downsample(x)\n\n out += residual\n out = self.relu(out)\n\n return out\n\nclass ResNet(nn.Module):\n \"\"\"ResNet Variants\n Parameters\n ----------\n block : Block\n Class for the residual block. Options are BasicBlockV1, BottleneckV1.\n layers : list of int\n Numbers of layers in each block\n classes : int, default 1000\n Number of classification classes.\n dilated : bool, default False\n Applying dilation strategy to pretrained ResNet yielding a stride-8 model,\n typically used in Semantic Segmentation.\n norm_layer : object\n Normalization layer used in backbone network (default: :class:`mxnet.gluon.nn.BatchNorm`;\n for Synchronized Cross-GPU BachNormalization).\n Reference:\n - He, Kaiming, et al. \"Deep residual learning for image recognition.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.\n - Yu, Fisher, and Vladlen Koltun. \"Multi-scale context aggregation by dilated convolutions.\"\n \"\"\"\n # pylint: disable=unused-variable\n def __init__(self, block, layers, radix=1, groups=1, bottleneck_width=64,\n num_classes=1000, dilated=False, dilation=1,\n deep_stem=False, stem_width=64, avg_down=False,\n rectified_conv=False, rectify_avg=False,\n avd=False, avd_first=False,\n final_drop=0.0, dropblock_prob=0,\n last_gamma=False, norm_layer=nn.BatchNorm2d):\n self.cardinality = groups\n self.bottleneck_width = bottleneck_width\n # ResNet-D params\n self.inplanes = stem_width*2 if deep_stem else 64\n self.avg_down = avg_down\n self.last_gamma = last_gamma\n # ResNeSt params\n self.radix = radix\n self.avd = avd\n self.avd_first = avd_first\n\n super(ResNet, self).__init__()\n self.rectified_conv = rectified_conv\n self.rectify_avg = rectify_avg\n if rectified_conv:\n from rfconv import RFConv2d\n conv_layer = RFConv2d\n else:\n conv_layer = nn.Conv2d\n conv_kwargs = {'average_mode': rectify_avg} if rectified_conv else {}\n if deep_stem:\n self.conv1 = nn.Sequential(\n conv_layer(3, stem_width, kernel_size=3, stride=2, padding=1, bias=False, **conv_kwargs),\n norm_layer(stem_width),\n nn.ReLU(inplace=True),\n conv_layer(stem_width, stem_width, kernel_size=3, stride=1, padding=1, bias=False, **conv_kwargs),\n norm_layer(stem_width),\n nn.ReLU(inplace=True),\n conv_layer(stem_width, stem_width*2, kernel_size=3, stride=1, padding=1, bias=False, **conv_kwargs),\n )\n else:\n self.conv1 = conv_layer(3, 64, kernel_size=7, stride=2, padding=3,\n bias=False, **conv_kwargs)\n self.bn1 = norm_layer(self.inplanes)\n self.relu = nn.ReLU(inplace=True)\n self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)\n self.layer1 = self._make_layer(block, 64, layers[0], norm_layer=norm_layer, is_first=False)\n self.layer2 = self._make_layer(block, 128, layers[1], stride=2, norm_layer=norm_layer)\n if dilated or dilation == 4:\n self.layer3 = self._make_layer(block, 256, layers[2], stride=1,\n dilation=2, norm_layer=norm_layer,\n dropblock_prob=dropblock_prob)\n self.layer4 = self._make_layer(block, 512, layers[3], stride=1,\n dilation=4, norm_layer=norm_layer,\n dropblock_prob=dropblock_prob)\n elif dilation==2:\n self.layer3 = self._make_layer(block, 256, layers[2], stride=2,\n dilation=1, norm_layer=norm_layer,\n dropblock_prob=dropblock_prob)\n self.layer4 = self._make_layer(block, 512, layers[3], stride=1,\n dilation=2, norm_layer=norm_layer,\n dropblock_prob=dropblock_prob)\n else:\n self.layer3 = self._make_layer(block, 256, layers[2], stride=2,\n norm_layer=norm_layer,\n dropblock_prob=dropblock_prob)\n self.layer4 = self._make_layer(block, 512, layers[3], stride=2,\n norm_layer=norm_layer,\n dropblock_prob=dropblock_prob)\n self.avgpool = GlobalAvgPool2d()\n self.drop = nn.Dropout(final_drop) if final_drop > 0.0 else None\n self.fc = nn.Linear(512 * block.expansion, num_classes)\n\n for m in self.modules():\n if isinstance(m, nn.Conv2d):\n n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels\n m.weight.data.normal_(0, math.sqrt(2. / n))\n elif isinstance(m, norm_layer):\n m.weight.data.fill_(1)\n m.bias.data.zero_()\n\n def _make_layer(self, block, planes, blocks, stride=1, dilation=1, norm_layer=None,\n dropblock_prob=0.0, is_first=True):\n downsample = None\n if stride != 1 or self.inplanes != planes * block.expansion:\n down_layers = []\n if self.avg_down:\n if dilation == 1:\n down_layers.append(nn.AvgPool2d(kernel_size=stride, stride=stride,\n ceil_mode=True, count_include_pad=False))\n else:\n down_layers.append(nn.AvgPool2d(kernel_size=1, stride=1,\n ceil_mode=True, count_include_pad=False))\n down_layers.append(nn.Conv2d(self.inplanes, planes * block.expansion,\n kernel_size=1, stride=1, bias=False))\n else:\n down_layers.append(nn.Conv2d(self.inplanes, planes * block.expansion,\n kernel_size=1, stride=stride, bias=False))\n down_layers.append(norm_layer(planes * block.expansion))\n downsample = nn.Sequential(*down_layers)\n\n layers = []\n if dilation == 1 or dilation == 2:\n layers.append(block(self.inplanes, planes, stride, downsample=downsample,\n radix=self.radix, cardinality=self.cardinality,\n bottleneck_width=self.bottleneck_width,\n avd=self.avd, avd_first=self.avd_first,\n dilation=1, is_first=is_first, rectified_conv=self.rectified_conv,\n rectify_avg=self.rectify_avg,\n norm_layer=norm_layer, dropblock_prob=dropblock_prob,\n last_gamma=self.last_gamma))\n elif dilation == 4:\n layers.append(block(self.inplanes, planes, stride, downsample=downsample,\n radix=self.radix, cardinality=self.cardinality,\n bottleneck_width=self.bottleneck_width,\n avd=self.avd, avd_first=self.avd_first,\n dilation=2, is_first=is_first, rectified_conv=self.rectified_conv,\n rectify_avg=self.rectify_avg,\n norm_layer=norm_layer, dropblock_prob=dropblock_prob,\n last_gamma=self.last_gamma))\n else:\n raise RuntimeError(\"=> unknown dilation size: {}\".format(dilation))\n\n self.inplanes = planes * block.expansion\n for i in range(1, blocks):\n layers.append(block(self.inplanes, planes,\n radix=self.radix, cardinality=self.cardinality,\n bottleneck_width=self.bottleneck_width,\n avd=self.avd, avd_first=self.avd_first,\n dilation=dilation, rectified_conv=self.rectified_conv,\n rectify_avg=self.rectify_avg,\n norm_layer=norm_layer, dropblock_prob=dropblock_prob,\n last_gamma=self.last_gamma))\n\n return nn.Sequential(*layers)\n\n def forward(self, x):\n x = self.conv1(x)\n x = self.bn1(x)\n x = self.relu(x)\n x = self.maxpool(x)\n\n x = self.layer1(x)\n x = self.layer2(x)\n x = self.layer3(x)\n x = self.layer4(x)\n\n x = self.avgpool(x)\n #x = x.view(x.size(0), -1)\n x = torch.flatten(x, 1)\n if self.drop:\n x = self.drop(x)\n x = self.fc(x)\n\n return x","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_model(args: tp.Dict):\n # # get resnest50_fast_1s1x64d\n model = ResNet(\n Bottleneck, [3, 4, 6, 3],\n radix=1, groups=1, bottleneck_width=64,\n deep_stem=True, stem_width=32, avg_down=True,\n avd=True, avd_first=True)\n \n del model.fc\n # # use the same head as the baseline notebook.\n model.fc = nn.Sequential(\n nn.Linear(2048, 1024), nn.ReLU(), nn.Dropout(p=0.2),\n nn.Linear(1024, 1024), nn.ReLU(), nn.Dropout(p=0.2),\n nn.Linear(1024, args[\"num_classes\"]))\n \n device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n state_dict = torch.load(args[\"trained_weights\"], map_location = device)\n model.load_state_dict(state_dict)\n model.to(device)\n model.eval()\n \n return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Prediction loop"},{"metadata":{"trusted":true},"cell_type":"code","source":"def max_pred_gen(site, duration):\n if site != \"site_3\":\n return 3\n else:\n rets = [(10, 3), (30, 5), (60, 7)]\n \n for ref_duration,thresh in rets:\n if ref_duration >= duration:\n return thresh\n return 10","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# max_pred_gen(\"site_3\", 50)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def prediction_for_clip(test_df: pd.DataFrame, \n clip: np.ndarray, \n model: ResNet, \n mel_params: dict, \n threshold=0.5,\n maxpreds = max_pred_gen):\n\n dataset = TestDataset(df=test_df, \n clip=clip,\n img_size=224,\n melspectrogram_parameters=mel_params)\n loader = data.DataLoader(dataset, batch_size=1, shuffle=False)\n device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n \n model.eval()\n prediction_dict = {}\n for image, row_id, site in progress_bar(loader):\n site = site[0]\n row_id = row_id[0]\n if site in {\"site_1\", \"site_2\"}:\n image = image.squeeze(0)\n image = image.to(device)\n\n with torch.no_grad():\n prediction = F.sigmoid(model(image))\n# print('prediction.shape', prediction.shape)\n proba = prediction.detach().cpu().numpy()#.reshape(-1)\n# print('proba.shape', proba.shape)\n \n score = np.sum(proba*(proba >= threshold), axis=0)\n events = (score >= threshold - 1e-5)\n labels = np.argsort(-score)[:events.sum()].tolist()\n\n else:\n # to avoid prediction on large batch\n image = image.squeeze(0)\n batch_size = 16\n whole_size = image.size(0)\n if whole_size % batch_size == 0:\n n_iter = whole_size // batch_size\n else:\n n_iter = whole_size // batch_size + 1\n \n# all_events = set()\n# all_events = []\n probas = []\n for batch_i in range(n_iter):\n batch = image[batch_i * batch_size:(batch_i + 1) * batch_size]\n if batch.ndim == 3:\n batch = batch.unsqueeze(0)\n\n batch = batch.to(device)\n with torch.no_grad():\n prediction = F.sigmoid(model(batch))\n proba = prediction.detach().cpu().numpy()\n \n probas.append(proba)\n \n# events = proba >= threshold\n# all_events.append(np.where(events)[1])\n probas = np.vstack(probas)\n score = np.sum(probas*(probas >= threshold), axis=0)\n all_events = (score >= threshold - 1e-5)\n labels = np.argsort(-score)[:all_events.sum()].tolist()\n# all_events = np.concatenate(all_events)\n# labels = pd.Series(all_events).value_counts().index.tolist()\n \n if len(labels) == 0:\n prediction_dict[row_id] = \"nocall\"\n else:\n# print(site, maxpreds(site, len(clip)/32_000), len(clip)/32_000)\n \n labels_str_list = list(map(lambda x: INV_BIRD_CODE[x], labels[:maxpreds(site, len(clip)/32_000)] ))\n label_string = \" \".join(labels_str_list)\n prediction_dict[row_id] = label_string\n return prediction_dict","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def prediction(test_df: pd.DataFrame,\n test_audio: Path,\n model_config: dict,\n mel_params: dict,\n target_sr: int,\n threshold=0.6,\n maxpreds=max_pred_gen):\n model = get_model(model_config)\n unique_audio_id = test_df.audio_id.unique()\n\n warnings.filterwarnings(\"ignore\")\n prediction_dfs = []\n for audio_id in unique_audio_id :\n with timer(f\"Loading {audio_id}\"):\n clip, _ = librosa.load(test_audio / (audio_id + \".mp3\"),\n sr=target_sr,\n mono=True,\n res_type=\"kaiser_fast\")\n \n test_df_for_audio_id = test_df.query(\n f\"audio_id == '{audio_id}'\").reset_index(drop=True)\n with timer(f\"Prediction on {audio_id}\"):\n prediction_dict = prediction_for_clip(test_df_for_audio_id,\n clip=clip,\n model=model,\n mel_params=mel_params,\n threshold=threshold,\n maxpreds = maxpreds,\n )\n row_id = list(prediction_dict.keys())\n birds = list(prediction_dict.values())\n prediction_df = pd.DataFrame({\n \"row_id\": row_id,\n \"birds\": birds\n })\n prediction_dfs.append(prediction_df)\n \n prediction_df = pd.concat(prediction_dfs, axis=0, sort=False).reset_index(drop=True)\n return prediction_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Prediction"},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"submission = prediction(test_df=test,\n test_audio=TEST_AUDIO_DIR,\n model_config=model_config,\n mel_params=melspectrogram_parameters,\n target_sr=TARGET_SR,\n threshold=.6,\n maxpreds = max_pred_gen,\n )\nsubmission.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"submission","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## EOF"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}