{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Credits\n\n\nMost of this notebook consists of those great baselines :\n* [First and original baseline](https://www.kaggle.com/hidehisaarai1213/inference-pytorch-birdcall-resnet-baseline) shared by @hidehisaarai1213\n* [Tahara's adaptation](https://www.kaggle.com/ttahara/inference-birdsong-baseline-resnest50-fast) shared by @ttahara\n\nThank you all for sharing !","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Introduction","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"This CBR competition seems interesting even though very challenging.\n\nAt the beginning, we all felt so weak when unable to beat the baseline. Fortunately @hidehisaarai1213 published his great baseline which allows many people to overpass the baseline a little bit. Since then, things become more and more boring : no real advancements was made and the leaderboard seems to stagnate around **0.568** (best open public score till now). Hence, many of kagglers was asking top ranked teams to reveal a little bit their ideas in order to make things interesting again.\n\n I will respond favorably to that call by sharing here some post-processing ideas which seem to be effective in this CBR competion.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Post-processing 1 : model perplexity reduction","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"\n\n \n \n \n \n \n \n \n \n \n \n \n\n\n \n \n \n \n \n \n \n \n \n \n \n\n
aldflyameavocoshumlinspapasflyrinducsnogoowestanyerwar
0.630.650.600.6250.610.640.620.630.615
","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Here is an example where the model is perplex about the singing bird. In such situation, the model could put slightly high probability around multiple birds. We can cure this misbehaviour by ranking the birds and limiting the total amount of predicted birds. In this notebook, I will ask the model to predict **no more than 3 birds** by audio file.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Post-processing 2 : max-pooling over the third site","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Instead of taking unique bird occurences, I will compute the highest occurence probability for each bird and then make my decision based on those probas.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Prepare","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### import libraries","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport gc\nimport time\nimport math\nimport shutil\nimport random\nimport warnings\nimport typing as tp\nfrom pathlib import Path\nfrom contextlib import contextmanager\n\nimport yaml\nfrom joblib import delayed, Parallel\n\nimport cv2\nimport librosa\nimport audioread\nimport soundfile as sf\n\nimport numpy as np\nimport pandas as pd\n\nfrom fastprogress import progress_bar\nfrom sklearn.metrics import f1_score\nfrom sklearn.model_selection import StratifiedKFold\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.nn import Conv2d, Module, Linear, BatchNorm2d, ReLU\nfrom torch.nn.modules.utils import _pair\nimport torch.utils.data as data\n\npd.options.display.max_rows = 500\npd.options.display.max_columns = 500","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"### define utilities","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def set_seed(seed: int = 42):\n random.seed(seed)\n np.random.seed(seed)\n os.environ[\"PYTHONHASHSEED\"] = str(seed)\n torch.manual_seed(seed)\n torch.cuda.manual_seed(seed) # type: ignore\n# torch.backends.cudnn.deterministic = True # type: ignore\n# torch.backends.cudnn.benchmark = True # type: ignore\n \n\n@contextmanager\ndef timer(name: str) -> None:\n \"\"\"Timer Util\"\"\"\n t0 = time.time()\n print(\"[{}] start\".format(name))\n yield\n print(\"[{}] done in {:.0f} s\".format(name, time.time() - t0))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# logger = get_logger(\"main.log\")\nset_seed(1213)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### read data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ROOT = Path.cwd().parent\nINPUT_ROOT = ROOT / \"input\"\nRAW_DATA = INPUT_ROOT / \"birdsong-recognition\"\nTRAIN_AUDIO_DIR = RAW_DATA / \"train_audio\"\n# TRAIN_RESAMPLED_AUDIO_DIRS = [\n# INPUT_ROOT / \"birdsong-resampled-train-audio-{:0>2}\".format(i) for i in range(5)\n# ]\nTEST_AUDIO_DIR = RAW_DATA / \"test_audio\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv(RAW_DATA / \"train.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if not TEST_AUDIO_DIR.exists():\n TEST_AUDIO_DIR = INPUT_ROOT / \"birdcall-check\" / \"test_audio\"\n test = pd.read_csv(INPUT_ROOT / \"birdcall-check\" / \"test.csv\")\nelse:\n test = pd.read_csv(RAW_DATA / \"test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub = pd.read_csv(\"../input/birdsong-recognition/sample_submission.csv\")\nsub.to_csv(\"submission.csv\", index=False) # this will be overwritten if everything goes well","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### set parameters","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"TARGET_SR = 32000\nmodel_config = {\n \"base_model_name\": \"resnest50_fast_1s1x64d\",\n \"pretrained\": False,\n \"num_classes\": 264,\n \"trained_weights\": \"../input/training-birdsong-baseline-resnest50-fast/best_model.pth\"\n}\n\nmelspectrogram_parameters = {\n \"n_mels\": 128,\n \"fmin\": 20,\n \"fmax\": 16000\n}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Definition","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Dataset\n\nFor `site_3`, I decided to use the same procedure as I did for `site_1` and `site_2`, which is, crop 5 seconds out of the clip and provide prediction on that short clip.\nThe only difference is that I crop 5 seconds short clip from start to the end of the `site_3` clip and aggeregate predictions for each short clip after I did prediction for all those short clips.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"BIRD_CODE = {\n 'aldfly': 0, 'ameavo': 1, 'amebit': 2, 'amecro': 3, 'amegfi': 4,\n 'amekes': 5, 'amepip': 6, 'amered': 7, 'amerob': 8, 'amewig': 9,\n 'amewoo': 10, 'amtspa': 11, 'annhum': 12, 'astfly': 13, 'baisan': 14,\n 'baleag': 15, 'balori': 16, 'banswa': 17, 'barswa': 18, 'bawwar': 19,\n 'belkin1': 20, 'belspa2': 21, 'bewwre': 22, 'bkbcuc': 23, 'bkbmag1': 24,\n 'bkbwar': 25, 'bkcchi': 26, 'bkchum': 27, 'bkhgro': 28, 'bkpwar': 29,\n 'bktspa': 30, 'blkpho': 31, 'blugrb1': 32, 'blujay': 33, 'bnhcow': 34,\n 'boboli': 35, 'bongul': 36, 'brdowl': 37, 'brebla': 38, 'brespa': 39,\n 'brncre': 40, 'brnthr': 41, 'brthum': 42, 'brwhaw': 43, 'btbwar': 44,\n 'btnwar': 45, 'btywar': 46, 'buffle': 47, 'buggna': 48, 'buhvir': 49,\n 'bulori': 50, 'bushti': 51, 'buwtea': 52, 'buwwar': 53, 'cacwre': 54,\n 'calgul': 55, 'calqua': 56, 'camwar': 57, 'cangoo': 58, 'canwar': 59,\n 'canwre': 60, 'carwre': 61, 'casfin': 62, 'caster1': 63, 'casvir': 64,\n 'cedwax': 65, 'chispa': 66, 'chiswi': 67, 'chswar': 68, 'chukar': 69,\n 'clanut': 70, 'cliswa': 71, 'comgol': 72, 'comgra': 73, 'comloo': 74,\n 'commer': 75, 'comnig': 76, 'comrav': 77, 'comred': 78, 'comter': 79,\n 'comyel': 80, 'coohaw': 81, 'coshum': 82, 'cowscj1': 83, 'daejun': 84,\n 'doccor': 85, 'dowwoo': 86, 'dusfly': 87, 'eargre': 88, 'easblu': 89,\n 'easkin': 90, 'easmea': 91, 'easpho': 92, 'eastow': 93, 'eawpew': 94,\n 'eucdov': 95, 'eursta': 96, 'evegro': 97, 'fiespa': 98, 'fiscro': 99,\n 'foxspa': 100, 'gadwal': 101, 'gcrfin': 102, 'gnttow': 103, 'gnwtea': 104,\n 'gockin': 105, 'gocspa': 106, 'goleag': 107, 'grbher3': 108, 'grcfly': 109,\n 'greegr': 110, 'greroa': 111, 'greyel': 112, 'grhowl': 113, 'grnher': 114,\n 'grtgra': 115, 'grycat': 116, 'gryfly': 117, 'haiwoo': 118, 'hamfly': 119,\n 'hergul': 120, 'herthr': 121, 'hoomer': 122, 'hoowar': 123, 'horgre': 124,\n 'horlar': 125, 'houfin': 126, 'houspa': 127, 'houwre': 128, 'indbun': 129,\n 'juntit1': 130, 'killde': 131, 'labwoo': 132, 'larspa': 133, 'lazbun': 134,\n 'leabit': 135, 'leafly': 136, 'leasan': 137, 'lecthr': 138, 'lesgol': 139,\n 'lesnig': 140, 'lesyel': 141, 'lewwoo': 142, 'linspa': 143, 'lobcur': 144,\n 'lobdow': 145, 'logshr': 146, 'lotduc': 147, 'louwat': 148, 'macwar': 149,\n 'magwar': 150, 'mallar3': 151, 'marwre': 152, 'merlin': 153, 'moublu': 154,\n 'mouchi': 155, 'moudov': 156, 'norcar': 157, 'norfli': 158, 'norhar2': 159,\n 'normoc': 160, 'norpar': 161, 'norpin': 162, 'norsho': 163, 'norwat': 164,\n 'nrwswa': 165, 'nutwoo': 166, 'olsfly': 167, 'orcwar': 168, 'osprey': 169,\n 'ovenbi1': 170, 'palwar': 171, 'pasfly': 172, 'pecsan': 173, 'perfal': 174,\n 'phaino': 175, 'pibgre': 176, 'pilwoo': 177, 'pingro': 178, 'pinjay': 179,\n 'pinsis': 180, 'pinwar': 181, 'plsvir': 182, 'prawar': 183, 'purfin': 184,\n 'pygnut': 185, 'rebmer': 186, 'rebnut': 187, 'rebsap': 188, 'rebwoo': 189,\n 'redcro': 190, 'redhea': 191, 'reevir1': 192, 'renpha': 193, 'reshaw': 194,\n 'rethaw': 195, 'rewbla': 196, 'ribgul': 197, 'rinduc': 198, 'robgro': 199,\n 'rocpig': 200, 'rocwre': 201, 'rthhum': 202, 'ruckin': 203, 'rudduc': 204,\n 'rufgro': 205, 'rufhum': 206, 'rusbla': 207, 'sagspa1': 208, 'sagthr': 209,\n 'savspa': 210, 'saypho': 211, 'scatan': 212, 'scoori': 213, 'semplo': 214,\n 'semsan': 215, 'sheowl': 216, 'shshaw': 217, 'snobun': 218, 'snogoo': 219,\n 'solsan': 220, 'sonspa': 221, 'sora': 222, 'sposan': 223, 'spotow': 224,\n 'stejay': 225, 'swahaw': 226, 'swaspa': 227, 'swathr': 228, 'treswa': 229,\n 'truswa': 230, 'tuftit': 231, 'tunswa': 232, 'veery': 233, 'vesspa': 234,\n 'vigswa': 235, 'warvir': 236, 'wesblu': 237, 'wesgre': 238, 'weskin': 239,\n 'wesmea': 240, 'wessan': 241, 'westan': 242, 'wewpew': 243, 'whbnut': 244,\n 'whcspa': 245, 'whfibi': 246, 'whtspa': 247, 'whtswi': 248, 'wilfly': 249,\n 'wilsni1': 250, 'wiltur': 251, 'winwre3': 252, 'wlswar': 253, 'wooduc': 254,\n 'wooscj2': 255, 'woothr': 256, 'y00475': 257, 'yebfly': 258, 'yebsap': 259,\n 'yehbla': 260, 'yelwar': 261, 'yerwar': 262, 'yetvir': 263\n}\n\nINV_BIRD_CODE = {v: k for k, v in BIRD_CODE.items()}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def mono_to_color(X: np.ndarray,\n mean=None,\n std=None,\n norm_max=None,\n norm_min=None,\n eps=1e-6):\n \"\"\"\n Code from https://www.kaggle.com/daisukelab/creating-fat2019-preprocessed-data\n \"\"\"\n # Stack X as [X,X,X]\n X = np.stack([X, X, X], axis=-1)\n\n # Standardize\n mean = mean or X.mean()\n X = X - mean\n std = std or X.std()\n Xstd = X / (std + eps)\n _min, _max = Xstd.min(), Xstd.max()\n norm_max = norm_max or _max\n norm_min = norm_min or _min\n if (_max - _min) > eps:\n # Normalize to [0, 255]\n V = Xstd\n V[V < norm_min] = norm_min\n V[V > norm_max] = norm_max\n V = 255 * (V - norm_min) / (norm_max - norm_min)\n V = V.astype(np.uint8)\n else:\n # Just zero\n V = np.zeros_like(Xstd, dtype=np.uint8)\n return V\n\n\nclass TestDataset(data.Dataset):\n def __init__(self, df: pd.DataFrame, clip: np.ndarray,\n img_size=224, melspectrogram_parameters={}):\n self.df = df\n self.clip = clip\n self.img_size = img_size\n self.melspectrogram_parameters = melspectrogram_parameters\n \n def __len__(self):\n return len(self.df)\n \n def __getitem__(self, idx: int):\n SR = 32000\n sample = self.df.loc[idx, :]\n site = sample.site\n row_id = sample.row_id\n \n if site == \"site_3\":\n y = self.clip.astype(np.float32)\n len_y = len(y)\n start = 0\n end = SR * 5\n images = []\n while len_y > start:\n y_batch = y[start:end].astype(np.float32)\n if len(y_batch) != (SR * 5):\n break\n start = end\n end = end + SR * 5\n \n melspec = librosa.feature.melspectrogram(y_batch,\n sr=SR,\n **self.melspectrogram_parameters)\n melspec = librosa.power_to_db(melspec).astype(np.float32)\n image = mono_to_color(melspec)\n height, width, _ = image.shape\n image = cv2.resize(image, (int(width * self.img_size / height), self.img_size))\n image = np.moveaxis(image, 2, 0)\n image = (image / 255.0).astype(np.float32)\n images.append(image)\n images = np.asarray(images)\n return images, row_id, site\n else:\n end_seconds = int(sample.seconds)\n start_seconds = int(end_seconds - 5)\n \n start_index = SR * start_seconds\n end_index = SR * end_seconds\n \n y = self.clip[start_index:end_index].astype(np.float32)\n\n melspec = librosa.feature.melspectrogram(y, sr=SR, **self.melspectrogram_parameters)\n melspec = librosa.power_to_db(melspec).astype(np.float32)\n\n image = mono_to_color(melspec)\n height, width, _ = image.shape\n image = cv2.resize(image, (int(width * self.img_size / height), self.img_size))\n image = np.moveaxis(image, 2, 0)\n image = (image / 255.0).astype(np.float32)\n\n return image, row_id, site","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### model\n\n* I forked this code from authors' original implementation. [GitHub](https://github.com/zhanghang1989/ResNeSt)","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"class SplAtConv2d(Module):\n \"\"\"Split-Attention Conv2d\n \"\"\"\n def __init__(self, in_channels, channels, kernel_size, stride=(1, 1), padding=(0, 0),\n dilation=(1, 1), groups=1, bias=True,\n radix=2, reduction_factor=4,\n rectify=False, rectify_avg=False, norm_layer=None,\n dropblock_prob=0.0, **kwargs):\n super(SplAtConv2d, self).__init__()\n padding = _pair(padding)\n self.rectify = rectify and (padding[0] > 0 or padding[1] > 0)\n self.rectify_avg = rectify_avg\n inter_channels = max(in_channels*radix//reduction_factor, 32)\n self.radix = radix\n self.cardinality = groups\n self.channels = channels\n self.dropblock_prob = dropblock_prob\n if self.rectify:\n from rfconv import RFConv2d\n self.conv = RFConv2d(in_channels, channels*radix, kernel_size, stride, padding, dilation,\n groups=groups*radix, bias=bias, average_mode=rectify_avg, **kwargs)\n else:\n self.conv = Conv2d(in_channels, channels*radix, kernel_size, stride, padding, dilation,\n groups=groups*radix, bias=bias, **kwargs)\n self.use_bn = norm_layer is not None\n if self.use_bn:\n self.bn0 = norm_layer(channels*radix)\n self.relu = ReLU(inplace=True)\n self.fc1 = Conv2d(channels, inter_channels, 1, groups=self.cardinality)\n if self.use_bn:\n self.bn1 = norm_layer(inter_channels)\n self.fc2 = Conv2d(inter_channels, channels*radix, 1, groups=self.cardinality)\n if dropblock_prob > 0.0:\n self.dropblock = DropBlock2D(dropblock_prob, 3)\n self.rsoftmax = rSoftMax(radix, groups)\n\n def forward(self, x):\n x = self.conv(x)\n if self.use_bn:\n x = self.bn0(x)\n if self.dropblock_prob > 0.0:\n x = self.dropblock(x)\n x = self.relu(x)\n\n batch, rchannel = x.shape[:2]\n if self.radix > 1:\n if torch.__version__ < '1.5':\n splited = torch.split(x, int(rchannel//self.radix), dim=1)\n else:\n splited = torch.split(x, rchannel//self.radix, dim=1)\n gap = sum(splited) \n else:\n gap = x\n gap = F.adaptive_avg_pool2d(gap, 1)\n gap = self.fc1(gap)\n\n if self.use_bn:\n gap = self.bn1(gap)\n gap = self.relu(gap)\n\n atten = self.fc2(gap)\n atten = self.rsoftmax(atten).view(batch, -1, 1, 1)\n\n if self.radix > 1:\n if torch.__version__ < '1.5':\n attens = torch.split(atten, int(rchannel//self.radix), dim=1)\n else:\n attens = torch.split(atten, rchannel//self.radix, dim=1)\n out = sum([att*split for (att, split) in zip(attens, splited)])\n else:\n out = atten * x\n return out.contiguous()\n\nclass rSoftMax(nn.Module):\n def __init__(self, radix, cardinality):\n super().__init__()\n self.radix = radix\n self.cardinality = cardinality\n\n def forward(self, x):\n batch = x.size(0)\n if self.radix > 1:\n x = x.view(batch, self.cardinality, self.radix, -1).transpose(1, 2)\n x = F.softmax(x, dim=1)\n x = x.reshape(batch, -1)\n else:\n x = torch.sigmoid(x)\n return x","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"class DropBlock2D(object):\n def __init__(self, *args, **kwargs):\n raise NotImplementedError\n\nclass GlobalAvgPool2d(nn.Module):\n def __init__(self):\n \"\"\"Global average pooling over the input's spatial dimensions\"\"\"\n super(GlobalAvgPool2d, self).__init__()\n\n def forward(self, inputs):\n return nn.functional.adaptive_avg_pool2d(inputs, 1).view(inputs.size(0), -1)\n\nclass Bottleneck(nn.Module):\n \"\"\"ResNet Bottleneck\n \"\"\"\n # pylint: disable=unused-argument\n expansion = 4\n def __init__(self, inplanes, planes, stride=1, downsample=None,\n radix=1, cardinality=1, bottleneck_width=64,\n avd=False, avd_first=False, dilation=1, is_first=False,\n rectified_conv=False, rectify_avg=False,\n norm_layer=None, dropblock_prob=0.0, last_gamma=False):\n super(Bottleneck, self).__init__()\n group_width = int(planes * (bottleneck_width / 64.)) * cardinality\n self.conv1 = nn.Conv2d(inplanes, group_width, kernel_size=1, bias=False)\n self.bn1 = norm_layer(group_width)\n self.dropblock_prob = dropblock_prob\n self.radix = radix\n self.avd = avd and (stride > 1 or is_first)\n self.avd_first = avd_first\n\n if self.avd:\n self.avd_layer = nn.AvgPool2d(3, stride, padding=1)\n stride = 1\n\n if dropblock_prob > 0.0:\n self.dropblock1 = DropBlock2D(dropblock_prob, 3)\n if radix == 1:\n self.dropblock2 = DropBlock2D(dropblock_prob, 3)\n self.dropblock3 = DropBlock2D(dropblock_prob, 3)\n\n if radix >= 1:\n self.conv2 = SplAtConv2d(\n group_width, group_width, kernel_size=3,\n stride=stride, padding=dilation,\n dilation=dilation, groups=cardinality, bias=False,\n radix=radix, rectify=rectified_conv,\n rectify_avg=rectify_avg,\n norm_layer=norm_layer,\n dropblock_prob=dropblock_prob)\n elif rectified_conv:\n from rfconv import RFConv2d\n self.conv2 = RFConv2d(\n group_width, group_width, kernel_size=3, stride=stride,\n padding=dilation, dilation=dilation,\n groups=cardinality, bias=False,\n average_mode=rectify_avg)\n self.bn2 = norm_layer(group_width)\n else:\n self.conv2 = nn.Conv2d(\n group_width, group_width, kernel_size=3, stride=stride,\n padding=dilation, dilation=dilation,\n groups=cardinality, bias=False)\n self.bn2 = norm_layer(group_width)\n\n self.conv3 = nn.Conv2d(\n group_width, planes * 4, kernel_size=1, bias=False)\n self.bn3 = norm_layer(planes*4)\n\n if last_gamma:\n from torch.nn.init import zeros_\n zeros_(self.bn3.weight)\n self.relu = nn.ReLU(inplace=True)\n self.downsample = downsample\n self.dilation = dilation\n self.stride = stride\n\n def forward(self, x):\n residual = x\n\n out = self.conv1(x)\n out = self.bn1(out)\n if self.dropblock_prob > 0.0:\n out = self.dropblock1(out)\n out = self.relu(out)\n\n if self.avd and self.avd_first:\n out = self.avd_layer(out)\n\n out = self.conv2(out)\n if self.radix == 0:\n out = self.bn2(out)\n if self.dropblock_prob > 0.0:\n out = self.dropblock2(out)\n out = self.relu(out)\n\n if self.avd and not self.avd_first:\n out = self.avd_layer(out)\n\n out = self.conv3(out)\n out = self.bn3(out)\n if self.dropblock_prob > 0.0:\n out = self.dropblock3(out)\n\n if self.downsample is not None:\n residual = self.downsample(x)\n\n out += residual\n out = self.relu(out)\n\n return out\n\nclass ResNet(nn.Module):\n \"\"\"ResNet Variants\n Parameters\n ----------\n block : Block\n Class for the residual block. Options are BasicBlockV1, BottleneckV1.\n layers : list of int\n Numbers of layers in each block\n classes : int, default 1000\n Number of classification classes.\n dilated : bool, default False\n Applying dilation strategy to pretrained ResNet yielding a stride-8 model,\n typically used in Semantic Segmentation.\n norm_layer : object\n Normalization layer used in backbone network (default: :class:`mxnet.gluon.nn.BatchNorm`;\n for Synchronized Cross-GPU BachNormalization).\n Reference:\n - He, Kaiming, et al. \"Deep residual learning for image recognition.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.\n - Yu, Fisher, and Vladlen Koltun. \"Multi-scale context aggregation by dilated convolutions.\"\n \"\"\"\n # pylint: disable=unused-variable\n def __init__(self, block, layers, radix=1, groups=1, bottleneck_width=64,\n num_classes=1000, dilated=False, dilation=1,\n deep_stem=False, stem_width=64, avg_down=False,\n rectified_conv=False, rectify_avg=False,\n avd=False, avd_first=False,\n final_drop=0.0, dropblock_prob=0,\n last_gamma=False, norm_layer=nn.BatchNorm2d):\n self.cardinality = groups\n self.bottleneck_width = bottleneck_width\n # ResNet-D params\n self.inplanes = stem_width*2 if deep_stem else 64\n self.avg_down = avg_down\n self.last_gamma = last_gamma\n # ResNeSt params\n self.radix = radix\n self.avd = avd\n self.avd_first = avd_first\n\n super(ResNet, self).__init__()\n self.rectified_conv = rectified_conv\n self.rectify_avg = rectify_avg\n if rectified_conv:\n from rfconv import RFConv2d\n conv_layer = RFConv2d\n else:\n conv_layer = nn.Conv2d\n conv_kwargs = {'average_mode': rectify_avg} if rectified_conv else {}\n if deep_stem:\n self.conv1 = nn.Sequential(\n conv_layer(3, stem_width, kernel_size=3, stride=2, padding=1, bias=False, **conv_kwargs),\n norm_layer(stem_width),\n nn.ReLU(inplace=True),\n conv_layer(stem_width, stem_width, kernel_size=3, stride=1, padding=1, bias=False, **conv_kwargs),\n norm_layer(stem_width),\n nn.ReLU(inplace=True),\n conv_layer(stem_width, stem_width*2, kernel_size=3, stride=1, padding=1, bias=False, **conv_kwargs),\n )\n else:\n self.conv1 = conv_layer(3, 64, kernel_size=7, stride=2, padding=3,\n bias=False, **conv_kwargs)\n self.bn1 = norm_layer(self.inplanes)\n self.relu = nn.ReLU(inplace=True)\n self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)\n self.layer1 = self._make_layer(block, 64, layers[0], norm_layer=norm_layer, is_first=False)\n self.layer2 = self._make_layer(block, 128, layers[1], stride=2, norm_layer=norm_layer)\n if dilated or dilation == 4:\n self.layer3 = self._make_layer(block, 256, layers[2], stride=1,\n dilation=2, norm_layer=norm_layer,\n dropblock_prob=dropblock_prob)\n self.layer4 = self._make_layer(block, 512, layers[3], stride=1,\n dilation=4, norm_layer=norm_layer,\n dropblock_prob=dropblock_prob)\n elif dilation==2:\n self.layer3 = self._make_layer(block, 256, layers[2], stride=2,\n dilation=1, norm_layer=norm_layer,\n dropblock_prob=dropblock_prob)\n self.layer4 = self._make_layer(block, 512, layers[3], stride=1,\n dilation=2, norm_layer=norm_layer,\n dropblock_prob=dropblock_prob)\n else:\n self.layer3 = self._make_layer(block, 256, layers[2], stride=2,\n norm_layer=norm_layer,\n dropblock_prob=dropblock_prob)\n self.layer4 = self._make_layer(block, 512, layers[3], stride=2,\n norm_layer=norm_layer,\n dropblock_prob=dropblock_prob)\n self.avgpool = GlobalAvgPool2d()\n self.drop = nn.Dropout(final_drop) if final_drop > 0.0 else None\n self.fc = nn.Linear(512 * block.expansion, num_classes)\n\n for m in self.modules():\n if isinstance(m, nn.Conv2d):\n n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels\n m.weight.data.normal_(0, math.sqrt(2. / n))\n elif isinstance(m, norm_layer):\n m.weight.data.fill_(1)\n m.bias.data.zero_()\n\n def _make_layer(self, block, planes, blocks, stride=1, dilation=1, norm_layer=None,\n dropblock_prob=0.0, is_first=True):\n downsample = None\n if stride != 1 or self.inplanes != planes * block.expansion:\n down_layers = []\n if self.avg_down:\n if dilation == 1:\n down_layers.append(nn.AvgPool2d(kernel_size=stride, stride=stride,\n ceil_mode=True, count_include_pad=False))\n else:\n down_layers.append(nn.AvgPool2d(kernel_size=1, stride=1,\n ceil_mode=True, count_include_pad=False))\n down_layers.append(nn.Conv2d(self.inplanes, planes * block.expansion,\n kernel_size=1, stride=1, bias=False))\n else:\n down_layers.append(nn.Conv2d(self.inplanes, planes * block.expansion,\n kernel_size=1, stride=stride, bias=False))\n down_layers.append(norm_layer(planes * block.expansion))\n downsample = nn.Sequential(*down_layers)\n\n layers = []\n if dilation == 1 or dilation == 2:\n layers.append(block(self.inplanes, planes, stride, downsample=downsample,\n radix=self.radix, cardinality=self.cardinality,\n bottleneck_width=self.bottleneck_width,\n avd=self.avd, avd_first=self.avd_first,\n dilation=1, is_first=is_first, rectified_conv=self.rectified_conv,\n rectify_avg=self.rectify_avg,\n norm_layer=norm_layer, dropblock_prob=dropblock_prob,\n last_gamma=self.last_gamma))\n elif dilation == 4:\n layers.append(block(self.inplanes, planes, stride, downsample=downsample,\n radix=self.radix, cardinality=self.cardinality,\n bottleneck_width=self.bottleneck_width,\n avd=self.avd, avd_first=self.avd_first,\n dilation=2, is_first=is_first, rectified_conv=self.rectified_conv,\n rectify_avg=self.rectify_avg,\n norm_layer=norm_layer, dropblock_prob=dropblock_prob,\n last_gamma=self.last_gamma))\n else:\n raise RuntimeError(\"=> unknown dilation size: {}\".format(dilation))\n\n self.inplanes = planes * block.expansion\n for i in range(1, blocks):\n layers.append(block(self.inplanes, planes,\n radix=self.radix, cardinality=self.cardinality,\n bottleneck_width=self.bottleneck_width,\n avd=self.avd, avd_first=self.avd_first,\n dilation=dilation, rectified_conv=self.rectified_conv,\n rectify_avg=self.rectify_avg,\n norm_layer=norm_layer, dropblock_prob=dropblock_prob,\n last_gamma=self.last_gamma))\n\n return nn.Sequential(*layers)\n\n def forward(self, x):\n x = self.conv1(x)\n x = self.bn1(x)\n x = self.relu(x)\n x = self.maxpool(x)\n\n x = self.layer1(x)\n x = self.layer2(x)\n x = self.layer3(x)\n x = self.layer4(x)\n\n x = self.avgpool(x)\n #x = x.view(x.size(0), -1)\n x = torch.flatten(x, 1)\n if self.drop:\n x = self.drop(x)\n x = self.fc(x)\n\n return x","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_model(args: tp.Dict):\n # # get resnest50_fast_1s1x64d\n model = ResNet(\n Bottleneck, [3, 4, 6, 3],\n radix=1, groups=1, bottleneck_width=64,\n deep_stem=True, stem_width=32, avg_down=True,\n avd=True, avd_first=True)\n \n del model.fc\n # # use the same head as the baseline notebook.\n model.fc = nn.Sequential(\n nn.Linear(2048, 1024), nn.ReLU(), nn.Dropout(p=0.2),\n nn.Linear(1024, 1024), nn.ReLU(), nn.Dropout(p=0.2),\n nn.Linear(1024, args[\"num_classes\"]))\n \n device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n model = model.to(device)\n state_dict = torch.load(args[\"trained_weights\"], map_location=device)\n model.load_state_dict(state_dict)\n \n model.eval()\n \n return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Prediction loop","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def prediction_for_clip(test_df: pd.DataFrame, \n clip: np.ndarray, \n model: ResNet, \n mel_params: dict, \n threshold=0.5,\n maxpreds=3, # New param --> @kkiller\n ):\n \n \"\"\"\n Original code: @hidehisaarai1213\n First refacto : @ttahara\n Second refacto: @kkiller\n \"\"\"\n\n dataset = TestDataset(df=test_df, \n clip=clip,\n img_size=224,\n melspectrogram_parameters=mel_params)\n loader = data.DataLoader(dataset, batch_size=1, shuffle=False)\n device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n \n model.eval()\n prediction_dict = {}\n for image, row_id, site in progress_bar(loader):\n site = site[0]\n row_id = row_id[0]\n if site in {\"site_1\", \"site_2\"}:\n image = image.to(device)\n\n with torch.no_grad():\n prediction = F.sigmoid(model(image))\n proba = prediction.detach().cpu().numpy().reshape(-1)\n\n events = proba >= threshold\n labels = np.argsort(-proba)[:events.sum()].tolist()\n\n else:\n # to avoid prediction on large batch\n image = image.squeeze(0)\n batch_size = 16\n whole_size = image.size(0)\n if whole_size % batch_size == 0:\n n_iter = whole_size // batch_size\n else:\n n_iter = whole_size // batch_size + 1\n \n all_events = set()\n probas = []\n for batch_i in range(n_iter):\n batch = image[batch_i * batch_size:(batch_i + 1) * batch_size]\n if batch.ndim == 3:\n batch = batch.unsqueeze(0)\n\n batch = batch.to(device)\n with torch.no_grad():\n prediction = F.sigmoid(model(batch))\n proba = prediction.detach().cpu().numpy()\n probas.append(proba)\n \n probas = np.vstack(probas)\n probas = probas.max(0)\n events = (probas>=threshold)\n labels = np.argsort(-probas)[:events.sum()].tolist()\n \n if len(labels) == 0:\n prediction_dict[row_id] = \"nocall\"\n else:\n labels_str_list = list(map(lambda x: INV_BIRD_CODE[x], labels))\n label_string = \" \".join(labels_str_list[:maxpreds])\n prediction_dict[row_id] = label_string\n return prediction_dict","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def prediction(test_df: pd.DataFrame,\n test_audio: Path,\n model_config: dict,\n mel_params: dict,\n target_sr: int,\n threshold=0.5,\n maxpreds = 3, # New param --> @kkiller\n ):\n model = get_model(model_config)\n unique_audio_id = test_df.audio_id.unique()\n\n warnings.filterwarnings(\"ignore\")\n prediction_dfs = []\n for audio_id in unique_audio_id:\n with timer(f\"Loading {audio_id}\"):\n clip, _ = librosa.load(test_audio / (audio_id + \".mp3\"),\n sr=target_sr,\n mono=True,\n res_type=\"kaiser_fast\")\n \n test_df_for_audio_id = test_df.query(\n f\"audio_id == '{audio_id}'\").reset_index(drop=True)\n with timer(f\"Prediction on {audio_id}\"):\n prediction_dict = prediction_for_clip(test_df_for_audio_id,\n clip=clip,\n model=model,\n mel_params=mel_params,\n threshold=threshold,\n maxpreds = maxpreds, # New param --> @kkiller\n )\n row_id = list(prediction_dict.keys())\n birds = list(prediction_dict.values())\n prediction_df = pd.DataFrame({\n \"row_id\": row_id,\n \"birds\": birds\n })\n prediction_dfs.append(prediction_df)\n \n prediction_df = pd.concat(prediction_dfs, axis=0, sort=False).reset_index(drop=True)\n return prediction_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Prediction","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"submission = prediction(test_df=test,\n test_audio=TEST_AUDIO_DIR,\n model_config=model_config,\n mel_params=melspectrogram_parameters,\n target_sr=TARGET_SR,\n threshold=0.56,\n maxpreds=2, # New param --> @kkiller\n )\nsubmission.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"submission","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"## EOF","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}