{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.11","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":91844,"databundleVersionId":11361821,"sourceType":"competition"},{"sourceId":245555259,"sourceType":"kernelVersion"}],"dockerImageVersionId":31041,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Setup\n\nImport the necessary modules and set up constants.","metadata":{}},{"cell_type":"code","source":"import os\nimport math\nimport json\nimport ast\n\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport IPython.display as ipd\nfrom sklearn.metrics import roc_auc_score\nfrom scipy import signal\nimport timm\n\nimport librosa\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.utils.data import Dataset, DataLoader\nfrom torchaudio.transforms import (\n    TimeMasking,\n    FrequencyMasking,\n    MelSpectrogram,\n    AmplitudeToDB\n)\nimport torchmetrics\nfrom torchmetrics.classification import (\n    MultilabelAccuracy,\n    MultilabelF1Score,\n    MultilabelPrecision,\n    MultilabelRecall\n)\nimport pytorch_lightning as pl\nfrom pytorch_lightning.callbacks import (\n    ModelCheckpoint, LearningRateMonitor, EarlyStopping\n)\n\nDATA_PATH = \"/kaggle/input/birdclef-2025\"\nEDA_CV_PATH = \"/kaggle/input/birdclef-2025-eda-cv-splits\"\nMODELS_PATH = \"/kaggle/working/birdclef-2025-effb0-cv-ensemble/weights\"\nCONFIG_PATH = \"/kaggle/working/birdclef-2025-models/train-configs\"\n\nMODEL_NAME = \"efficientnet_b0\"\nSAMPLE_RATE = 32000\n\n# Ensure output directories exist\nfor path in [MODELS_PATH, CONFIG_PATH]:\n    os.makedirs(path, exist_ok=True)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Function for displaying audio samples.","metadata":{}},{"cell_type":"code","source":"def plot_wave_spectrograms(waveforms, sample_rate, class_names, num_cols=2, specs=None):\n    \"\"\"\n    Plots the spectrograms and waveforms of given audio waveforms using a shared sample rate.\n    \n    Args:\n        waveforms (list of np.ndarray): List of audio waveforms (NumPy arrays of shape [samples, channels]).\n        sample_rate (int): Common sample rate for all waveforms.\n        class_names (list of str): List of corresponding class names.\n        num_cols (int): Number of columns in the plot layout. Default is 2.\n        specs (list of np.ndarray, optional): List of precomputed spectrograms (2D tensors, only `sxx` values).\n                                              If None, spectrograms will be computed automatically.\n    \"\"\"\n    num_files = len(waveforms)\n    num_rows = math.ceil(num_files / num_cols) * 2  # Each audio takes 2 rows (spectrogram + waveform)\n\n    fig, axs = plt.subplots(num_rows, num_cols, figsize=(num_cols * 2.6, num_rows * 2))\n\n    if num_rows == 2:\n        axs = np.reshape(axs, (num_rows, num_cols))  # Ensure correct indexing for small cases\n\n    for idx, (waveform, class_name) in enumerate(zip(waveforms, class_names)):\n\n        # Determine row and column indices\n        i, j = (idx // num_cols) * 2, idx % num_cols  # Spectrogram in row i, waveform in i+1\n        \n        # Compute spectrogram if not provided\n        if specs is None:\n            sampleFreqs, segmentTimes, sxx = signal.spectrogram(waveform, sample_rate)\n\n            # Plot spectrogram\n            axs[i][j].pcolormesh(segmentTimes, sampleFreqs, 10 * np.log10(sxx + 1e-15))\n            axs[i][j].set_title(f\"{class_name}\", fontsize=10)\n            axs[i][j].set_axis_off()\n        else:\n            # Plot spectrogram\n            axs[i][j].imshow(specs[idx])\n            axs[i][j].set_title(f\"{class_name}\", fontsize=10)\n            axs[i][j].set_axis_off()\n\n        # Plot waveform\n        axs[i + 1][j].plot(waveform)\n        axs[i + 1][j].set_axis_off()\n\n    plt.tight_layout()\n    plt.show()\n\n    # Play audio\n    for waveform, class_name in zip(waveforms, class_names):\n        print(f\"Playing: {class_name}\")\n        ipd.display(ipd.Audio(waveform, rate=sample_rate))","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# EDA & Cross-Validation\n\nMaybe I got a little too curious (don't worry - no cats were harmed) while exploring the data, so you'll find a separate **Data Exploration Notebook** here: [BirdCLEF+ 2025 | EDA | CV Splits](https://www.kaggle.com/code/mariadodonova/birdclef-2025-eda-cv-splits). Some key takeaways from a modelling perspective:\n\n* There is significantly greater diversity among birds compared to other classes.\n* There's a **strong imbalance** in the labels.\n* There are **duplicated audios** with different primary labels.\n* Only 9.33% of samples have more than one label.\n* The labels are sparse - the max number of labels per sample is 12 (out of 206 possible).\n\nIn that notebook, we fixed the duplicated audio issue, combined primary and secondary labels into a single column, and counted occurrences across *all* labels (not just primary ones).\n\n### Cross-Validation\n\nThe data were split into train/test subsets, with rare species excluded from the test set. Cross-validation folds (5 in total) were created using stratification by primary labels. This part is also covered in the [BirdCLEF+ 2025 | EDA | CV Splits](https://www.kaggle.com/code/mariadodonova/birdclef-2025-eda-cv-splits) notebook.\n\nNow, we're going to load the data from the preprocessed `birdclef_2025_cv_train.csv` file. Just to be safe, we'll also check whether all the file paths are still valid.","metadata":{}},{"cell_type":"code","source":"def parse_list_column(x):\n    items = ast.literal_eval(x)\n    if len(items) == 1 and items[0] == \"\":\n        items.pop()\n    return items\n\n# Load preprocessed CV splits\ndf = pd.read_csv(os.path.join(EDA_CV_PATH, \"birdclef_2025_cv_train.csv\"))\ndf = df[[\"filepath\", \"labels\", \"split\", \"fold\"]]\n\n# Parse labels and cast fold to nullable int\ndf[\"labels\"] = df[\"labels\"].apply(parse_list_column)\ndf[\"fold\"] = df[\"fold\"].astype(\"Int64\")\n\nassert all(os.path.exists(p) for p in df[\"filepath\"]), \"some audio file paths do not exist\"\n\ndf.head()","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Now, we load the updated taxonomy from the EDA notebook. It includes the number of label occurrences, which we'll need later.","metadata":{}},{"cell_type":"code","source":"df_taxonomy = pd.read_csv(os.path.join(EDA_CV_PATH, \"birdclef_2025_taxonomy.csv\"))\ndf_taxonomy.head()","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Let's also prepare mappings and other handy stuff for further use.","metadata":{}},{"cell_type":"code","source":"# Create mappings\nLABEL2NAME = {k: v for k, v in df_taxonomy[[\"primary_label\", \"common_name\"]].values}\nLABEL2IDX = {k: v for v, k in enumerate(df_taxonomy[\"primary_label\"].values)}\nIDX2LABEL = {k: v for v, k in LABEL2IDX.items()}\n\n# Total number of classes and label counts\nNUM_CLASSES = len(LABEL2IDX)\nLABEL_COUNTS = df_taxonomy[\"counts\"].values\nprint(f\"Number of unique labels found: {NUM_CLASSES}\")","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Dataset & Dataloader\n\n`BirdCLEFTrainDataset` handles loading the training audio and processing it in the time domain by applying simple augmentations (pitch shifting, time stretching, adding noise, applying MixUp) and chunking. It returns a processed waveform chunk of shape `(n_channels, n_chunk_samples)` along with its labels. In the case of testing data, the augmentations are skipped.\n\n#### Librosa Augmentations\n\n* The idea is to apply **time stretching** before chunking, so we don't lose information - for example, when the audio is longer than the chunk duration and is stretched with a rate > 1.0 (\"sped up\").\n  * Note: \"speeding up\" is in quotes because time stretching affects only the time domain - unlike typical speeding up, it doesn't alter the pitch.\n* To speed things up (and avoid wasting resources), we clip the audio before stretching: we take a random chunk of fixed size (slightly more than 5s to leave room for \"speeding up\").\n* The **pitch shift** is applied before time stretching to simulate a different \"speaker\" of the same species, and then we change the tempo.\n\nUnfortunately, the `librosa`'s pitch shifting and time stretching significantly slowed down training, so they weren't applied :(  \nBut this is definitely a field for further exploration.\n\n#### Padding\n\n* If the audio is longer than the required chunk size (5s), we take a random chunk from it.\n* If it's shorter, we apply padding using a randomly selected mode with different probabilities. This is done because of the different benefits each one may provide (or rather, the drawbacks the others introduce):\n  * `\"constant\"` -> the original sound and nothing more (but we could use more information if available);\n  * `\"wrap\"` -> repeats the original audio (may be unnatural for some species);\n  * `\"reflect\"` -> why not (though do the species sound symmetrical?).\n* For validation and test datasets, the padding mode is fixed to `\"constant\"`.\n\nExperiments showed worse results for `\"wrap\"` than for `\"reflect\"`. A possible explanation is the following: wrapping creates a constantly repeated sound, which is unnatural (birds need to breathe between phrases, after all). Reflecting introduces this pause - so maybe the reflected part isn't classified as the species, but it helps the overall soundscape feel more natural.\n\nInterestingly, combining all the methods still gave the best result.\n\n#### Other\n\n* **Noise** is added after all the transformations above. It's applied after time stretching and padding, since we don't expect to find digital silence in real soundscapes.\n* Finally, **normalization** is applied to the chunk.\n\n#### MixUp\n\nThe MixUp augmentation is performed using a fixed lambda value (chosen via tuning). Its probability is kept low to save computation time.\n\nFirst experiments showed that using MixUp gave worse results than not using it. But after fixing a few things elsewhere, it finally worked! Some possible reasons and ideas for making MixUp more effective:\n* lambda might need to be closer to 1.0 - values around 0.5 seem to hurt performance (though I'm not sure it even makes sense to apply MixUp with such small lambda - worth trying);\n* lambda could be sampled from a Beta distribution - as it was originally meant. I didn’t do this because I wanted full control over the mixing, but maybe a little bit of chaos can actually make things better;\n* the second sample could sometimes be more dominant - but I didn't want to lose focus on the main one;\n* MixUp could be smarter - mixing species that don't naturally occur together may confuse the model more than help.\n\nOne more thing I believe is a **problem** with training the model on a random chunk while assigning labels for the whole audio. The species might appear only in some chunks - or sometimes not at all.\n\nSo basically, what we're telling our model is: *Look, here probably are species 1, 2, maybe 3, and maybe 4!* But the model gets sad and disappointed because we lied: instead of bird songs, it gets just speech (which is interesting but unexpected) or even just noise.\n\nWithout MixUp, we'd get less misleading labels in this case.","metadata":{}},{"cell_type":"code","source":"class BirdCLEFTrainDataset(Dataset):\n    def __init__(\n        self,\n        df,\n        label2idx,\n        input_col=\"filepath\",\n        label_col=\"labels\",\n        test_mode=False,\n        sample_rate=32000,\n        chunk_duration=5,\n        augmentation_prob=0.5,\n        max_abs_pitch_shift_steps=2,\n        min_stretch_rate=0.8,\n        max_stretch_rate=1.2,\n        max_noise_rate=0.03,\n        padding_modes=[\"constant\", \"reflect\", \"wrap\"],\n        padding_probs=[0.4, 0.4, 0.2],\n        use_mixup=True,\n        mixup_lambda=0.9,\n        mixup_prob=0.3,\n        normalize_audio=True,\n        use_librosa=True\n    ):\n        self.df = df.reset_index(drop=True)\n        self.label2idx = label2idx\n\n        self.input_col = input_col\n        self.label_col = label_col\n        self.test_mode = test_mode\n        self.sample_rate = sample_rate\n        self.chunk_duration = chunk_duration\n        self.chunk_samples = self.chunk_duration * self.sample_rate\n        self.augmentation_prob = augmentation_prob\n\n        self.min_stretch_rate = min_stretch_rate\n        self.max_stretch_rate = max_stretch_rate\n        self.max_abs_pitch_shift_steps=max_abs_pitch_shift_steps\n        self.max_noise_rate = max_noise_rate\n        self.padding_modes = padding_modes\n        self.padding_probs = padding_probs\n        self.use_mixup = use_mixup\n        self.mixup_lambda = mixup_lambda\n        self.mixup_prob = mixup_prob\n        self.normalize_audio = normalize_audio\n        self.use_librosa = use_librosa\n\n    def _clip(self, audio, length):\n        # Cut a random chunk of the given length from the audio\n        if len(audio) > length:\n            start = np.random.randint(0, audio.size - length + 1)\n            audio = audio[start:start + length]\n        return audio\n    \n    def _pitch_shift(self, audio):\n        # Random pitch shift in semitone steps (up/down)\n        n_steps = np.random.uniform(\n            -self.max_abs_pitch_shift_steps,\n            self.max_abs_pitch_shift_steps\n        )\n        return librosa.effects.pitch_shift(audio, sr=self.sample_rate, n_steps=n_steps)\n\n    def _time_stretch(self, audio):\n        # Randomly stretch or compress audio in time\n        rate = np.random.uniform(self.min_stretch_rate, self.max_stretch_rate)\n        return librosa.effects.time_stretch(audio, rate=rate)\n\n    def _add_noise(self, audio):\n        # Add Gaussian noise scaled to a random proportion of signal std\n        rate = np.random.uniform(0, self.max_noise_rate)\n        noise = np.random.normal(0, audio.std(), audio.size)\n        return audio + rate * noise\n\n    def _select_chunk(self, audio):\n        padding_size = self.chunk_samples - audio.size\n\n        if padding_size > 0:\n            # Pad the end of the audio if it's too short\n            if not self.test_mode:\n                mode = np.random.choice(self.padding_modes, p=self.padding_probs)\n            else:\n                mode = \"constant\"\n            audio = np.pad(audio, (0, padding_size), mode=mode)\n\n        elif padding_size < 0:\n            # Crop a random chunk if audio is too long\n            audio = self._clip(audio, self.chunk_samples)\n\n        return audio\n\n    def _prepare_features(self, idx):\n        # Load the audio file\n        audio_path = self.df.loc[idx, self.input_col]\n        audio, sr = librosa.load(audio_path, sr=self.sample_rate)\n\n        # Sanity checks\n        assert sr == self.sample_rate, \"loaded sample has invalid sample rate\"\n        assert len(audio.shape) == 1, \"loaded sample has invalid shape\"\n\n        if self.use_librosa and not self.test_mode:\n            # To make augmentation faster, we first clip to a slightly longer chunk\n            # If the clip is later stretched (\"sped up\"), it'll still be long enough\n            max_length = int(self.chunk_samples * self.max_stretch_rate)\n            audio = self._clip(audio, max_length)\n            \n            # Random pitch shift\n            if np.random.random() < self.augmentation_prob:\n                audio = self._pitch_shift(audio)\n\n            # Random time stretch\n            if np.random.random() < self.augmentation_prob:\n                audio = self._time_stretch(audio)\n        \n        # Pad or clip to exactly self.chunk_samples length\n        audio = self._select_chunk(audio)\n\n        assert audio.size == self.chunk_samples, \"chunk length is incorrect\"\n\n        # Add noise if in training mode\n        if not self.test_mode and np.random.random() < self.augmentation_prob:\n            audio = self._add_noise(audio)\n        \n        # Normalize if enabled\n        if self.normalize_audio:\n            audio = librosa.util.normalize(audio)\n\n        # Convert to shape (1, num_samples) tensor\n        return torch.from_numpy(audio).float().unsqueeze(0)\n\n    def _prepare_labels(self, idx):\n        # Convert list of label strings into a binary multi-hot vector\n        labels = self.df.loc[idx, self.label_col]\n        label_ids = np.array([self.label2idx[x] for x in labels])\n\n        all_labels = np.zeros(len(self.label2idx))\n        all_labels[label_ids] = 1\n\n        return torch.from_numpy(all_labels).float()\n\n    def _mixup(self, features, labels, idx):\n        # Sample a different index to mix with\n        other_idx = np.random.randint(0, len(self.df))\n        if other_idx == idx:\n            return features, labels\n\n        # Get the second sample\n        other_features = self._prepare_features(other_idx)\n        other_labels = self._prepare_labels(other_idx)\n\n        # Mix both samples (features and labels)\n        mixed_features = self.mixup_lambda * features + (1 - self.mixup_lambda) * other_features\n        mixed_labels = self.mixup_lambda * labels + (1 - self.mixup_lambda) * other_labels\n\n        return mixed_features, mixed_labels\n    \n    def __getitem__(self, idx):\n        features = self._prepare_features(idx)\n        labels = self._prepare_labels(idx)\n\n        # Apply Mixup augmentation\n        if self.use_mixup and not self.test_mode and np.random.random() < self.mixup_prob:\n            features, labels = self._mixup(features, labels, idx)\n\n        return {\n            \"input_features\": features,\n            \"labels\": labels\n        }\n\n    def __len__(self):\n        return len(self.df)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Check that the dataset is working correctly.","metadata":{}},{"cell_type":"code","source":"def get_names(labels):\n    names_list = []\n    for labs in labels:\n        names = torch.where(labs > 0)[0].tolist()\n        names = [LABEL2NAME[IDX2LABEL[name]] for name in names]\n        # Truncate long label lists for cleaner plots\n        if len(names) > 1:\n            names = names[:1] + [\"...\"]\n        names_list.append(\", \".join(names))\n    return names_list\n\ndef test_dataset(dataset, idx=10):\n    features = dataset[idx][\"input_features\"]\n    labels = dataset[idx][\"labels\"]\n\n    print(f\"Dataset length: {len(dataset)} samples\")\n    print(f\"Sample at index {idx}:\\n\")\n\n    print(f\"Input shape: {features.shape} (channels, samples)\")\n    print(\"Input tensor:\")\n    print(features)\n\n    print(f\"\\nLabel shape: {labels.shape} (num_classes)\")\n    print(\"Label tensor:\")\n    print(labels)\n\n    plot_wave_spectrograms(\n        features,\n        sample_rate=SAMPLE_RATE,\n        class_names=get_names([labels]),\n        num_cols=1\n    )\n\ntrain_ds = BirdCLEFTrainDataset(df[df[\"split\"] == \"TRAIN\"], LABEL2IDX)\ntest_dataset(train_ds)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Let's check that the dataloader is working as expected.","metadata":{}},{"cell_type":"code","source":"def test_dataloader(dataloader):\n    for batch in dataloader:\n        features = batch[\"input_features\"]\n        labels = batch[\"labels\"]\n\n        print(f\"Batch input shape: {features.shape} (batch_size, channels, samples)\")\n        print(\"First input tensor:\")\n        print(features[0])\n    \n        print(f\"\\nBatch label shape: {labels.shape} (batch_size, num_classes)\")\n        print(\"First label tensor:\")\n        print(labels[0])\n\n        plot_wave_spectrograms(\n            features.squeeze(1), # remove channel dim for plotting (B, T)\n            sample_rate=SAMPLE_RATE,\n            class_names=get_names(labels),\n            num_cols=len(features)\n        )\n\n        break\n\ntrain_loader = DataLoader(train_ds, batch_size=2, shuffle=False, num_workers=3)\ntest_dataloader(train_loader)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Modelling\n\nIn this work, we use a pretrained EfficientNet B0 model from `timm` as a backbone. To feed it our data, we first need to convert the waveforms into spectrograms.\n\n## Mel Spectrogram Extractor\n\nThe `MelSpecExtractor` extracts mel spectrograms (captain obvious), transforms amplitudes to dB, and applies min-max normalization. It also performs time and frequency masking on each sample in the batch - but only during training.\n\nIt could be part of the main model, but we decided to keep it as a separate instance. This seems more logical in terms of the responsibilities of each component in training (why should the model convert audios to spectrograms and augment its own data? what if it starts cheating? :)  \nSeparating it also helps reduce computations when using *model ensembles*.","metadata":{}},{"cell_type":"code","source":"class SpecNormalization(nn.Module):\n    def __init__(self, eps=1e-6):\n        super().__init__()\n        self.eps = eps\n\n    def forward(self, x):\n        max_vals = torch.amax(x, dim=(-2, -1), keepdim=True)\n        min_vals = torch.amin(x, dim=(-2, -1), keepdim=True)\n        x = (x - min_vals) / (max_vals - min_vals + self.eps)\n        return x\n\n\nclass Masking(nn.Module):\n    def __init__(self, max_masks_num, mask_module, p):\n        super().__init__()\n        self.max_masks_num = max_masks_num\n        self.mask_module = mask_module\n        self.p = p\n\n    def forward(self, x):\n        for i in range(x.shape[0]):\n            if np.random.random() < self.p:\n                n_applies = np.random.randint(low=1, high=self.max_masks_num + 1)\n                for _ in range(n_applies):\n                    x[i : i + 1][0] = self.mask_module(x[i : i + 1][0])\n        return x\n\n\nclass MelSpecExtractor(nn.Module):\n    def __init__(\n        self,\n        sample_rate=32000,\n        n_fft=1024,\n        hop_length=512,\n        n_mels=128,\n        top_db=80.0,\n        augmentation_prob=0.5,\n        time_mask_param=20,\n        max_time_masks=5,\n        freq_mask_param=20,\n        max_freq_masks=5\n    ):\n        super().__init__()\n\n        self.extractor = nn.Sequential(\n            MelSpectrogram(\n                sample_rate=sample_rate,\n                n_fft=n_fft,\n                hop_length=hop_length,\n                n_mels=n_mels\n            ),\n            AmplitudeToDB(top_db=top_db),\n            SpecNormalization()\n        )\n\n        # Time and frequency masking augmentations\n        self.time_masking = Masking(\n            max_time_masks, \n            TimeMasking(time_mask_param=time_mask_param),\n            augmentation_prob\n        )\n        self.freq_masking = Masking(\n            max_freq_masks,\n            FrequencyMasking(freq_mask_param=freq_mask_param),\n            augmentation_prob\n        )\n\n    def forward(self, x, test_mode=False):\n        x = self.extractor(x)\n        \n        if not test_mode:\n            x = self.time_masking(x)\n            x = self.freq_masking(x)\n        return x","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Check, whether the `MelSpecExtractor` is working correctly.","metadata":{}},{"cell_type":"code","source":"def test_spec_extractor(spec_extractor, dataloader):\n    for batch in dataloader:\n        features = batch[\"input_features\"]\n        labels = batch[\"labels\"]\n\n        specs = spec_extractor(features)\n\n        print(f\"Spectrograms shape: {specs.shape} (batch_size, channels, mel_bins, time)\")\n        print(\"First spectrogram:\")\n        print(specs[0])\n\n        plot_wave_spectrograms(\n            features.squeeze(1),\n            sample_rate=SAMPLE_RATE,\n            class_names=get_names(labels),\n            num_cols=len(features),\n            specs=[torch.flip(spec.squeeze(0), (0,)) for spec in specs]\n        )\n        break\n\nspec_extractor = MelSpecExtractor()\ntest_spec_extractor(spec_extractor, train_loader)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## BirdCLEF Model\n\nThe `BirdCLEFModel` uses EfficientNet B0 from `timm` as backbone, with a pooling layer and a linear layer as the classification head.\n\nTwo pooling strategies are available: the classic `AdaptiveAvgPool2d`, and the real *hidden gem* of this setup - `GeneralizedMeanPool`. Swapping in GeM instead of average pooling led to noticeably better results: +0.026 Local / +0.024 Private / +0.018 Public Score.\n\nIn terms of modelling, making the classification head slightly more complex might help improve performance. I also had the idea to experiment with other feature layers of the backbone. I don't really expect it to show better results - but who knows?","metadata":{}},{"cell_type":"code","source":"class GeneralizedMeanPool(nn.Module):\n    def __init__(self, p=3.0, eps=1e-6):\n        super().__init__()\n        self.eps = eps\n        self.p = nn.Parameter(torch.ones(1) * p)\n\n    def forward(self, x):\n        x = x.clamp(min=self.eps)\n        x = x.pow(self.p)\n        x = F.avg_pool2d(x, (x.size(-2), x.size(-1)))\n        x = x.pow(1. / self.p)\n        return x\n\n\nclass BirdCLEFModel(nn.Module):\n    def __init__(\n        self,\n        backbone,\n        num_classes,\n        pretrained=True,\n        use_gem_pooling=True,\n        gem_p=3.0\n    ):\n        super().__init__()\n        self.num_classes = num_classes\n        self.use_gem_pooling = use_gem_pooling\n\n        self.backbone = timm.create_model(\n            backbone,\n            features_only=True,\n            pretrained=pretrained,\n            in_chans=1\n        )\n        n_features = self.backbone.feature_info.channels()[-1]\n\n        if self.use_gem_pooling:\n            self.pool = GeneralizedMeanPool(p=gem_p) \n        else:\n            self.pool = nn.AdaptiveAvgPool2d(1)\n\n        self.classifier = nn.Linear(n_features, num_classes)\n\n    def forward(self, x):\n        x = self.backbone(x)[-1]\n        x = self.pool(x).squeeze(-1).squeeze(-1) # (B, C, 1, 1) -> (B, C)\n        logits = self.classifier(x)\n        return logits","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Don't Lose Focus\n\nSince the task is multilabel classification and the classes are highly imbalanced, we use `FocalLoss` in this work. It focuses on learning from hard examples - those the model struggles to classify correctly - which are often the rare classes.\n\nAn interesting part here is the `alpha` weighting factor. Usually, it's used to address the imbalance between positive and negative examples for each class, treating all classes in the same way. But in our case, there's a strong class imbalance across the dataset. So, using per-class weights makes more sense - and they should depend on how frequent the classes are in the data.\n\nThat's why we needed label counts. We used the inverse log of class counts as per-class weights: this approach is smoothed, avoids extreme values for rare classes, and actually gave better results than the commonly used `alpha = 0.25` on Local Validation and Public Score. It also significantly improved recall. But unfortunately, it didn't help on the Private Score:  \n+0.012 Local / -0.018 Private / +0.015 Public Score.\n\nThat said, since the labels are sparse, it might be helpful to combine the per-class alpha with a regular one, or compute `alpha_t` based on the per-class weights. Also, there might be better functions for calculating the weights - so this area definitely deserves more exploration.\n\nIt would be interesting to compare the results to not using balancing at all.","metadata":{}},{"cell_type":"code","source":"ALPHA_PC_LOG_INVERSE = 1 / np.log(LABEL_COUNTS + 1)\n\nclass FocalLoss(nn.Module):\n    def __init__(\n        self,\n        gamma=2.0,\n        reduction=\"mean\",\n        alpha=0.25,\n        alpha_per_class=None\n    ):\n        super().__init__()\n        self.gamma = gamma\n        self.reduction = reduction\n        self.alpha = alpha\n        self.alpha_per_class = alpha_per_class\n        if self.alpha_per_class is not None:\n            self.alpha_per_class = torch.tensor(self.alpha_per_class).float()\n            \n\n    def forward(self, logits, labels):\n        bce_loss = F.binary_cross_entropy_with_logits(logits, labels.float(), reduction=\"none\")\n\n        probs = torch.sigmoid(logits)\n        p_t = probs * labels + (1 - probs) * (1 - labels) # p_t = prob if label==1 else 1-prob\n        if self.alpha_per_class is not None:\n            alpha_t = self.alpha_per_class\n        else:\n            alpha_t = self.alpha * labels + (1 - self.alpha) * (1 - labels)\n        loss = alpha_t.to(logits.device) * (1 - p_t) ** self.gamma * bce_loss\n\n        if self.reduction == \"mean\":\n            return loss.mean()\n        elif self.reduction == \"sum\":\n            return loss.sum()\n        return loss","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Evaluation Metrics\n\nOf course, we use the main competition metric - ROC AUC. The `IgnoreEmptyROCAUC` is a wrapper around `sklearn.metrics.roc_auc_score()`, which ignores classes without any true labels. While `torchmetrics` provides built-in `MultilabelAUROC`, we wanted to use exactly the same function as in the competition.\n\nIn addition, we'll compute `MultilabelAccuracy`, `MultilabelF1Score`, `MultilabelPrecision`, and `MultilabelRecall` from `torchmetrics.classification`.","metadata":{}},{"cell_type":"code","source":"class IgnoreEmptyROCAUC(torchmetrics.Metric):\n    def __init__(self, num_labels, average=\"macro\"):\n        super().__init__()\n        self.num_labels = num_labels\n        self.average = average\n\n        self.add_state(\"preds\", default=[], dist_reduce_fx=\"cat\")\n        self.add_state(\"targets\", default=[], dist_reduce_fx=\"cat\")\n\n    def update(self, preds, targets):\n        self.preds.append(preds.detach().cpu())\n        self.targets.append(targets.detach().cpu())\n\n    def compute(self):\n        preds = torch.cat(self.preds, dim=0)\n        targets = torch.cat(self.targets, dim=0)\n\n        # Ignore classes with no positive samples\n        mask = targets.sum(dim=0) > 0\n        if not mask.any():\n            return torch.tensor(float(\"nan\"))\n\n        preds_np = preds[:, mask].numpy()\n        targets_np = torch.round(targets[:, mask]).int().numpy()\n\n        auc = roc_auc_score(targets_np, preds_np, average=self.average)\n        return torch.tensor(auc)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Lit Stuff\n\nThe `BirdCLEFLitTrainer` includes both the trainable model and the `MelSpecExtractor`. It uses `FocalLoss` as the loss function, the `AdamW` optimizer, and a `CosineAnnealingLR` scheduler.\n\nDuring the training step, the loss is computed and logged. If the model uses GeM pooling, its parameter is logged as well. In the validation and test steps, both the loss and predicted probabilities are computed. Metrics are then calculated and logged at the end of each validation and test epoch.","metadata":{}},{"cell_type":"code","source":"class BirdCLEFLitTrainer(pl.LightningModule):\n    def __init__(\n        self,\n        model,\n        steps_per_epoch,\n        spec_extractor_configs,\n        loss_configs,\n        learning_rate=1e-3,\n        eta_min=1e-6\n    ):\n        super().__init__()\n        self.save_hyperparameters(\"steps_per_epoch\", \"learning_rate\", \"eta_min\")\n\n        self.spec_extractor = MelSpecExtractor(**spec_extractor_configs)\n        self.model = model\n        self.criterion = FocalLoss(**loss_configs)\n\n        # Metrics\n        average = \"weighted\"\n        num_classes = self.model.num_classes\n        self.val_metrics = torchmetrics.MetricCollection({\n            \"accuracy\": MultilabelAccuracy(num_labels=num_classes, average=average),\n            \"f1\": MultilabelF1Score(num_labels=num_classes, average=average),\n            \"precision\": MultilabelPrecision(num_labels=num_classes, average=average),\n            \"recall\": MultilabelRecall(num_labels=num_classes, average=average),\n            \"roc_auc\": IgnoreEmptyROCAUC(num_labels=num_classes, average=\"macro\")\n        }, prefix=\"val_\")\n        self.test_metrics = self.val_metrics.clone(prefix=\"test_\")\n\n    def forward(self, input_features, test_mode=False):\n        spec_features = self.spec_extractor(input_features, test_mode)\n        return self.model(spec_features)\n\n    def training_step(self, batch, batch_idx):\n        loss = self._compute_loss_and_probs(batch, return_probs=False)\n        self.log(\"train_loss\", loss, on_step=True, on_epoch=True, prog_bar=True)\n        if self.model.use_gem_pooling:\n            self.log(\"train_gem_p\", self.model.pool.p.item(), on_step=True, on_epoch=False, prog_bar=True)\n        return loss\n\n    def validation_step(self, batch, batch_idx):\n        labels = batch[\"labels\"]\n        loss, probs = self._compute_loss_and_probs(batch, test_mode=True)\n\n        self.val_metrics.update(probs, labels)\n        self.log(\"val_loss\", loss, on_step=True, on_epoch=True, prog_bar=True)\n\n        return loss\n\n    def on_validation_epoch_end(self):\n        metric_values = self.val_metrics.compute()\n        self.log_dict(metric_values, on_step=False, on_epoch=True, prog_bar=True)\n        self.val_metrics.reset()\n\n    def test_step(self, batch, batch_idx):\n        labels = batch[\"labels\"]\n        loss, probs = self._compute_loss_and_probs(batch, test_mode=True)\n\n        self.test_metrics.update(probs, labels)\n        self.log(\"test_loss\", loss, on_step=True, on_epoch=True, prog_bar=True)\n\n        return loss\n\n    def on_test_epoch_end(self):\n        metric_values = self.test_metrics.compute()\n        self.log_dict(metric_values, on_step=False, on_epoch=True, prog_bar=True)\n        self.test_metrics.reset()\n\n    def _compute_loss_and_probs(self, batch, return_probs=True, test_mode=False):\n        input_features = batch[\"input_features\"]\n        labels = batch[\"labels\"]\n\n        logits = self(input_features, test_mode)\n        loss = self.criterion(logits, labels)\n\n        # Optionally return probabilities\n        if return_probs:\n            return loss, torch.sigmoid(logits)\n\n        return loss\n\n    def configure_optimizers(self):\n        optimizer = torch.optim.AdamW(self.model.parameters(), self.hparams.learning_rate)\n\n        # Cosine Annealing learning rate scheduler\n        total_steps = self.trainer.max_epochs * self.hparams.steps_per_epoch\n        scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(\n            optimizer,\n            T_max=total_steps,\n            eta_min=self.hparams.eta_min\n        )\n\n        return {\n            \"optimizer\": optimizer,\n            \"lr_scheduler\": {\n                \"scheduler\": scheduler,\n                \"interval\": \"step\",\n                \"frequency\": 1\n            }\n        }","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Cross-Validation Training\n\nSetting up the main training part - we will need a bunch of functions.\n\nWhile training, we will use callbacks as `ModelCheckpoint` to save the best model, `LearningRateMonitor`, and `EarlyStopping` with patience of 3 epochs.","metadata":{}},{"cell_type":"code","source":"def prepare_dataloader(\n    df,\n    dataset_configs,\n    dataloader_configs,\n    test_mode=True,\n    shuffle=False\n):\n    dataset = BirdCLEFTrainDataset(df, test_mode=test_mode, **dataset_configs)\n    dataloader = DataLoader(dataset, shuffle=shuffle, **dataloader_configs)\n    return dataloader\n\n\ndef prepare_cv_dataloaders(\n    df,\n    val_fold,\n    fold_col,\n    dataset_configs,\n    dataloader_configs\n):\n    # Prepare training DataLoader using all folds except the validation fold\n    train_loader = prepare_dataloader(\n        df[df[fold_col] != val_fold],\n        dataset_configs,\n        dataloader_configs,\n        test_mode=False,\n        shuffle=True\n    )\n\n    # Prepare validation DataLoader using only the selected validation fold\n    valid_loader = prepare_dataloader(\n        df[df[fold_col] == val_fold],\n        dataset_configs,\n        dataloader_configs,\n        test_mode=True,\n        shuffle=False\n    )\n\n    return train_loader, valid_loader\n\n\ndef prepare_trainer(trainer_configs, val_fold):\n    monitor = \"val_roc_auc\"\n    mode = \"max\"\n\n    all_callbacks = [\n        ModelCheckpoint(\n            dirpath=f\"/kaggle/working/checkpoints/fold_{val_fold}\",\n            filename=\"{epoch}-{val_roc_auc:.4f}\",\n            save_top_k=1,\n            monitor=monitor,\n            mode=mode\n        ),\n        LearningRateMonitor(logging_interval=\"step\"),\n        EarlyStopping(\n            patience=3,\n            monitor=monitor,\n            mode=mode\n        )\n    ]\n\n    trainer = pl.Trainer(\n        callbacks=all_callbacks,\n        **trainer_configs\n    )\n\n    # Return trainer and model checkpoint callback\n    return trainer, all_callbacks[0]","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"We train five `BirdCLEFModel` models on different folds, load the best-performing ones, and test them on their respective validation folds as well as on the general test split. Finally, we save their weights for use in the submission.\n\nNow comes the main training function.","metadata":{}},{"cell_type":"code","source":"def train_with_cv(\n    df,\n    split_col,\n    fold_col,\n    dataset_configs,\n    dataloader_configs,\n    birdclef_model_configs,\n    spec_extractor_configs,\n    loss_configs,\n    lit_model_configs,\n    trainer_configs,\n    test_on_valid=True\n):\n    # Prepare dataloader for the TEST split\n    test_loader = prepare_dataloader(\n        df[df[split_col] == \"TEST\"],\n        dataset_configs,\n        dataloader_configs,\n        test_mode=True,\n        shuffle=False\n    )\n\n    # Filter only the TRAIN split and get sorted list of folds\n    train_df = df[df[split_col] == \"TRAIN\"]\n    folds = sorted(set(train_df[fold_col]))\n    all_results = []\n\n    for val_fold in folds:\n        print(f\"Fold {val_fold}\")\n\n        # Prepare training and validation dataloaders for the current fold\n        train_loader, valid_loader = prepare_cv_dataloaders(\n            train_df,\n            val_fold,\n            fold_col,\n            dataset_configs,\n            dataloader_configs\n        )\n        steps_per_epoch = len(train_loader)\n\n        # Initialize base model and LightningModule wrapper\n        birdclef_model = BirdCLEFModel(**birdclef_model_configs)\n        lit_model = BirdCLEFLitTrainer(\n            birdclef_model,\n            steps_per_epoch,\n            spec_extractor_configs,\n            loss_configs,\n            **lit_model_configs\n        )\n\n        # Prepare trainer and callbacks for current fold\n        trainer, checkpoint_cb = prepare_trainer(trainer_configs, val_fold)\n        \n        # Train the model on current fold\n        trainer.fit(lit_model, train_loader, valid_loader)\n\n        # Load the best model from checkpoint after training\n        best_model = BirdCLEFLitTrainer.load_from_checkpoint(\n            checkpoint_cb.best_model_path,\n            model=birdclef_model,\n            steps_per_epoch=steps_per_epoch,\n            spec_extractor_configs=spec_extractor_configs,\n            loss_configs=loss_configs,\n            **lit_model_configs\n        )\n        \n        # Optionally test the best model on validation fold\n        if test_on_valid:\n            print(f\"Results of the best model for the {val_fold} fold\")\n            val_results = trainer.test(best_model, dataloaders=valid_loader)[0]\n\n        # Test the best model on the TEST split\n        print(f\"Results of the best model (fold {val_fold}) for the TEST split\")\n        results = trainer.test(best_model, dataloaders=test_loader)[0]\n\n        # Save only the internal model weights\n        filename = f\"val_fold_{val_fold}-test_roc_auc={results['test_roc_auc']:.4f}.pth\"\n        save_path = os.path.join(MODELS_PATH, filename)\n        torch.save(best_model.model.state_dict(), save_path)\n\n        # Merge validation results into results\n        if test_on_valid:\n            val_results = {\n                key.replace(\"test_\", \"val_fold_\"): value\n                for key, value in val_results.items()\n            }\n            results.update(val_results)\n\n        # Log model name and fold number\n        results[\"model\"] = filename\n        results[\"val_fold\"] = val_fold\n\n        all_results.append(results)\n\n    return all_results","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"We're almost there - just need to set the configs and save them.","metadata":{}},{"cell_type":"code","source":"dataset_configs = {\n    \"label2idx\": LABEL2IDX,\n    \"input_col\": \"filepath\",\n    \"label_col\": \"labels\",\n    \"sample_rate\": SAMPLE_RATE,\n    \"chunk_duration\": 5,\n    \"augmentation_prob\": 0.3,\n    \"max_abs_pitch_shift_steps\": 2,\n    \"min_stretch_rate\": 0.8,\n    \"max_stretch_rate\": 1.2,\n    \"max_noise_rate\": 0.03,\n    \"padding_modes\": [\"constant\", \"reflect\", \"wrap\"],\n    \"padding_probs\": [0.4, 0.4, 0.2],\n    \"normalize_audio\": True,\n    \"use_mixup\": True,\n    \"mixup_lambda\": 0.9,\n    \"mixup_prob\": 0.3,\n    \"use_librosa\": False\n}\n\ndataloader_configs = {\n    \"batch_size\": 32,\n    \"num_workers\": 3,\n    \"pin_memory\": True\n}\n\nspec_extractor_configs = {\n    \"sample_rate\": SAMPLE_RATE,\n    \"n_fft\": 1024,\n    \"hop_length\": 512,\n    \"n_mels\": 128,\n    \"top_db\": 80.0,\n    \"augmentation_prob\": 0.3,\n    \"time_mask_param\": 20,\n    \"max_time_masks\": 3,\n    \"freq_mask_param\": 10,\n    \"max_freq_masks\": 3\n}\n\nbirdclef_model_configs = {\n    \"backbone\": MODEL_NAME,\n    \"num_classes\": NUM_CLASSES,\n    \"pretrained\": True,\n    \"use_gem_pooling\": True,\n    \"gem_p\": 3.0\n}\n\nloss_configs = {\n    \"gamma\": 2.0,\n    \"reduction\": \"mean\",\n    \"alpha\": 0.25,\n    \"alpha_per_class\": None # ALPHA_PC_LOG_INVERSE.tolist()\n}\n\nlit_model_configs = {\n    \"learning_rate\": 1e-3,\n    \"eta_min\": 1e-5\n}\n\ntrainer_configs = {\n    \"max_epochs\": 5,\n    \"accelerator\": \"cuda\" if torch.cuda.is_available() else \"cpu\",\n    \"devices\": 1\n}","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def save_configs(configs, name):\n    filename = name + \".json\"\n    path = os.path.join(CONFIG_PATH, filename)\n    with open(path, \"w\") as f:\n        json.dump(configs, f, indent=4)\n\nsave_configs(dataset_configs, \"dataset.json\")\nsave_configs(dataloader_configs, \"dataloader.json\")\nsave_configs(spec_extractor_configs, \"spec_extractor.json\")\nsave_configs(birdclef_model_configs, \"birdclef_model.json\")\nsave_configs(loss_configs, \"focal_loss.json\")\nsave_configs(lit_model_configs, \"lit_model.json\")\nsave_configs(trainer_configs, \"trainer.json\")","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Showtime!","metadata":{}},{"cell_type":"code","source":"results = train_with_cv(\n    df,\n    \"split\",\n    \"fold\",\n    dataset_configs,\n    dataloader_configs,\n    birdclef_model_configs,\n    spec_extractor_configs,\n    loss_configs,\n    lit_model_configs,\n    trainer_configs,\n    test_on_valid=True\n)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Results and Outcomes\n\nLet's take a look at what we've got.","metadata":{}},{"cell_type":"code","source":"df_results = pd.DataFrame(results).set_index(\"val_fold\")\ndf_results.to_csv(\"/kaggle/working/results.csv\", index=True)\n\ndf_results","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"As expected with imbalanced data, we observe high accuracy. Recall is lower than precision, which is expected given the label sparsity - we get more false negatives than false positives.\n\nIn the **Submission Notebook** (BirdCLEF+ 2025 | Submission), we use an ensemble of models trained during cross-validation. It performs better than individual models and provides more stable results by combining predictions from different \"experts\". The final predictions are obtained by averaging the probability outputs from all models.\n\nAfter long hours (days, *weeks*?) of working on it, this lab now feels like such a *basic-basic base-base baseline*, because there are still so many things left to explore and discover. I've left some thoughts throughout the notebook, but here are a few more ideas:\n* use more training epochs;\n* try using different models;\n* explore ensemble strategies beyond simple averaging;\n* come up with some out-of-the-box augmentation ideas;\n* train models on the high-rated data first;\n* find a way to focus on chunks with a high probability of containing the species.\n\nAnd of course, there's always room for more parameter tuning :)","metadata":{}}]}