{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":19596,"databundleVersionId":1292430,"sourceType":"competition"}],"dockerImageVersionId":30840,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Bird Audio Classification with EfficientNet-B7 and SpecAugment\n\n### Overview\nThis project implements a deep learning-based audio classification model using **EfficientNet-B7** for bird sound classification. The approach leverages **transfer learning** and **SpecAugment** for improved feature extraction and robustness.\n\nThe workflow consists of:\n- **Preprocessing Audio Data**: Converting bird sound recordings into **Mel spectrograms**.\n- **Data Augmentation**: Applying **SpecAugment** (time and frequency masking).\n- **Model Architecture**: Utilizing a **pretrained EfficientNet-B7** with a modified classifier layer.\n- **Training and Evaluation**: Implementing **mixed-precision training (AMP)** and **OneCycleLR** for optimized learning rate scheduling.\n- **Checkpointing and Resumption**: Saving the best model weights and allowing for resuming training.\n\n### Data Preprocessing\n1. **Audio Loading & Resampling**: Audio files are loaded and resampled to a fixed sample rate of **32 kHz**.\n2. **Spectrogram Generation**: Mel spectrograms are computed using **torchaudio**.\n3. **SpecAugment**: Random **time and frequency masking** is applied to improve generalization.\n4. **Resizing**: Spectrograms are resized to **224x224 pixels** and converted to **3-channel images**.\n\n### Model Architecture\n- Uses **EfficientNet-B7** pretrained on ImageNet.\n- The final classifier layer is replaced with a **fully connected layer** matching the number of bird species.\n- **Cross-entropy loss with label smoothing** is applied for better regularization.\n\n### Training Pipeline\n- Uses **AdamW optimizer** with **OneCycleLR** learning rate scheduling.\n- **Automatic Mixed Precision (AMP)** for efficient training.\n- **Checkpoints**:\n  - Loads pretrained weights on first run.\n  - Resumes training from the best saved checkpoint if available.\n  - Saves the best-performing model (`best_model.pth`).\n\n### Steps to Run the Model\n1. **Install Dependencies**: Ensure that `torch`, `torchaudio`, `torchvision`, and other required libraries are installed.\n2. **Set Configuration**: Update `config` parameters, including `data_path`, `learning_rate`, and `epochs`.\n3. **Run Training**: Execute the script to preprocess audio, train the model, and evaluate performance.\n\n### Outputs\n- **Best Model Weights**: Saved as `best_model.pth`.\n- **Training Metrics**: Loss and accuracy stored in `training_metrics.pkl`.\n- **Evaluation Results**: Displays test accuracy on unseen bird audio samples.\n\n### Key Features\n✅ **Transfer Learning**: EfficientNet-B7 for high-performance classification.  \n✅ **Data Augmentation**: SpecAugment to improve robustness.  \n✅ **AMP for Speedup**: Faster and memory-efficient training.  \n✅ **Checkpointing**: Saves and resumes best model performance.  \n✅ **OneCycleLR**: Dynamic learning rate scheduling for improved convergence.  \n","metadata":{}},{"cell_type":"markdown","source":"# Research Papers & Resources  \n\nBelow are the key research papers and resources that inspired and contributed to this project:  \n\n🔹 **[PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://arxiv.org/abs/1912.10211)**  \nA comprehensive study on large-scale pretrained audio neural networks, demonstrating their effectiveness in **audio pattern recognition** tasks.  \n\n🔹 **[SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://arxiv.org/abs/1904.08779)**  \nIntroduces **SpecAugment**, a powerful data augmentation technique that enhances **robustness in speech recognition models** by applying time and frequency masking.  \n","metadata":{}},{"cell_type":"code","source":"# Load necessary library\nimport os\nimport glob\nimport random\nimport pickle\nimport numpy as np\nimport pandas as pd\nfrom sklearn.model_selection import train_test_split\n\nimport torch\nimport torch.nn as nn\nimport torch.optim as optim\nfrom torch.utils.data import Dataset, DataLoader\n\nimport torchaudio\nfrom torchvision.models import efficientnet_b7, EfficientNet_B7_Weights\nimport torch.nn.functional as F\n\nfrom tqdm import tqdm\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-19T06:21:13.531788Z","iopub.execute_input":"2025-02-19T06:21:13.53202Z","iopub.status.idle":"2025-02-19T06:21:21.928519Z","shell.execute_reply.started":"2025-02-19T06:21:13.531977Z","shell.execute_reply":"2025-02-19T06:21:21.927853Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# -----------------------------\n# 1. Reproducibility Utilities\n# -----------------------------\ndef set_seed(seed):\n    \"\"\"Set seed for reproducibility.\"\"\"\n    random.seed(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    if torch.cuda.is_available():\n        torch.cuda.manual_seed_all(seed)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-19T06:21:21.929282Z","iopub.execute_input":"2025-02-19T06:21:21.929713Z","iopub.status.idle":"2025-02-19T06:21:21.934275Z","shell.execute_reply.started":"2025-02-19T06:21:21.929688Z","shell.execute_reply":"2025-02-19T06:21:21.933244Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Configuration\n\nThe `config` dictionary stores key hyperparameters and settings for the project. Each parameter plays a crucial role in controlling how the model is trained, tested, and executed. Below is a description of each configuration parameter:\n\n| Parameter                   | Description                                                                                 | Example Value |\n|----------------------------|-----------------------------------------------------------------------------------------------|---------------|\n| `seed`                     | Random seed to ensure reproducibility. Helps produce consistent results across multiple runs. | `42`          |\n| `data_path`                | Path to the dataset. In this case, it's pointing to bioacoustics data in `.mp3` format.      | `/kaggle/input/bioacoustics-data/osa bird recordings/**/*.mp3` |\n| `test_size`                | Fraction of the dataset used for testing.                                                    | `0.1` (10%)   |\n| `val_size`                 | Fraction of the dataset used for validation.                                                 | `0.1` (10%)   |\n| `sample_rate`              | Sampling rate for audio files in Hertz (Hz).                                                 | `32000`       |\n| `duration`                 | Duration of the audio to use per file (in seconds).                                           | `60`          |\n| `num_workers`              | Number of worker threads for data loading operations. Improves data loading performance.     | `4`           |\n| `num_epochs`               | Number of training epochs (how many complete passes through the dataset).                    | `1`           |\n| `learning_rate`            | The base learning rate for the optimizer. Controls how much the model adjusts in each step.  | `1e-3`        |\n| `weight_decay`             | Regularization parameter to avoid overfitting. Applies L2 penalty on model weights.          | `1e-4`        |\n| `max_lr`                   | Maximum learning rate.                                                                        | `1e-3`        |\n| `device`                   | Specifies whether the code should use a GPU (`cuda`) or CPU for computations.                | `\"cuda:0\"` or `\"cpu\"` |\n| `initial_checkpoint_path`  | Path to the model checkpoint file to initialize training.                                     | `/kaggle/working/best_model.pth` |\n| `save_checkpoint_dir`   | Path to a saved model checkpoint to resume training from.                         | `/kaggle/working/best_model.pth` |\n","metadata":{}},{"cell_type":"code","source":"# -----------------------------\n# 2. Configuration\n# -----------------------------\nconfig = {\n    \"seed\": 42,\n\n    # Load multiple folder directories\n    \"data_path\": [\n        \"/kaggle/input/birdsong-recognition/train_audio/**/*.mp3\",\n    ],\n    \n    \"test_size\": 0.1,\n    \"val_size\": 0.1,\n    \"sample_rate\": 32000,\n    \"duration\": 60,          # seconds of audio to use per file\n    \"num_workers\": 4,\n    \"num_epochs\": 20,\n    \"learning_rate\": 1e-3,\n    \"weight_decay\": 1e-4,\n    \"max_lr\": 1e-3,\n    \"device\": \"cuda:0\" if torch.cuda.is_available() else \"cpu\",\n    \"initial_checkpoint_path\": \"/kaggle/input/bioacoustics-model-weight/best_model.pth\",\n    \"save_checkpoint_dir\": \"/kaggle/working/best_model.pth\" \n}\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-19T06:21:21.935194Z","iopub.execute_input":"2025-02-19T06:21:21.93549Z","iopub.status.idle":"2025-02-19T06:21:22.007365Z","shell.execute_reply.started":"2025-02-19T06:21:21.93546Z","shell.execute_reply":"2025-02-19T06:21:22.006455Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Data Collection and Splitting\n\nThis section contains functions to **load audio files** from one or more directories and **split the dataset** into training, validation, and test sets.\n\n### `load_audio_files(path_patterns)`\nThis function scans one or more provided file path patterns, extracts file paths, and assigns labels based on their parent directory. It accepts either a single glob pattern (as a string) or a list of glob patterns.\n\n**Parameters:**\n- `path_patterns` *(str or list of str)*: A file path pattern (e.g., `\"/path/to/audio/**/*.mp3\"`) or a list of such patterns to search for audio files recursively.\n\n**Returns:**\n- `df` *(pandas.DataFrame)*: A DataFrame containing:\n  - `'filepath'`: Full path of the audio file.\n  - `'label'`: The category or label of the audio file, inferred from its parent directory.\n\n### `split_data(df, test_size, val_size, random_state=42)`\nThis function splits the dataset into **training, validation, and test sets**, ensuring that the splits are stratified by label.\n\n**Parameters:**\n- `df` *(pandas.DataFrame)*: The dataset containing `'filepath'` and `'label'`.\n- `test_size` *(float)*: Proportion of the dataset to allocate for testing.\n- `val_size` *(float)*: Proportion of the remaining dataset to allocate for validation.\n- `random_state` *(int, default=42)*: Seed value for reproducibility.\n\n**Returns:**\n- `train` *(pandas.DataFrame)*: Training set.\n- `val` *(pandas.DataFrame)*: Validation set.\n- `test` *(pandas.DataFrame)*: Test set.\n\n**This ensures that:**\n- The test set is `test_size` fraction of the full dataset.\n- The validation set is `val_size` fraction of the remaining data after the test split.\n- Data is stratified, meaning the label distribution remains consistent across all splits.\n","metadata":{}},{"cell_type":"code","source":"# -----------------------------\n# 3. Data Collection and Splitting\n# -----------------------------\ndef load_audio_files(path_patterns):\n    \"\"\"\n    Returns a DataFrame with columns: 'filepath' and 'label'.\n    Assumes that the parent directory of each file is its label.\n    Accepts a single glob pattern (str) or a list of glob patterns.\n    \"\"\"\n    if isinstance(path_patterns, str):\n        path_patterns = [path_patterns]\n    \n    data = []\n    for pattern in path_patterns:\n        file_paths = glob.glob(pattern, recursive=True)\n        for fp in file_paths:\n            label = os.path.basename(os.path.dirname(fp))\n            data.append({'filepath': fp, 'label': label})\n    return pd.DataFrame(data)\n\ndef split_data(df, test_size, val_size, random_state=42):\n    \"\"\"\n    Split the dataframe into train, validation, and test sets.\n    Stratify based on the label.\n    \"\"\"\n    train_val, test = train_test_split(df, test_size=test_size, \n                                       stratify=df['label'], random_state=random_state)\n    train, val = train_test_split(train_val, test_size=val_size, \n                                  stratify=train_val['label'], random_state=random_state)\n    return train, val, test\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-19T06:21:22.009813Z","iopub.execute_input":"2025-02-19T06:21:22.010094Z","iopub.status.idle":"2025-02-19T06:21:22.022338Z","shell.execute_reply.started":"2025-02-19T06:21:22.010071Z","shell.execute_reply":"2025-02-19T06:21:22.021562Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Dataset Definition with Audio Augmentation\n\nThis section defines the **`BirdAudioDataset`** class, a custom PyTorch dataset designed to process bird audio recordings, apply transformations, and generate spectrogram images suitable for deep learning models.\n\n### `BirdAudioDataset`\nThis class loads audio files, converts them into **Mel spectrograms**, applies **data augmentation** (if enabled), and normalizes the data.\n\n### **Initialization (`__init__` method)**\nThe dataset is initialized with key parameters:\n\n**Parameters:**\n- `df` *(pandas.DataFrame)*: A dataset containing columns `'filepath'` (audio file path) and `'label_id'` (numeric label).\n- `sample_rate` *(int)*: The target sampling rate (Hz) to which all audio files will be resampled.\n- `duration` *(int)*: The number of seconds of audio to use per file.\n- `augment` *(bool, default=False)*: Whether to apply **SpecAugment**-style data augmentation.\n\n**Preprocessing Steps:**\n- The number of samples per file is calculated as `sample_rate * duration`.\n- **Mel Spectrogram Transformation**:\n  - Converts audio into a **Mel Spectrogram** with 128 Mel frequency bins.\n  - Converts amplitude values to **decibels (dB)** using `AmplitudeToDB()`.\n- **SpecAugment (if enabled)**:\n  - `FrequencyMasking`: Masks random frequency bands to improve generalization.\n  - `TimeMasking`: Masks random time intervals to simulate real-world distortions.\n\n### **Dataset Length (`__len__` method)**\nReturns the number of samples in the dataset:\n```python\ndef __len__(self):\n    return len(self.df)\n```\n\n### **Loading & Processing Audio (`__getitem__` method)**\nThis method:\n1. **Loads the audio file** using `torchaudio.load(filepath)`.\n2. **Converts stereo to mono** (if applicable).\n3. **Resamples** to the target `sample_rate` (if needed).\n4. **Trims/Pads** the waveform to the required duration.\n5. **Generates a Mel spectrogram** and converts it to dB scale.\n6. **Applies SpecAugment transformations** (if enabled).\n7. **Normalizes and resizes** the spectrogram to **224×224** pixels.\n8. **Converts the spectrogram into a 3-channel image** for compatibility with deep learning models.\n9. **Returns** the processed **spectrogram image** and the **label**.\n\n**Returns:**\n- `image` *(Tensor, shape `[3, 224, 224]`)*: A spectrogram image with 3 color channels.\n- `label` *(int)*: The corresponding label for the audio file.\n","metadata":{}},{"cell_type":"code","source":"# -----------------------------\n# 4. Dataset Definition with Audio Augmentation\n# -----------------------------\nclass BirdAudioDataset(Dataset):\n    def __init__(self, df, sample_rate, duration, augment=False):\n        \"\"\"\n        df: DataFrame with columns 'filepath' and 'label_id'\n        sample_rate: target sample rate (Hz)\n        duration: duration (in seconds) to use from each audio file\n        augment: whether to apply SpecAugment style augmentation\n        \"\"\"\n        self.df = df.reset_index(drop=True)\n        self.sample_rate = sample_rate\n        self.duration = duration\n        self.num_samples = sample_rate * duration\n        self.augment = augment\n        \n        # Create MelSpectrogram transform.\n        self.mel_transform = torchaudio.transforms.MelSpectrogram(\n            sample_rate=sample_rate, n_fft=1024, hop_length=512, n_mels=128\n        )\n        self.amplitude_to_db = torchaudio.transforms.AmplitudeToDB()\n        \n        # SpecAugment transforms (if augment=True)\n        if self.augment:\n            self.freq_mask = torchaudio.transforms.FrequencyMasking(freq_mask_param=15)\n            self.time_mask = torchaudio.transforms.TimeMasking(time_mask_param=30)\n        \n    def __len__(self):\n        return len(self.df)\n    \n    def __getitem__(self, idx):\n        row = self.df.iloc[idx]\n        filepath = row['filepath']\n        \n        # Load and process audio.\n        waveform, sr = torchaudio.load(filepath)\n        if waveform.shape[0] > 1:\n            waveform = waveform.mean(dim=0, keepdim=True)\n        if sr != self.sample_rate:\n            resampler = torchaudio.transforms.Resample(sr, self.sample_rate)\n            waveform = resampler(waveform)\n        if waveform.shape[1] < self.num_samples:\n            padding = self.num_samples - waveform.shape[1]\n            waveform = F.pad(waveform, (0, padding))\n        else:\n            waveform = waveform[:, :self.num_samples]\n        \n        mel_spec = self.mel_transform(waveform)\n        mel_spec_db = self.amplitude_to_db(mel_spec)\n        \n        if self.augment:\n            mel_spec_db = self.freq_mask(mel_spec_db)\n            mel_spec_db = self.time_mask(mel_spec_db)\n            \n        # Normalize and resize.\n        mel_spec_db = (mel_spec_db - mel_spec_db.mean()) / (mel_spec_db.std() + 1e-9)\n        mel_spec_db = F.interpolate(mel_spec_db.unsqueeze(0), size=(224, 224),\n                                    mode='bilinear', align_corners=False).squeeze(0)\n        image = mel_spec_db.repeat(3, 1, 1)  # Convert to 3 channels\n        \n        label = row['label_id']\n        return image, label\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-19T06:21:22.023146Z","iopub.execute_input":"2025-02-19T06:21:22.02333Z","iopub.status.idle":"2025-02-19T06:21:22.043194Z","shell.execute_reply.started":"2025-02-19T06:21:22.023314Z","shell.execute_reply":"2025-02-19T06:21:22.042591Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Model Definition (EfficientNet-B7 with Fine-Tuning)\n\nThis section defines the **`BirdClassifier`** model, which fine-tunes **EfficientNet-B7** for bird sound classification. EfficientNet-B7 is a **state-of-the-art** convolutional neural network (CNN) known for its high performance on image classification tasks.\n\n### **`BirdClassifier` Class**\nThis class initializes a pre-trained **EfficientNet-B7** model and modifies the classifier layer to match the number of output classes.\n\n### **Initialization (`__init__` method)**\n- Loads a pre-trained **EfficientNet-B7** model using `efficientnet_b7()`.\n- Retrieves the default **pre-trained weights** (`EfficientNet_B7_Weights.DEFAULT`).\n- Replaces the final **fully connected (FC) layer** with a new `nn.Linear` layer to match the number of output classes.\n\n**Parameters:**\n- `num_classes` *(int)*: The number of classes in the dataset.\n\n**Modifications:**\n- Extracts the number of input features from the **original classifier**.\n- Replaces the final FC layer with a new linear layer of shape **(in_features, num_classes)**.\n\n### **Forward Pass (`forward` method)**\nDefines the forward propagation of input **image tensors** through the model.\n\n- **Input**: A batch of images (`x`) with shape `[batch_size, 3, 224, 224]`.\n- **Output**: A tensor of shape `[batch_size, num_classes]`, containing class logits (before applying softmax).\n\n```python\ndef forward(self, x):\n    return self.model(x)\n```\n\n### **Key Features of the Model**\n- **Uses EfficientNet-B7**, a **highly efficient** CNN with a strong accuracy-to-performance ratio.\n- **Leverages pre-trained weights**, allowing for **transfer learning**—reducing the need for large datasets.\n- **Replaces the classifier** to accommodate a new classification task (i.e., bird sound spectrograms).\n- **Outputs logits** for classification, which can be converted to probabilities using `torch.nn.functional.softmax`.\n","metadata":{}},{"cell_type":"code","source":"# -----------------------------\n# 5. Model Definition (EfficientNet-B7 with Fine-Tuning)\n# -----------------------------\nclass BirdClassifier(nn.Module):\n    def __init__(self, num_classes):\n        super(BirdClassifier, self).__init__()\n        # Use EfficientNet-B7 and its default weights.\n        weights = EfficientNet_B7_Weights.DEFAULT\n        self.model = efficientnet_b7(weights=weights)\n        in_features = self.model.classifier[1].in_features\n        self.model.classifier[1] = nn.Linear(in_features, num_classes)\n        \n    def forward(self, x):\n        return self.model(x)\n        ","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-19T06:21:22.043969Z","iopub.execute_input":"2025-02-19T06:21:22.044279Z","iopub.status.idle":"2025-02-19T06:21:22.060361Z","shell.execute_reply.started":"2025-02-19T06:21:22.044245Z","shell.execute_reply":"2025-02-19T06:21:22.059677Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Training and Evaluation (with AMP and Checkpointing)\n\nThis section defines functions for **training, evaluating, and testing** the model using **Automatic Mixed Precision (AMP)** for efficient computation and **checkpointing** to save the best model.\n\n### **`train_model` Function**\nThis function trains the **EfficientNet-B7** model using **AdamW optimizer, OneCycle learning rate scheduling, and label smoothing** for better generalization.\n\n### **Parameters:**\n- `model` *(nn.Module)*: The PyTorch model to train.\n- `train_loader` *(DataLoader)*: DataLoader for the training dataset.\n- `val_loader` *(DataLoader)*: DataLoader for the validation dataset.\n- `device` *(str)*: `\"cuda\"` or `\"cpu\"`, based on availability.\n- `num_epochs` *(int)*: Total number of training epochs.\n- `max_lr` *(float)*: Maximum learning rate for **OneCycleLR** scheduler.\n\n### **Training Process:**\n1. **Loss & Optimization Setup**\n   - Uses **CrossEntropyLoss** with `label_smoothing=0.1` to improve generalization.\n   - Uses **AdamW optimizer** for training.\n   - Uses **OneCycleLR scheduler** for dynamic learning rate adjustments.\n   - Uses **AMP GradScaler** for mixed precision training (reducing memory and increasing speed).\n\n2. **Training Loop**\n   - Iterates through the dataset, computes loss, and updates model parameters.\n   - Uses **autocast** for mixed precision computations.\n   - Tracks **training loss and accuracy**.\n\n3. **Validation Loop**\n   - Evaluates the model on the validation set without updating weights.\n   - Tracks **validation loss and accuracy**.\n\n4. **Model Checkpointing**\n   - Saves the model if the validation accuracy improves.\n\n5. **Returns:**\n   - The **trained model**.\n   - A dictionary containing **training and validation metrics** (loss and accuracy).\n\n### **`test_model` Function**\nThis function evaluates the trained model on a **test dataset**.\n\n### **Parameters:**\n- `model` *(nn.Module)*: Trained model.\n- `test_loader` *(DataLoader)*: DataLoader for the test dataset.\n- `device` *(str)*: `\"cuda\"` or `\"cpu\"`, based on availability.\n\n### **Testing Process:**\n1. Sets the model to **evaluation mode** (`model.eval()`).\n2. Iterates through the test dataset, making predictions.\n3. Computes the **test accuracy**.\n\n### **Returns:**\n- Prints the **test accuracy**.\n\n### **Key Features:**\n- **Automatic Mixed Precision (AMP)**: Uses `torch.amp.autocast()` for faster training with reduced memory consumption.\n- **OneCycle Learning Rate Scheduler**: Adjusts learning rate dynamically for stable convergence.\n- **Label Smoothing (0.1)**: Reduces overconfidence in predictions and improves generalization.\n- **Checkpointing**: Saves the best model when validation accuracy improves.\n- **Progress Tracking**: Uses `tqdm` for real-time monitoring of training progress.\n","metadata":{}},{"cell_type":"code","source":"# -----------------------------\n# 6. Training and Evaluation (with AMP and Checkpointing)\n# -----------------------------\ndef train_model(model, train_loader, val_loader, device, num_epochs, max_lr):\n    model.to(device)\n    \n    criterion = nn.CrossEntropyLoss(label_smoothing=0.1)\n    optimizer = optim.AdamW(model.parameters(), lr=max_lr, weight_decay=config[\"weight_decay\"])\n    total_steps = len(train_loader) * num_epochs\n    scheduler = optim.lr_scheduler.OneCycleLR(optimizer, max_lr=max_lr, \n                                              total_steps=total_steps,\n                                              pct_start=0.1, anneal_strategy='cos',\n                                              div_factor=25.0, final_div_factor=1e4)\n    \n    # Initialize AMP GradScaler using the updated API.\n    scaler = torch.amp.GradScaler()\n    \n    best_val_acc = 0.0\n    train_losses, train_accuracies = [], []\n    val_losses, val_accuracies = [], []\n    \n    for epoch in range(num_epochs):\n        model.train()\n        running_loss, running_corrects, total = 0.0, 0, 0\n        \n        train_pbar = tqdm(train_loader, desc=f\"Epoch {epoch+1}/{num_epochs} Training\", leave=False)\n        for inputs, labels in train_pbar:\n            inputs, labels = inputs.to(device), labels.to(device)\n            optimizer.zero_grad()\n            \n            # Use the new autocast API with explicit device type.\n            with torch.amp.autocast(device_type=\"cuda\"):\n                outputs = model(inputs)\n                loss = criterion(outputs, labels)\n            \n            scaler.scale(loss).backward()\n            scaler.step(optimizer)\n            scheduler.step()\n            scaler.update()\n\n            \n            running_loss += loss.item() * inputs.size(0)\n            _, preds = torch.max(outputs, 1)\n            running_corrects += (preds == labels).sum().item()\n            total += labels.size(0)\n            train_pbar.set_postfix(loss=loss.item())\n        \n        epoch_loss = running_loss / total\n        epoch_acc = running_corrects / total\n        train_losses.append(epoch_loss)\n        train_accuracies.append(epoch_acc)\n        \n        # Validation phase.\n        model.eval()\n        val_running_loss, val_running_corrects, val_total = 0.0, 0, 0\n        val_pbar = tqdm(val_loader, desc=f\"Epoch {epoch+1}/{num_epochs} Validation\", leave=False)\n        with torch.no_grad():\n            for inputs, labels in val_pbar:\n                inputs, labels = inputs.to(device), labels.to(device)\n                with torch.amp.autocast(device_type=\"cuda\"):\n                    outputs = model(inputs)\n                    loss = criterion(outputs, labels)\n                val_running_loss += loss.item() * inputs.size(0)\n                _, preds = torch.max(outputs, 1)\n                val_running_corrects += (preds == labels).sum().item()\n                val_total += labels.size(0)\n                val_pbar.set_postfix(loss=loss.item())\n        \n        epoch_val_loss = val_running_loss / val_total\n        epoch_val_acc = val_running_corrects / val_total\n        val_losses.append(epoch_val_loss)\n        val_accuracies.append(epoch_val_acc)\n        \n        print(f\"Epoch [{epoch+1}/{num_epochs}] | Train Loss: {epoch_loss:.4f}, Train Acc: {epoch_acc:.4f} | \"\n              f\"Val Loss: {epoch_val_loss:.4f}, Val Acc: {epoch_val_acc:.4f}\")\n        \n        # Checkpointing: Save model if validation accuracy improves.\n        if epoch_val_acc > best_val_acc:\n            best_val_acc = epoch_val_acc\n            torch.save(model.state_dict(), config[\"save_checkpoint_dir\"])\n            print(f\"Best model updated (Val Acc: {best_val_acc:.4f}). Checkpoint saved.\")\n    \n    metrics = {\n        'train_loss': train_losses,\n        'train_acc': train_accuracies,\n        'val_loss': val_losses,\n        'val_acc': val_accuracies\n    }\n    return model, metrics\n\ndef test_model(model, test_loader, device):\n    model.eval()\n    correct, total = 0, 0\n    for inputs, labels in tqdm(test_loader, desc=\"Testing\"):\n        inputs, labels = inputs.to(device), labels.to(device)\n        with torch.amp.autocast(device_type=\"cuda\"):\n            outputs = model(inputs)\n        _, preds = torch.max(outputs, 1)\n        correct += (preds == labels).sum().item()\n        total += labels.size(0)\n    acc = correct / total\n    print(f\"Test Accuracy: {acc:.4f}\")\n    ","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-19T06:21:22.061248Z","iopub.execute_input":"2025-02-19T06:21:22.061448Z","iopub.status.idle":"2025-02-19T06:21:22.082034Z","shell.execute_reply.started":"2025-02-19T06:21:22.061424Z","shell.execute_reply":"2025-02-19T06:21:22.081254Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Main Function: Data Loading, Training, and Saving Metrics\n\nThis section defines the **`main()`** function, which handles the **entire pipeline** from data loading, dataset preparation, model training, evaluation, and saving metrics.\n\n### **Overview of `main()`**\n1. **Sets the random seed** for reproducibility.\n2. **Loads and processes the dataset**, assigning unique labels to each bird species.\n3. **Splits the data** into training, validation, and test sets.\n4. **Creates DataLoaders** for efficient batch processing.\n5. **Initializes the model** with EfficientNet-B7.\n6. **Loads model checkpoints** (if available) to resume training.\n7. **Trains the model** and evaluates it on the validation dataset.\n8. **Tests the trained model** on the test dataset.\n9. **Saves training metrics** to a `.pkl` file.\n\n### **Step-by-Step Breakdown**\n\n### **1. Set Random Seed for Reproducibility**\n```python\nset_seed(config[\"seed\"])\n```\n- Ensures consistent results across multiple runs.\n\n### **2. Load and Process the Data**\n```python\ndf = load_audio_files(config[\"data_path\"])\nlabels_sorted = sorted(df['label'].unique())\nlabel2id = {label: i for i, label in enumerate(labels_sorted)}\ndf['label_id'] = df['label'].map(label2id)\nprint(f\"Found {len(df)} audio files across {len(labels_sorted)} classes.\")\n```\n- Loads audio file paths and assigns labels based on directory names.\n- Maps each unique label to a numeric **label ID**.\n\n### **3. Split Data into Train, Validation, and Test Sets**\n```python\ntrain_df, val_df, test_df = split_data(df, config[\"test_size\"], config[\"val_size\"], random_state=config[\"seed\"])\nprint(f\"Train: {len(train_df)}, Val: {len(val_df)}, Test: {len(test_df)}\")\n```\n- Uses **stratified splitting** to maintain class balance.\n\n### **4. Create Datasets and DataLoaders**\n```python\ntrain_dataset = BirdAudioDataset(train_df, config[\"sample_rate\"], config[\"duration\"], augment=True)\nval_dataset = BirdAudioDataset(val_df, config[\"sample_rate\"], config[\"duration\"], augment=False)\ntest_dataset = BirdAudioDataset(test_df, config[\"sample_rate\"], config[\"duration\"], augment=False)\n\ntrain_loader = DataLoader(train_dataset, batch_size=32, shuffle=True, num_workers=config[\"num_workers\"])\nval_loader = DataLoader(val_dataset, batch_size=32, shuffle=False, num_workers=config[\"num_workers\"])\ntest_loader = DataLoader(test_dataset, batch_size=8, shuffle=False, num_workers=config[\"num_workers\"])\n```\n- **Augments training data** using SpecAugment.\n- **Creates DataLoaders** for efficient mini-batch processing.\n\n### **5. Initialize Model**\n```python\nnum_classes = len(labels_sorted)\nmodel = BirdClassifier(num_classes)\ndevice = torch.device(config[\"device\"])\n```\n- Creates an **EfficientNet-B7** model.\n- Moves model to **CPU or GPU** based on availability.\n\n### **6. Load Model Checkpoints (If Available)**\n```python\nif os.path.exists(config[\"initial_checkpoint_path\"]):\n    print(f\"Existing resume checkpoint found at {config['initial_checkpoint_path']}. Loading model weights to resume training.\")\n    model.load_state_dict(torch.load(config[\"initial_checkpoint_path\"], map_location=device, weights_only=True))\nelse:\n    print(\"No checkpoint found. Starting training from scratch.\")\n```\n- **Resumes training** from the latest checkpoint if available.\n- **Loads pre-trained weights** if no resume checkpoint exists.\n- **Starts fresh training** if no checkpoints are found.\n\n### **7. Clear GPU Cache**\n```python\ntorch.cuda.empty_cache()\n```\n- Frees unused GPU memory before training starts.\n\n\n### **8. Train the Model**\n```python\nmodel, metrics = train_model(model, train_loader, val_loader, device,\n                             num_epochs=config[\"num_epochs\"], max_lr=config[\"max_lr\"])\n```\n- Trains the model using **OneCycleLR, label smoothing, and AMP**.\n- Returns **training metrics** (loss & accuracy).\n\n### **9. Test the Model**\n```python\ntest_model(model, test_loader, device)\n```\n- Evaluates the model on the **test dataset**.\n\n### **10. Save Training Metrics**\n```python\nwith open(\"/kaggle/working/training_metrics.pkl\", \"wb\") as f:\n    pickle.dump(metrics, f)\nprint(\"Training metrics saved to training_metrics.pkl\")\n```\n- Saves training and validation **loss & accuracy** for further analysis.\n\n### **Key Features:**\n✅ **End-to-End Workflow** – Handles data preparation, training, evaluation, and saving results.  \n✅ **Model Checkpointing** – Prevents loss of progress by saving the best model.  \n✅ **GPU Optimization** – Uses **AMP and CUDA** for faster and efficient training.  \n✅ **Reproducibility** – Ensures consistent results using a **fixed random seed**.  \n✅ **Efficient Data Processing** – Uses **multi-threaded DataLoaders** for faster data loading.  \n","metadata":{}},{"cell_type":"code","source":"# -----------------------------\n# 7. Main Function: Data Loading, Training, and Saving Metrics\n# -----------------------------\ndef main():\n    # Set seed for reproducibility.\n    set_seed(config[\"seed\"])\n    \n    # Load and prepare the data.\n    df = load_audio_files(config[\"data_path\"])\n    labels_sorted = sorted(df['label'].unique())\n    label2id = {label: i for i, label in enumerate(labels_sorted)}\n    df['label_id'] = df['label'].map(label2id)\n    print(f\"Found {len(df)} audio files across {len(labels_sorted)} classes.\")\n    \n    train_df, val_df, test_df = split_data(df, config[\"test_size\"], config[\"val_size\"], random_state=config[\"seed\"])\n    print(f\"Train: {len(train_df)}, Val: {len(val_df)}, Test: {len(test_df)}\")\n    \n    train_dataset = BirdAudioDataset(train_df, config[\"sample_rate\"], config[\"duration\"], augment=True)\n    val_dataset = BirdAudioDataset(val_df, config[\"sample_rate\"], config[\"duration\"], augment=False)\n    test_dataset = BirdAudioDataset(test_df, config[\"sample_rate\"], config[\"duration\"], augment=False)\n    \n    train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True, num_workers=config[\"num_workers\"])\n    val_loader = DataLoader(val_dataset, batch_size=32, shuffle=False, num_workers=config[\"num_workers\"])\n    test_loader = DataLoader(test_dataset, batch_size=8, shuffle=False, num_workers=config[\"num_workers\"])\n    \n    num_classes = len(labels_sorted)\n    model = BirdClassifier(num_classes)\n    \n    device = torch.device(config[\"device\"])\n    \n    # ---------------\n    # Check for and load checkpoints:\n    # ---------------\n    if os.path.exists(config[\"initial_checkpoint_path\"]):\n        print(f\"Loading initial weights from {config['initial_checkpoint_path']} for the first run.\")\n        checkpoint = torch.load(config[\"initial_checkpoint_path\"], map_location=device, weights_only=True)\n        old_weight_count = checkpoint[\"model.classifier.1.weight\"].shape[0]\n        num_new_classes = num_classes\n        if num_new_classes > 0:\n            print(f\"Expanding classifier layer: adding {num_new_classes} new classes.\")\n            current_fc_weight = model.model.classifier[1].weight.data\n            current_fc_bias = model.model.classifier[1].bias.data\n            # Copy weights and biases for the old classes from the checkpoint.\n            current_fc_weight[:old_weight_count, :] = checkpoint[\"model.classifier.1.weight\"]\n            current_fc_bias[:old_weight_count] = checkpoint[\"model.classifier.1.bias\"]\n            # Update the checkpoint with the modified classifier weights.\n            checkpoint[\"model.classifier.1.weight\"] = current_fc_weight\n            checkpoint[\"model.classifier.1.bias\"] = current_fc_bias\n        model.load_state_dict(checkpoint, strict=False)\n    else:\n        print(\"No initial checkpoint found. Starting training from scratch.\")\n        torch.cuda.empty_cache()\n\n    # Train the model.\n    model, metrics = train_model(model, train_loader, val_loader, device,\n                                 num_epochs=config[\"num_epochs\"], max_lr=config[\"max_lr\"])\n    \n    # Test the model.\n    test_model(model, test_loader, device)\n    \n    # Save training metrics.\n    with open(\"/kaggle/working/training_metrics.pkl\", \"wb\") as f:\n        pickle.dump(metrics, f)\n    print(\"Training metrics saved to training_metrics.pkl\")\n    ","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-19T06:21:22.082881Z","iopub.execute_input":"2025-02-19T06:21:22.083165Z","iopub.status.idle":"2025-02-19T06:21:22.098427Z","shell.execute_reply.started":"2025-02-19T06:21:22.083144Z","shell.execute_reply":"2025-02-19T06:21:22.09786Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Runs the full pipeline from data loading to model evaluation\nif __name__ == '__main__':\n    main()\n    ","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-19T06:21:22.099079Z","iopub.execute_input":"2025-02-19T06:21:22.09926Z","iopub.status.idle":"2025-02-19T15:26:02.774037Z","shell.execute_reply.started":"2025-02-19T06:21:22.099243Z","shell.execute_reply":"2025-02-19T15:26:02.772834Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Open the pickle file in read-binary mode\nwith open(\"/kaggle/working/training_metrics.pkl\", \"rb\") as f:\n    data = pickle.load(f)\n\n# Print or inspect the data  \nprint(data)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-19T15:26:02.778198Z","iopub.execute_input":"2025-02-19T15:26:02.77847Z","iopub.status.idle":"2025-02-19T15:26:02.784662Z","shell.execute_reply.started":"2025-02-19T15:26:02.778446Z","shell.execute_reply":"2025-02-19T15:26:02.783768Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\nplt.plot(range(1,21),data['train_loss'])\nplt.plot(range(1,21),data['val_loss'])\nplt.xlabel(\"Epoch\")\nplt.ylabel(\"Loss\")\nplt.title(\"Loss function\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-19T15:38:04.691751Z","iopub.execute_input":"2025-02-19T15:38:04.692143Z","iopub.status.idle":"2025-02-19T15:38:04.878189Z","shell.execute_reply.started":"2025-02-19T15:38:04.692107Z","shell.execute_reply":"2025-02-19T15:38:04.877344Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plt.plot(range(1,21),data['train_acc'])\nplt.plot(range(1,21),data['val_acc'])\nplt.xlabel(\"Epoch\")\nplt.ylabel(\"Accuracy\")\nplt.title(\"Accuracy function\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-19T15:43:54.203068Z","iopub.execute_input":"2025-02-19T15:43:54.203481Z","iopub.status.idle":"2025-02-19T15:43:54.410479Z","shell.execute_reply.started":"2025-02-19T15:43:54.203456Z","shell.execute_reply":"2025-02-19T15:43:54.40957Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"torch.save(model, \"full_model.pth\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-19T15:45:52.175549Z","iopub.execute_input":"2025-02-19T15:45:52.175949Z","iopub.status.idle":"2025-02-19T15:45:52.195525Z","shell.execute_reply.started":"2025-02-19T15:45:52.17592Z","shell.execute_reply":"2025-02-19T15:45:52.194379Z"}},"outputs":[],"execution_count":null}]}