{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.11.11"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceType":"competition","sourceId":91844,"databundleVersionId":11361821},{"sourceType":"datasetVersion","sourceId":12010270,"datasetId":7528517,"databundleVersionId":12528417},{"sourceType":"datasetVersion","sourceId":12051777,"datasetId":7505901,"databundleVersionId":12575373}],"dockerImageVersionId":31041,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true},"papermill":{"default_parameters":{},"duration":66.275304,"end_time":"2025-05-20T09:23:45.886660","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2025-05-20T09:22:39.611356","version":"2.6.0"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 🐦 BirdCLEF 2025 - Audio Classification & Sound Event Detection\n\n## 📘 Introduction\n\nThis notebook is part of the **BirdCLEF 2025** challenge, which focuses on the **identification of bird species from audio recordings** in natural environments. It is a part of the **LifeCLEF** series organized by the **CLEF (Conference and Labs of the Evaluation Forum)**, and provides a real-world scenario for bioacoustic research and machine learning in the wild.\n\nThe main objective of this competition is to build models that can detect the presence of bird species in field recordings — potentially overlapping, noisy, and highly imbalanced — using audio classification and sound event detection (SED) techniques.\n\n---\n\n## 📂 Dataset Overview\n\nThe dataset consists of:\n\n- `train_audio/` – Raw training audio files (typically 1-minute recordings).\n- `train_metadata.csv` – Contains metadata for each audio file such as species label, rating, and location.\n- `train_soundscapes/` and `test_soundscapes/` – Simulated soundscape audio segments for evaluation.\n- `taxonomy.csv` – A hierarchy of species, genus, and family for all classes in the dataset.\n\n---\n### Data Preparation & Mel Generation\n\nBefore training this model, the dataset was carefully prepared using the following steps:\n\n- **Data Preparation Notebook:**  \n  [BirdCLEF 2025 Data Preparation](https://www.kaggle.com/code/sheemamasood/birdclef-2025-data-prepartion)  \n  - Created 10-second audio chunks from both train and soundscape recordings  \n  - Applied VAD (Voice Activity Detection) filtering to remove unwanted noise and silence  \n  - Built a clean, balanced dataset ready for feature extraction and training\n\n- **Mel Spectrogram Generation Notebook:**  \n  [BirdCLEF2025 Mel Generation](https://www.kaggle.com/code/sheemamasood/birdclef2025-mel-generation)  \n  - Converted audio chunks into mel spectrograms  \n  - Saved mel spectrograms as inputs for model training  \n\nThese preprocessing steps ensure the model receives high-quality, meaningful inputs, improving training effectiveness and overall performance.\n************\n\n## ⚙️ Preprocessing Pipeline\n\nIn this notebook, we focus on:\n\n- Creating a **custom PyTorch Dataset class** to efficiently load and manage training samples.\n- Loading **preprocessed metadata** (filtered by rating, etc.) and **precomputed mel spectrograms** for fast training.\n- Preparing inputs to be used directly in model training or evaluation.\n\nWe have saved the mel spectrograms in `.npz` format for efficient access, reducing the on-the-fly computation cost during model training.\n\n---\n\nLet's begin! 🚀\n","metadata":{}},{"cell_type":"markdown","source":"## 🔗 BirdCLEF 2025 - Project Notebook Links\n\nHere are the different stages of my BirdCLEF 2025 pipeline, organized by functionality:\n\n### 📊 Data Preparation\n- [BirdCLEF 2025 - Data Preparation](https://www.kaggle.com/code/sheemamasood/birdclef-2025-data-prepartion)\n\n### 🎛️ Mel Spectrogram Generation\n- [BirdCLEF 2025 - Mel Generation](https://www.kaggle.com/code/sheemamasood/birdclef2025-mel-generation)\n\n### 🏷️ Pseudo Labelling for SSL\n- [BirdCLEF 2025 - Pseudo Labelling for SSL](https://www.kaggle.com/code/sheemamasood/birdclef2025-psedolabelling-for-ssl)\n\n### 🧠 Model Training\n- [BirdCLEF 2025 - Model Training (Phase 1)](https://www.kaggle.com/code/sheemamasood/birdclef2025-model-training-phase1)\n\n### 📦 Inference & Submissions\n- [BirdCLEF 2025 - Submissions](https://www.kaggle.com/code/sheemamasood/birdclef2025-submissions)\n","metadata":{}},{"cell_type":"code","source":"# Standard Libraries\nimport os\nimport gc\nimport time\nimport math\nimport random\nimport warnings\nimport logging\nfrom pathlib import Path\nfrom glob import glob\nfrom typing import Union\nimport copy\nimport concurrent.futures\n\n# Data Handling\nimport numpy as np\nimport pandas as pd\nimport joblib\nimport pickle\nimport collections\nfrom sklearn.model_selection import train_test_split, StratifiedKFold\nfrom sklearn.metrics import roc_auc_score, classification_report\n\n# Audio Processing\nimport librosa\nimport librosa.display\nimport soundfile as sf\nfrom soundfile import SoundFile\nimport torchaudio\n\n# Machine Learning & PyTorch\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nfrom torch.utils.data import Dataset, DataLoader, WeightedRandomSampler\nfrom torch.cuda.amp import autocast, GradScaler\nimport timm\n\n# Visualization\nimport cv2\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n# Progress Bars\nfrom tqdm import tqdm\nfrom tqdm.notebook import tqdm as notebook_tqdm\n\n# Logging and Warnings\nwarnings.filterwarnings(\"ignore\")\nlogging.basicConfig(level=logging.ERROR)\n\n# Check versions\nprint(f\"librosa version : {librosa.__version__}\")\nprint(f\"librosa files : {librosa.__file__}\")\nprint(\"✅ All libraries imported in the environment.\")\n","metadata":{"execution":{"iopub.status.busy":"2025-06-03T15:46:04.183062Z","iopub.execute_input":"2025-06-03T15:46:04.183371Z","iopub.status.idle":"2025-06-03T15:46:16.980517Z","shell.execute_reply.started":"2025-06-03T15:46:04.183317Z","shell.execute_reply":"2025-06-03T15:46:16.979732Z"},"papermill":{"duration":23.046365,"end_time":"2025-05-20T09:23:07.831968","exception":false,"start_time":"2025-05-20T09:22:44.785603","status":"completed"},"tags":[],"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"\nclass Config:\n    # ===== General Settings =====\n    seed = 42\n    print_freq = 100\n    num_workers = 4\n\n    # ===== Audio Settings =====\n    FS = 32000  # Sampling rate\n    N_FFT = 1024\n    HOP_LENGTH = 512\n    FMIN = 50\n    FMAX = 14000\n    N_MELS = 128\n    TARGET_DURATION = 10.0  # duration in seconds for full input\n    TARGET_DURATION_TRAIN = 10\n    TARGET_DURATION_TEST = 5\n\n    # ===== Image / Mel Spectrogram Settings =====\n    MEL_SHAPE = (256, 256)         # (height, width)\n    TARGET_SHAPE = (3, 256, 256)   # RGB Image Shape (C, H, W)\n\n    \n    # ===== File Paths =====\n    test_soundscapes = \"/kaggle/input/birdclef-2025/test_soundscapes\"\n    submission_csv = \"/kaggle/input/birdclef-2025/sample_submission.csv\"\n    model_path = \"/kaggle/input/birdcleft-clean-and-vad-filtered-data/best_model_187.pth\"\n    backbone_weights = \"/kaggle/input/birdcleft-clean-and-vad-filtered-data/seresnext_backbone_weights.pth\"\n    master_labels = \"/kaggle/input/birdcleft-clean-and-vad-filtered-data/valid_labels.pkl\"\n\n    # ===== Model Settings =====\n    model_name = 'seresnext26t_32x4d'\n    pretrained = False\n    in_channels = 1\n\n    # ===== Derived Attributes (initialized later) =====\n    master_labels = None\n    NUM_CLASSES = None\n\n\n# Instantiate and initialize\nconfig = Config()\n\nprint(f\"✅ Loaded master label list. Total number of classes: {config.NUM_CLASSES}\")\n# Device setup\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nspecies_ids = pd.read_csv(config.submission_csv).columns[1:].tolist()","metadata":{"execution":{"iopub.status.busy":"2025-06-03T15:49:50.864380Z","iopub.execute_input":"2025-06-03T15:49:50.864653Z","iopub.status.idle":"2025-06-03T15:49:50.897120Z","shell.execute_reply.started":"2025-06-03T15:49:50.864630Z","shell.execute_reply":"2025-06-03T15:49:50.896537Z"},"papermill":{"duration":0.012329,"end_time":"2025-05-20T09:23:07.848146","exception":false,"start_time":"2025-05-20T09:23:07.835817","status":"completed"},"tags":[],"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 🏷️ Master Label List (After Cleaning and VAD Filtering)\n\nTo ensure training on high-quality and relevant audio data, we applied:\n\n- ✅ **Voice Activity Detection (VAD) filtering** to remove silent and noisy segments.\n- 🧹 **Data cleaning** to exclude low-rated or irrelevant samples.\n- 🔁 **Chunking** of longer audio files into fixed-duration segments (e.g., 5s or 10s).\n\nAfter these preprocessing steps, we extracted a list of **valid bird species labels** that are actually present in the cleaned dataset. This label list will serve as our **master label set**, which defines the total number of output classes for our model.\n\nWe loaded this list from the file `valid_labels.pkl`:\n","metadata":{}},{"cell_type":"code","source":"# Load the master label list and set NUM_CLASSES\nwith open(\"/kaggle/input/birdcleft-clean-and-vad-filtered-data/valid_labels.pkl\", \"rb\") as f:\n    master_labels = pickle.load(f)\nNUM_CLASSES = len(master_labels)  # should be 206\n\n\nprint(f\"total number of labels in full data : {len(master_labels)}\")  # Should print 206","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-03T15:46:17.072637Z","iopub.execute_input":"2025-06-03T15:46:17.073023Z","iopub.status.idle":"2025-06-03T15:46:17.091964Z","shell.execute_reply.started":"2025-06-03T15:46:17.073003Z","shell.execute_reply":"2025-06-03T15:46:17.091292Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def set_seed(seed=42):\n    \"\"\"\n    Set seed for reproducibility\n    \"\"\"\n    random.seed(seed)\n    os.environ[\"PYTHONHASHSEED\"] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.cuda.manual_seed_all(seed)\n    torch.backends.cudnn.deterministic = True\n    torch.backends.cudnn.benchmark = False\n\nset_seed(config.seed)\n","metadata":{"execution":{"iopub.status.busy":"2025-06-03T15:46:17.543696Z","iopub.execute_input":"2025-06-03T15:46:17.543967Z","iopub.status.idle":"2025-06-03T15:46:17.553898Z","shell.execute_reply.started":"2025-06-03T15:46:17.543947Z","shell.execute_reply":"2025-06-03T15:46:17.553204Z"},"papermill":{"duration":0.021291,"end_time":"2025-05-20T09:23:07.910633","exception":false,"start_time":"2025-05-20T09:23:07.889342","status":"completed"},"tags":[],"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 📥 Loading Clean Mel Spectrogram Data\n\nWe use a **cleaned and VAD-filtered CSV file** that contains metadata for high-quality audio chunks (10-second segments). Corresponding **precomputed mel spectrograms** are stored in `.npz` format for fast and efficient access during training.\n\nOnly files prefixed with `clean_train` are loaded to ensure that we are working strictly with clean training data. We then build an **index mapping from each spectrogram key to its `.npz` file**, allowing us to quickly locate and load the correct mel data for any given audio chunk.\n\nThis setup enables:\n- ⚡ Fast data loading during training  \n- 🧠 Memory-efficient batch management  \n- ✅ Guaranteed alignment between metadata and audio features  \n","metadata":{}},{"cell_type":"code","source":"# Your input directory\nmel_dir = \"/kaggle/input/birdclef-2025-mel-spectrogram\"\ncsv_path = \"/kaggle/input/birdcleft-clean-and-vad-filtered-data/train_audio_10sec_chunks_VAD_filtered.csv\"\n\n# Read only clean train dataframe\nclean_train_df = pd.read_csv(csv_path)\nprint(\"✅ CSV loaded:\", clean_train_df.shape)\n\n# Filter only clean_train mel batches\nclean_train_files = sorted([\n    f for f in os.listdir(mel_dir)\n    if f.startswith(\"clean_train\") and f.endswith(\".npz\")\n])\nprint(f\"🟢 Found {len(clean_train_files)} clean_train .npz files\")\n\n# Build key-to-file index for clean_train only\nkey_to_file = {}\n\nprint(\"🔍 Indexing clean_train mel keys...\")\nfor batch_file in tqdm(clean_train_files):\n    path = os.path.join(mel_dir, batch_file)\n    try:\n        npz = np.load(path)\n        for key in npz.files:\n            key_to_file[key] = batch_file\n    except Exception as e:\n        print(f\"⚠️ Error reading {batch_file}: {e}\")\n\nprint(f\"✅ Total keys indexed: {len(key_to_file)}\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-03T15:46:20.895957Z","iopub.execute_input":"2025-06-03T15:46:20.896620Z","iopub.status.idle":"2025-06-03T15:46:22.848541Z","shell.execute_reply.started":"2025-06-03T15:46:20.896595Z","shell.execute_reply":"2025-06-03T15:46:22.847783Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Master label filtering before any split\n#label_counts = clean_train_df['primary_label'].value_counts()\n#valid_labels = label_counts[label_counts >= 2].index\n\n\n\n#with open(\"valid_labels.pkl\", \"wb\") as f:\n    #pickle.dump(valid_labels.tolist(), f)\n# i have saved it once and then we will be loading it directly in dataset class","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-03T15:46:24.460316Z","iopub.execute_input":"2025-06-03T15:46:24.460621Z","iopub.status.idle":"2025-06-03T15:46:24.464090Z","shell.execute_reply.started":"2025-06-03T15:46:24.460601Z","shell.execute_reply":"2025-06-03T15:46:24.463447Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## ✅ Sanity Check: Load and Inspect Sample Mel Spectrograms\n\nTo verify that the precomputed mel spectrograms are loading correctly, we randomly selected a few `chunk_id`s from the clean training data and inspected their:\n\n- ✅ Shape\n- 📊 Min, Max, and Mean values\n\nThis helps ensure that:\n- All chunks are correctly indexed\n- Mel arrays are in expected format and range\n- There are no corrupted or missing keys in our `.npz` files\n","metadata":{}},{"cell_type":"code","source":"# Pick 3 random chunk_ids from the clean train CSV\nsample_keys = random.sample(clean_train_df['chunk_id'].tolist(), 3)\n\ndef load_clean_train_mel(chunk_id):\n    if chunk_id not in key_to_file:\n        raise ValueError(f\"❌ Key {chunk_id} not found.\")\n    file = key_to_file[chunk_id]\n    path = os.path.join(mel_dir, file)\n    with np.load(path) as npz:\n        return npz[chunk_id]\n\nfor key in sample_keys:\n    mel = load_clean_train_mel(key)\n    print(f\"\\nChunk ID: {key}\")\n    print(f\"Shape: {mel.shape}\")\n    print(f\"Min: {mel.min():.4f}, Max: {mel.max():.4f}, Mean: {mel.mean():.4f}\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-03T15:46:25.066603Z","iopub.execute_input":"2025-06-03T15:46:25.066853Z","iopub.status.idle":"2025-06-03T15:46:25.164532Z","shell.execute_reply.started":"2025-06-03T15:46:25.066835Z","shell.execute_reply":"2025-06-03T15:46:25.163914Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 🎛️ Custom Dataset: `BirdMelNPZDataset`\n\nThis class defines a PyTorch-compatible dataset that handles:\n\n- 📦 Loading precomputed **mel spectrograms** (saved in `.npz` files)\n- 🧠 Mapping each `chunk_id` to its corresponding spectrogram\n- 🏷️ Encoding **primary and secondary labels** as multi-hot vectors\n- 🔁 Optionally applying **data augmentation** (time/frequency masking, noise)\n\n### 🔍 Key Features:\n- ✅ Uses an indexed dictionary `key_to_file` to locate each spectrogram efficiently\n- 🧊 Caches `.npz` files per batch to reduce repeated disk reads\n- 🪄 Applies **on-the-fly audio-level augmentations** during training if `augment=True`\n- 🎯 Supports **multi-label targets** using both primary and secondary labels\n\nThis modular and efficient dataset class makes it easy to plug into a `DataLoader` and ensures high performance during training and validation.\n","metadata":{}},{"cell_type":"code","source":"class BirdMelNPZDataset(Dataset):\n    def __init__(self, df, mel_dir, key_to_file, labels=None, augment=False, sample_rate=32000):\n        self.df = df.reset_index(drop=True)\n        self.mel_dir = mel_dir\n        self.key_to_file = key_to_file\n        self.augment = augment\n        self.sample_rate = sample_rate\n\n        if labels is None:\n            with open(\"/kaggle/input/birdcleft-clean-and-vad-filtered-data/valid_labels.pkl\", \"rb\") as f:\n                self.labels = pickle.load(f)\n        else:\n            self.labels = labels\n\n        self.label2idx = {label: idx for idx, label in enumerate(self.labels)}\n        self.num_classes = len(self.labels)\n\n        self.df['chunk_id_clean'] = self.df['chunk_id'].str.replace('/', '-', regex=False)\n\n        # Cache loaded npz files to avoid multiple reloads in an epoch\n        self.loaded_batches = {}\n\n    def __len__(self):\n        return len(self.df)\n\n    def __getitem__(self, idx):\n        row = self.df.iloc[idx]\n        key = row['chunk_id_clean']\n\n        batch_file = self.key_to_file.get(key, None)\n        if batch_file is None:\n            # Key not found in index\n            mel = np.zeros((3, 256, 256), dtype=np.float32)  # assuming shape\n        else:\n            # Load batch file if not already loaded\n            if batch_file not in self.loaded_batches:\n                path = os.path.join(self.mel_dir, batch_file)\n                self.loaded_batches[batch_file] = np.load(path)\n\n            mel = self.loaded_batches[batch_file][key]\n\n        if self.augment:\n            mel = self.apply_augmentation(mel)\n\n        mel_tensor = torch.tensor(mel, dtype=torch.float32)  # (3, 256, 256)\n\n        label_vec = np.zeros(self.num_classes, dtype=np.float32)\n        label_vec[self.label2idx[row['primary_label']]] = 1.0\n\n        # Handling secondary labels if present\n        if 'secondary_labels' in row and isinstance(row['secondary_labels'], list):\n            for sec in row['secondary_labels']:\n                if sec in self.label2idx:\n                    label_vec[self.label2idx[sec]] = 1.0\n\n        return mel_tensor, torch.tensor(label_vec)\n\n    # (Augmentation methods )\n\n\n    def apply_augmentation(self, mel):\n        mel = self.time_mask(mel, T=30)\n        mel = self.freq_mask(mel, F=15)\n        mel = self.add_noise(mel, noise_level=0.01)\n        return mel\n\n    def time_mask(self, mel, T=30):\n        t = mel.shape[2]  # time dimension\n        t0 = np.random.randint(0, max(1, t - T))\n        mel[:, :, t0:t0 + T] = 0\n        return mel\n\n    def freq_mask(self, mel, F=15):\n        f = mel.shape[1]  # frequency dimension\n        f0 = np.random.randint(0, max(1, f - F))\n        mel[:, f0:f0 + F, :] = 0\n        return mel\n\n    def add_noise(self, mel, noise_level=0.01):\n        noise = np.random.randn(*mel.shape) * noise_level\n        return mel + noise\n\n\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-03T15:46:29.006363Z","iopub.execute_input":"2025-06-03T15:46:29.006650Z","iopub.status.idle":"2025-06-03T15:46:29.017539Z","shell.execute_reply.started":"2025-06-03T15:46:29.006630Z","shell.execute_reply":"2025-06-03T15:46:29.016819Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 🧩 Collate Function: `collate_pad_mel`\n\nThis custom collate function ensures all mel spectrograms in a batch have the **same time dimension** by:\n\n- 🔍 Finding the max time length in the batch\n- 🧱 Padding shorter spectrograms on the time axis using `torch.nn.functional.pad`\n- 📦 Stacking padded tensors into a batch of shape `(B, C, n_mels, max_time)`\n\nThis is essential when working with variable-length audio chunks to enable smooth batching in PyTorch `DataLoader`.\n","metadata":{}},{"cell_type":"code","source":"# collate function to pad mel spectrograms on time dimension (if varying length)\ndef collate_pad_mel(batch):\n    mel_specs = [item[0] for item in batch]  # each shape (C, n_mels, time)\n    targets = [item[1] for item in batch]\n\n    max_time = max([m.shape[-1] for m in mel_specs])\n\n    padded_mels = []\n    for m in mel_specs:\n        pad_len = max_time - m.shape[-1]\n        padded = F.pad(m, (0, pad_len))\n        padded_mels.append(padded)\n\n    mel_specs_padded = torch.stack(padded_mels)  # (B, C, n_mels, max_time)\n    targets = torch.stack(targets)\n\n    return mel_specs_padded, targets\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-03T15:46:36.214007Z","iopub.execute_input":"2025-06-03T15:46:36.214638Z","iopub.status.idle":"2025-06-03T15:46:36.219072Z","shell.execute_reply.started":"2025-06-03T15:46:36.214615Z","shell.execute_reply":"2025-06-03T15:46:36.218533Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## ⚖️ Balanced Sampling: `get_balanced_sampler`\n\nTo address class imbalance during training, we use a **`WeightedRandomSampler`**, which ensures that underrepresented classes are sampled more frequently.\n\n### 🔧 How it works:\n- 📊 Calculates class frequencies from the `primary_label` column\n- 🧮 Assigns **inverse-frequency weights** to each sample\n- 🎲 Uses `WeightedRandomSampler` to draw samples with replacement based on these weights\n\nThis helps create more balanced mini-batches during training and improves model performance on rare classes.\n","metadata":{}},{"cell_type":"code","source":"def get_balanced_sampler(df, label_col='primary_label'):\n    \"\"\"\n    Create a WeightedRandomSampler to balance class frequencies.\n\n    Args:\n        df: pandas DataFrame with a 'primary_label' column\n        label_col: name of the column containing class labels\n\n    Returns:\n        torch.utils.data.WeightedRandomSampler\n    \"\"\"\n    # Count samples per class\n    class_counts = df[label_col].value_counts()\n    class_weights = 1. / class_counts\n\n    # Assign each sample its weight\n    sample_weights = df[label_col].map(class_weights).values\n    sample_weights = torch.tensor(sample_weights, dtype=torch.float32)\n\n    sampler = WeightedRandomSampler(\n        weights=sample_weights,\n        num_samples=len(sample_weights),\n        replacement=True\n    )\n    return sampler","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-03T15:46:39.556965Z","iopub.execute_input":"2025-06-03T15:46:39.557677Z","iopub.status.idle":"2025-06-03T15:46:39.562014Z","shell.execute_reply.started":"2025-06-03T15:46:39.557650Z","shell.execute_reply":"2025-06-03T15:46:39.561358Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 🧪 Train/Validation Split & DataLoader Setup\n\nWe prepare the final dataset and dataloaders as follows:\n\n### 🧹 Filter Rare Classes:\n- Removed any `primary_label` classes that have **fewer than 2 samples** to avoid training instability.\n\n### 🔀 Stratified Split:\n- Used **stratified sampling** to split `clean_train_df` into **80% training** and **20% validation** while preserving class distribution.\n\n### 🧺 Dataset Construction:\n- Created two instances of `BirdMelNPZDataset`:\n  - `train_dataset` with **on-the-fly augmentation**\n  - `val_dataset` without augmentation\n\n### ⚖️ Balanced Sampling:\n- For training, used a **`WeightedRandomSampler`** to balance class representation per mini-batch.\n\n### 🚚 Dataloader Initialization:\n- Created `train_loader` and `val_loader` using PyTorch `DataLoader` with:\n  - `batch_size = 32`\n  - `collate_pad_mel()` to pad mel spectrograms\n  - `train_loader` uses balanced sampler\n  - `val_loader` shuffles disabled for evaluation consistency\n\n### ✅ Dataset Sanity Check:\nPrinted a few sample entries from the train and validation datasets to ensure:\n- Mel spectrogram shapes are correct\n- Label vectors are properly one-hot/multi-hot encoded\n","metadata":{}},{"cell_type":"code","source":"# Count instances per class\nlabel_counts = clean_train_df['primary_label'].value_counts()\n\n# Keep only those classes with at least 2 samples\nvalid_labels = label_counts[label_counts >= 2].index\nclean_train_df= clean_train_df[clean_train_df['primary_label'].isin(valid_labels)].reset_index(drop=True)\nvalid_labels.nunique()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-03T15:46:42.687550Z","iopub.execute_input":"2025-06-03T15:46:42.687822Z","iopub.status.idle":"2025-06-03T15:46:42.714678Z","shell.execute_reply.started":"2025-06-03T15:46:42.687799Z","shell.execute_reply":"2025-06-03T15:46:42.714088Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Assume mel_dir and key_to_file already defined as:\nmel_dir = \"/kaggle/input/birdclef-2025-mel-spectrogram\"\n\n# Split dataframe into train and validation stratified on primary_label\ntrain_idx, val_idx = train_test_split(\n    range(len(clean_train_df)),\n    test_size=0.2,\n    stratify=clean_train_df['primary_label'],\n    random_state=42\n)\n\ntrain_df = clean_train_df.iloc[train_idx].reset_index(drop=True)\nval_df = clean_train_df.iloc[val_idx].reset_index(drop=True)\n\n#alag alag datset create \ntrain_dataset = BirdMelNPZDataset(\n    train_df,\n    mel_dir=mel_dir,\n    key_to_file=key_to_file,\n    augment=True  # Enable augmentations in training\n)\n\nval_dataset = BirdMelNPZDataset(\n    val_df,\n    mel_dir=mel_dir,\n    key_to_file=key_to_file,\n    augment=False\n)\n\n# 🔄 Balanced Sampler for training (if you have implemented `get_balanced_sampler`)\ntrain_sampler = get_balanced_sampler(train_df, label_col='primary_label')\n\ntrain_loader = torch.utils.data.DataLoader(\n    train_dataset,\n    batch_size=32,\n    sampler=train_sampler,\n    collate_fn=collate_pad_mel\n)\n\nval_loader = torch.utils.data.DataLoader(\n    val_dataset,\n    batch_size=32,\n    shuffle=False,\n    collate_fn=collate_pad_mel\n)\n\ndef check_dataset(dataset, name=\"Dataset\"):\n    mel_tensor,  label_tensor = dataset[0]\n    print(f\"=== {name.upper()} ===\")\n    print(f\"Mel Spectrogram shape: {mel_tensor.shape}\")         # Expected: (128, Time)\n    #print(f\"YAMNet Embedding shape: {embedding_tensor.shape}\")  # Expected: (1024,)\n    print(f\"Label shape: {label_tensor.shape}\")                 # Expected: (206,)\n\n# Check train dataset\ncheck_dataset(train_dataset, \"Train Dataset\")\n\nsample_idx = 0  # koi b sample index\n\n# Train dataset ka sample\nprint(\"=== TRAIN DATASET SAMPLE ===\")\nprint(type(train_dataset))\nprint(train_dataset[sample_idx])\n\n# Validation dataset ka sample\nprint(\"=== VALIDATION DATASET SAMPLE ===\")\nprint(type(val_dataset))\nprint(val_dataset[sample_idx])\n\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-03T15:46:44.862596Z","iopub.execute_input":"2025-06-03T15:46:44.863154Z","iopub.status.idle":"2025-06-03T15:46:45.138165Z","shell.execute_reply.started":"2025-06-03T15:46:44.863136Z","shell.execute_reply":"2025-06-03T15:46:45.137525Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 📊 Visualizing Mel Spectrograms\n\nTo better understand our training data, we define a helper function `plot_multiple_mels` to visualize multiple mel spectrograms side-by-side.\n\n### 🔍 What it does:\n- Plots mel spectrograms from given sample indices\n- Uses only the **first channel** of each mel tensor\n- Adds time and frequency axis labels for clarity\n- Automatically includes a colorbar for reference\n\nThis is useful for verifying the quality, shape, and content of mel spectrograms before training.\n\n### 🖼️ Example Outputs:\nBelow we plot 5 random samples from both the training and validation datasets to visually inspect the spectrograms.\n","metadata":{}},{"cell_type":"code","source":"def plot_multiple_mels(dataset, indices, title_prefix='Sample Mel Spectrogram'):\n    n = len(indices)\n    fig, axes = plt.subplots(1, n, figsize=(4*n, 4))  # 1 row, n columns\n\n    for i, idx in enumerate(indices):\n        mel_tensor, _ = dataset[idx]\n        mel = mel_tensor[0].numpy()  # first channel\n\n        ax = axes[i] if n > 1 else axes\n        im = ax.imshow(mel, aspect='auto', origin='lower')\n        ax.set_title(f\"{title_prefix} #{idx}\")\n        ax.set_xlabel(\"Time Frames\")\n        ax.set_ylabel(\"Mel Bands\")\n        ax.label_outer()  # Only show outer labels for clean look\n\n    fig.colorbar(im, ax=axes, orientation='vertical', fraction=0.02, pad=0.04)\n    plt.tight_layout()\n    plt.show()\n\n# Example usage:\nprint(\"Train Samples:\")\nplot_multiple_mels(train_dataset, indices=list(range(5)), title_prefix='Train Sample Mel')\n\nprint(\"Validation Samples:\")\nplot_multiple_mels(val_dataset, indices=list(range(5)), title_prefix='Validation Sample Mel')\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-03T15:46:51.898946Z","iopub.execute_input":"2025-06-03T15:46:51.899566Z","iopub.status.idle":"2025-06-03T15:46:54.446134Z","shell.execute_reply.started":"2025-06-03T15:46:51.899546Z","shell.execute_reply":"2025-06-03T15:46:54.445302Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 🎲 Sampling Analysis with Balanced Sampler ( uper humny sampler bnaya tha yahan bus use kr k check kry gy)\n\nTo verify that the `WeightedRandomSampler` balances the training data, we:\n\n- Print the number of unique labels and sampler length\n- Show sample weights of the first 10 samples (inverse frequency weights)\n- Collect labels sampled during the first 50 batches from the `train_loader`\n- Count and visualize the class distribution of these sampled labels\n\n### 📈 Observation:\nThe WeightedRandomSampler successfully balances training data by assigning higher sampling weights to rare classes and lower weights to frequent ones. The sampled class distribution over the first 50 batches shows roughly equal representation across classes, improving the model's ability to learn from minority classes and reducing bias toward majority classes.","metadata":{}},{"cell_type":"code","source":"sampler = get_balanced_sampler(train_df)\n\nprint(\"Unique labels:\", train_df['primary_label'].nunique())\nprint(\"Sampler length:\", len(sampler))\nsample_weights = [sampler.weights[i].item() for i in range(10)]\nprint(\"First 10 sample weights:\", sample_weights)\n\nsampled_labels = []\nfor i, (mels, labels) in enumerate(train_loader):\n    sampled_labels.extend(labels.argmax(dim=1).tolist())  # One-hot to class index\n    if i > 50:  # Analyze only first 50 batches\n        break\n\nprint(\"Sampled class distribution in first 50 batches:\", collections.Counter(sampled_labels))\nlabel_counts = dict(collections.Counter(sampled_labels))\n\nplt.figure(figsize=(14, 6))\nsns.barplot(x=list(label_counts.keys()), y=list(label_counts.values()))\nplt.title(\"Sampled Label Distribution (First 50 Batches)\")\nplt.xlabel(\"Class Index\")\nplt.ylabel(\"Count\")\nplt.xticks(rotation=90)\nplt.tight_layout()\nplt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-03T15:46:55.165794Z","iopub.execute_input":"2025-06-03T15:46:55.166278Z","iopub.status.idle":"2025-06-03T15:47:34.142758Z","shell.execute_reply.started":"2025-06-03T15:46:55.166257Z","shell.execute_reply":"2025-06-03T15:47:34.142066Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 🦜 Improved BirdCLEF Model Architecture\n\nWe utilize a powerful pretrained backbone from the `timm` library:\n\n- **Backbone:** `seresnext26t_32x4d` pretrained on ImageNet-1k\n  - Accepts 3-channel mel spectrogram inputs\n  - Outputs a 2048-dimensional feature vector (no classification head)\n\n- **Classifier Head:**\n  - Fully connected layer reducing 2048 → 512 features\n  - Batch normalization + ReLU activation + dropout (0.3)\n  - Final linear layer projecting 512 → number of bird classes (default 187)\n\nThis design helps in fine-tuning a strong feature extractor while training a task-specific classifier.\n\n---\n\n## 🧊 Backbone Freezing & Unfreezing Functions\n\nTo control training dynamics and speed up convergence:\n\n- `freeze_backbone(model)`: Freeze all backbone layers to train only the classifier head initially.\n- `unfreeze_last_blocks(model, num_blocks=2)`: Unfreeze the last few backbone blocks to fine-tune deeper features gradually.\n\nThis gradual unfreezing is useful to avoid catastrophic forgetting and improve performance.\n","metadata":{}},{"cell_type":"code","source":"#model download kr k weights save kr lety hain (inference me intenet allowed nhi hai to it would be helpful that time)\n#model = timm.create_model(\n#    \"hf_hub:timm/seresnext26t_32x4d.bt_in1k\",\n#    pretrained=True,\n#    in_chans=3,\n#    num_classes=0)\n\n# Save the backbone weights locally\n#torch.save(model.state_dict(), \"seresnext_backbone_weights.pth\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-03T15:21:37.435231Z","iopub.execute_input":"2025-06-03T15:21:37.435473Z","iopub.status.idle":"2025-06-03T15:21:37.439356Z","shell.execute_reply.started":"2025-06-03T15:21:37.435457Z","shell.execute_reply":"2025-06-03T15:21:37.438521Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"class ImprovedBirdCLEFModel(nn.Module):\n    def __init__(self, num_classes=187, pretrained=True):\n        super().__init__()\n        \n        self.backbone = timm.create_model(\n            \"seresnext26t_32x4d\",  # ⬅️ hf_hub hata diya downloaded weights use kry gy\n            pretrained=False,  \n            in_chans=3,\n            num_classes=0\n        )\n\n        if pretrained:\n            state_dict = torch.load(\"/kaggle/input/birdcleft-clean-and-vad-filtered-data/seresnext_backbone_weights.pth\", map_location=device)  # local path\n            self.backbone.load_state_dict(state_dict, strict=False)  # load weights manually download kiye hain\n\n        \n        self.classifier = nn.Sequential(\n            nn.Linear(self.backbone.num_features, 512),\n            nn.BatchNorm1d(512),\n            nn.ReLU(inplace=True),\n            nn.Dropout(0.3),\n            nn.Linear(512, num_classes)\n        )\n\n    def forward(self, x):\n        x = self.backbone(x)             # Already [B, 2048]\n        #print(\"After backbone:\", x.shape)\n        x = self.classifier(x)           # [B, num_classes]\n        return x\n\n\ndef freeze_backbone(model):\n    for param in model.backbone.parameters():\n        param.requires_grad = False\n\ndef unfreeze_last_blocks(model, num_blocks=2):\n    # Unfreezes last few blocks of the backbone\n    children = list(model.backbone.children())\n    for block in children[-num_blocks:]:\n        for param in block.parameters():\n            param.requires_grad = True\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-03T15:37:36.640324Z","iopub.execute_input":"2025-06-03T15:37:36.640634Z","iopub.status.idle":"2025-06-03T15:37:36.648896Z","shell.execute_reply.started":"2025-06-03T15:37:36.640602Z","shell.execute_reply":"2025-06-03T15:37:36.647921Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 🚀 Model Initialization and Layer Freezing Strategy\n\n- We initialize the `ImprovedBirdCLEFModel` with the full number of classes and load pretrained weights.\n- The model is moved to the available device (GPU if available, else CPU).\n- **Step 1:** Freeze the entire backbone to prevent updating pretrained weights initially.\n- **Step 2:** Gradually unfreeze the last 3 blocks of the backbone to allow fine-tuning on more task-specific features while keeping earlier layers fixed.\n- Finally, we print out the names of all trainable parameters to verify which layers will be updated during training.\n\nThis strategy helps in faster convergence and avoids overfitting early in training.\n","metadata":{}},{"cell_type":"code","source":"\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n\nmodel = ImprovedBirdCLEFModel(num_classes=NUM_CLASSES, pretrained=True).to(device)\n\n# Step 1: Freeze full backbone\nfreeze_backbone(model)\n\n# Step 2: Unfreeze last N blocks (e.g., last 3 blocks)\nunfreeze_last_blocks(model, num_blocks=3)\n\n# ✅ Print trainable parameters (for sanity check)\nfor name, param in model.named_parameters():\n    if param.requires_grad:\n        print(f\"Trainable: {name}\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-03T15:21:54.102971Z","iopub.execute_input":"2025-06-03T15:21:54.103259Z","iopub.status.idle":"2025-06-03T15:21:55.319096Z","shell.execute_reply.started":"2025-06-03T15:21:54.103237Z","shell.execute_reply":"2025-06-03T15:21:55.318373Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 🔧 Training Setup\n\n1. **Device Configuration:** Automatically use GPU if available for faster training.\n2. **Model Initialization:** Load the pretrained `ImprovedBirdCLEFModel` with the specified number of classes.\n3. **Backbone Freezing:** Freeze all backbone layers to preserve pretrained features initially.\n4. **Selective Fine-tuning:** Unfreeze the last 3 backbone blocks to allow gradual adaptation.\n5. **Loss Function:** Use `BCEWithLogitsLoss` suitable for multi-label classification.\n6. **Optimizer:** Use `AdamW` optimizer with weight decay for better generalization.\n7. **Learning Rate Scheduler:** Optional cosine annealing scheduler to adjust learning rate smoothly during training.\n\nThis setup balances leveraging pretrained knowledge and fine-tuning on the BirdCLEF 2025 dataset effectively.\n","metadata":{}},{"cell_type":"code","source":"# Device setup\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n\n# Step 1: Load model with classifier\nmodel = ImprovedBirdCLEFModel(num_classes=NUM_CLASSES, pretrained=True).to(device)\n\n# Step 2: Freeze backbone\nfreeze_backbone(model)\n\n# Step 3: Unfreeze last N blocks for fine-tuning\nunfreeze_last_blocks(model, num_blocks=3)\n\n# Step 4: Define loss function\ncriterion = nn.BCEWithLogitsLoss()\n\n# Step 5: Define optimizer (AdamW for better regularization)\noptimizer = optim.AdamW(\n    filter(lambda p: p.requires_grad, model.parameters()),  # only trainable params\n    lr=1e-4,\n    weight_decay=1e-4\n)\n\n# Step 6: Define LR scheduler (optional, you can add in training loop)\nscheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=10)\n\n# ✅ Print summary\nprint(\"🔥 Model compiled and ready for training!\")\nprint(f\"Loss: {criterion.__class__.__name__}\")\nprint(f\"Optimizer: {optimizer.__class__.__name__} | LR: {optimizer.param_groups[0]['lr']}\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-03T15:22:07.591329Z","iopub.execute_input":"2025-06-03T15:22:07.591611Z","iopub.status.idle":"2025-06-03T15:22:07.948417Z","shell.execute_reply.started":"2025-06-03T15:22:07.591588Z","shell.execute_reply":"2025-06-03T15:22:07.947678Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Phase 1 training\n### Mixup and Training Utilities (Short Description)\n\n- **mixup_data:** Mixes inputs and targets using Beta distribution for augmentation.  \n- **mixup_criterion:** Calculates loss for mixed targets with weighting.  \n- **EarlyStopping:** Stops training if metric doesn’t improve for `patience` epochs by at least `min_delta`.  \n- **kaggle_macro_roc_auc_ignoring_empty:** Computes macro ROC-AUC ignoring empty classes (Kaggle metric).  \n- **train_one_epoch:** Trains model for one epoch with optional mixup augmentation.  \n- **validate_model:** Evaluates model on validation set using given metric without gradients.  \n\nThis helps with robust multi-label model training and evaluation.\n","metadata":{}},{"cell_type":"code","source":"# Mixup helper functions\ndef mixup_data(x, y, alpha=0.4):\n    \"\"\"Returns mixed inputs, pairs of targets, and lambda\"\"\"\n    if alpha > 0:\n        lam = np.random.beta(alpha, alpha)\n    else:\n        lam = 1.0\n\n    batch_size = x.size(0)\n    index = torch.randperm(batch_size).to(x.device)\n\n    mixed_x = lam * x + (1 - lam) * x[index, :]\n    y_a, y_b = y, y[index]\n    return mixed_x, y_a, y_b, lam\n\ndef mixup_criterion(criterion, pred, y_a, y_b, lam):\n    return lam * criterion(pred, y_a) + (1 - lam) * criterion(pred, y_b)\n\n# EarlyStopping Class\nclass EarlyStopping:\n    def __init__(self, patience=5, min_delta=0.001):\n        self.patience = patience\n        self.min_delta = min_delta\n        self.best_score = None\n        self.counter = 0\n        self.early_stop = False\n\n    def __call__(self, metric):\n        if self.best_score is None or metric > self.best_score + self.min_delta:\n            self.best_score = metric\n            self.counter = 0\n        else:\n            self.counter += 1\n            if self.counter >= self.patience:\n                self.early_stop = True\n\n\n# Your kaggle metric function you shared:\ndef kaggle_macro_roc_auc_ignoring_empty(y_true: np.ndarray, y_probs: np.ndarray) -> float:\n    valid_classes = np.where(y_true.sum(axis=0) > 0)[0]\n    if len(valid_classes) == 0:\n        return float('nan')\n    return roc_auc_score(y_true[:, valid_classes], y_probs[:, valid_classes], average='macro')\n\n\n# Training Loop\ndef train_one_epoch(model, optimizer, criterion, dataloader, device, alpha=0.4):\n    model.train()\n    running_loss = 0.0\n\n    for batch in tqdm(dataloader, desc=\"Training\"):\n        inputs, targets = batch  # <- tuple unpacking here\n        inputs = inputs.to(device)\n        targets = targets.to(device)\n\n        if alpha > 0:\n            inputs, targets_a, targets_b, lam = mixup_data(inputs, targets, alpha)\n            inputs, targets_a, targets_b = inputs.to(device), targets_a.to(device), targets_b.to(device)\n            outputs = model(inputs)\n            loss = mixup_criterion(criterion, outputs, targets_a, targets_b, lam)\n        else:\n            outputs = model(inputs)\n            loss = criterion(outputs, targets)\n\n        optimizer.zero_grad()\n        loss.backward()\n        optimizer.step()\n\n        running_loss += loss.item()\n\n    return running_loss / len(dataloader)\n\n# Validation function (using your Kaggle ROC metric)\ndef validate_model(model, dataloader, device, kaggle_macro_roc_auc_ignoring_empty):\n    model.eval()\n    all_preds = []\n    all_targets = []\n\n    with torch.no_grad():\n        for batch in dataloader:\n            inputs, targets = batch  # ✅ fixed here\n            inputs = inputs.to(device)\n            targets = targets.cpu().numpy()\n\n            outputs = model(inputs)\n            preds = torch.sigmoid(outputs).cpu().numpy()\n\n            all_preds.append(preds)\n            all_targets.append(targets)\n\n    all_preds = np.vstack(all_preds)\n    all_targets = np.vstack(all_targets)\n\n    score = kaggle_macro_roc_auc_ignoring_empty(all_targets, all_preds)\n    return score\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-03T15:47:34.144010Z","iopub.execute_input":"2025-06-03T15:47:34.144234Z","iopub.status.idle":"2025-06-03T15:47:34.156560Z","shell.execute_reply.started":"2025-06-03T15:47:34.144218Z","shell.execute_reply":"2025-06-03T15:47:34.155678Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Training Loop \n\n- Runs for a maximum of `num_epochs` (30).\n- For each epoch:\n  - Trains model for one epoch with mixup.\n  - Validates model and computes Kaggle ROC-AUC.\n  - Saves training loss and validation score to history.\n  - Saves model weights if validation score improves.\n  - Checks early stopping condition (patience=2, min_delta=0.0005).\n- If no improvement after `patience` epochs, training stops early.\n- Finally, loads best model weights saved during training.\n","metadata":{}},{"cell_type":"code","source":"#training setup\nnum_epochs = 30\nearly_stopping = EarlyStopping(patience=2, min_delta=0.0005)\nbest_model_wts = copy.deepcopy(model.state_dict())\nbest_score = -np.inf\n\nhistory = {'train_loss': [], 'val_score': []}\n\nfor epoch in range(num_epochs):\n    print(f\"\\nEpoch {epoch+1}/{num_epochs}\")\n\n    train_loss = train_one_epoch(model, optimizer, criterion, train_loader, device, alpha=0.4)\n    val_score = validate_model(model, val_loader, device, kaggle_macro_roc_auc_ignoring_empty)\n\n    history['train_loss'].append(train_loss)\n    history['val_score'].append(val_score)\n\n    print(f\"Train Loss: {train_loss:.4f} | Validation Kaggle ROC-AUC: {val_score:.5f}\")\n\n    # Save best model\n    if val_score > best_score:\n        best_score = val_score\n        best_model_wts = copy.deepcopy(model.state_dict())\n        torch.save(model.state_dict(), \"best_model_phase1.pth\")\n        print(\"=> Saved Best Model!\")\n\n    # Early stopping check\n    early_stopping(val_score)\n    if early_stopping.early_stop:\n        print(\"Early stopping triggered!\")\n        break\n\n# Load best model weights after training\nmodel.load_state_dict(best_model_wts)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-01T23:36:07.574599Z","iopub.execute_input":"2025-06-01T23:36:07.575493Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### plot_training_history(history)\n\n- Takes `history` dictionary containing 'train_loss' and 'val_score' lists.\n- Plots training loss and validation Kaggle Macro ROC-AUC over epochs.\n- Shows trends to help diagnose underfitting, overfitting, or convergence issues.\n- Useful to run right after training to visualize model performance.\n","metadata":{}},{"cell_type":"code","source":"def plot_training_history(history):\n    epochs = range(1, len(history['train_loss']) + 1)\n\n    plt.figure(figsize=(12,5))\n\n    # Train loss plot\n    plt.subplot(1, 2, 1)\n    plt.plot(epochs, history['train_loss'], 'b-', label='Train Loss')\n    plt.xlabel('Epoch')\n    plt.ylabel('Loss')\n    plt.title('Training Loss Over Epochs')\n    plt.legend()\n\n    # Validation Kaggle Macro ROC-AUC plot\n    plt.subplot(1, 2, 2)\n    plt.plot(epochs, history['val_score'], 'g-', label='Validation Kaggle Macro ROC-AUC')\n    plt.xlabel('Epoch')\n    plt.ylabel('ROC-AUC')\n    plt.title('Validation Score Over Epochs')\n    plt.legend()\n\n    plt.tight_layout()\n    plt.show()\n\n# Use this after training:\nplot_training_history(history)\n","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"---\n### Next Steps: Phase 2 Training\n\nTo further improve the model, proceed with **Phase 2 training**, which incorporates advanced fine-tuning techniques and potentially additional data augmentation strategies.\n\nExplore the detailed Phase 2 training notebook here:  \n[BirdCLEF 2025 Model Training Phase 2](https://www.kaggle.com/code/sheemamasood/birdclef2025-model-training-phase2)  \n- This phase includes self-supervised learning using pseudo-labeling to boost performance.\n-------------","metadata":{}},{"cell_type":"markdown","source":"## for debugging","metadata":{}},{"cell_type":"code","source":"# ===================== MODEL =====================\nclass ImprovedBirdCLEFModel(nn.Module):\n    def __init__(self, num_classes=187, backbone_weights=None, device=device):\n        super().__init__()\n        self.backbone = timm.create_model(\n            \"seresnext26t_32x4d\",\n            pretrained=False,\n            in_chans=3,\n            num_classes=0\n        )\n        if backbone_weights:\n            state_dict = torch.load(backbone_weights, map_location=device)\n            self.backbone.load_state_dict(state_dict, strict=False)\n        self.classifier = nn.Sequential(\n            nn.Linear(self.backbone.num_features, 512),\n            nn.BatchNorm1d(512),\n            nn.ReLU(inplace=True),\n            nn.Dropout(0.3),\n            nn.Linear(512, num_classes)\n        )\n    def forward(self, x):\n        x = self.backbone(x)\n        x = self.classifier(x)\n        return x\n\ndef load_model(model_path, device, num_classes, backbone_weights):\n    model = ImprovedBirdCLEFModel(num_classes=num_classes, backbone_weights=backbone_weights, device=device)\n    state_dict = torch.load(model_path, map_location=device)\n    model.load_state_dict(state_dict)\n    model = model.to(device)\n    model.eval()\n    return model\n    \nmodel = load_model(\n    model_path=config.model_path,\n    device=device,\n    num_classes=187,\n    backbone_weights=config.backbone_weights  # Pass it as a keyword argument\n)\nprint(\"✅ Model loaded!\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-03T15:47:34.157410Z","iopub.execute_input":"2025-06-03T15:47:34.157993Z","iopub.status.idle":"2025-06-03T15:47:35.564110Z","shell.execute_reply.started":"2025-06-03T15:47:34.157974Z","shell.execute_reply":"2025-06-03T15:47:35.563314Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Validate model on validation dataloader\nval_score = validate_model(model, val_loader, device, kaggle_macro_roc_auc_ignoring_empty)\nprint(f\"📊 Validation Macro ROC-AUC: {val_score:.4f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-03T15:47:35.565820Z","iopub.execute_input":"2025-06-03T15:47:35.566085Z","iopub.status.idle":"2025-06-03T15:49:50.855269Z","shell.execute_reply.started":"2025-06-03T15:47:35.566066Z","shell.execute_reply":"2025-06-03T15:49:50.854593Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def debug_validate_model(model, dataloader, device, species_ids, num_batches=1):\n    model.eval()\n    all_preds = []\n    all_targets = []\n\n    with torch.no_grad():\n        for batch_idx, batch in enumerate(dataloader):\n            if batch_idx >= num_batches:\n                break\n\n            inputs, targets = batch\n            inputs = inputs.to(device)\n            targets = targets.cpu().numpy()\n\n            outputs = model(inputs)\n            preds = torch.sigmoid(outputs).cpu().numpy()\n\n            all_preds.append(preds)\n            all_targets.append(targets)\n\n            # 🖨️ Print predictions\n            for i in range(len(preds)):\n                top5_idx = preds[i].argsort()[-5:][::-1]  # Top 5 predictions\n                top5_probs = preds[i][top5_idx]\n                true_indices = np.where(targets[i] == 1)[0]\n\n                print(f\"\\n🔊 Sample {i+1}:\")\n                print(f\"✅ True Labels: {[species_ids[j] for j in true_indices]}\")\n                print(f\"🔮 Top Predictions:\")\n                for rank, (cls_idx, prob) in enumerate(zip(top5_idx, top5_probs), start=1):\n                    print(f\"  {rank}. {species_ids[cls_idx]} ({prob:.3f})\")\n\n    return\n\n\ndebug_validate_model(\n    model=model,\n    dataloader=val_loader,\n    device=device,\n    species_ids=species_ids,  # a list like ['species_1', ..., 'species_187']\n    num_batches=2  # check first 2 batches only\n)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-03T15:50:06.511424Z","iopub.execute_input":"2025-06-03T15:50:06.511938Z","iopub.status.idle":"2025-06-03T15:50:07.263996Z","shell.execute_reply.started":"2025-06-03T15:50:06.511916Z","shell.execute_reply":"2025-06-03T15:50:07.263354Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}