{"metadata":{"kernelspec":{"display_name":"src-xS3fZVNL-py3.10","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.12"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":91844,"databundleVersionId":11361821,"sourceType":"competition"}],"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"id":"5ed02a1b-72ee-475e-9794-a6c0396fea10","cell_type":"markdown","source":"# BirdSet Data Pipeline Tutorial [Copied from BirdSet Repository](https://github.com/DBD-research-group/BirdSet)\n\nThis Jupyter notebook provides a comprehensive guide to setting up and configuring a data pipeline tailored for bird classification in audio files. The tutorial is structured to walk you through each component of the pipeline, ensuring a clear understanding of its functionality and configuration. Whether you are processing raw audio data or spectrograms, this notebook aims to provide you with the necessary knowledge to efficiently set up your data pipeline.","metadata":{}},{"id":"0909c001-7d00-4a0f-a16c-c984e7d91740","cell_type":"markdown","source":"## Installation\n\n### Prerequisites\nBefore initiating the installation process of the BirdSet pipeline, it's crucial to ensure that your computing environment meets the following prerequisites:\n- **Python**: You should have Python version 3.10 or higher installed on your system.\n\n### Installation Steps\nThe BirdSet pipeline can be installed using either of the two methods: via Conda with Pip, or using Poetry. Select the method that best suits your preference and follow the corresponding steps below.\n\n#### Using Conda and Pip\n\n1. **Create a Conda Environment**: Begin by setting up a dedicated environment for BirdSet. This is a best practice to manage dependencies and avoid potential conflicts with other packages in your system.\n\n   ```bash\n   conda create -n birdset python=3.10\n   ```\n\n   After the environment is successfully created, activate it:\n\n   ```bash\n   conda activate birdset\n   ```\n\n2. **Install BirdSet**: Proceed with cloning the BirdSet repository and installing the package in editable mode. This approach is beneficial as it allows any modifications you make to the BirdSet code to be reflected immediately without the need for reinstallation.\n\n   ```bash\n   git clone https://github.com/DBD-research-group/BirdSet.git\n   cd BirdSet\n   pip install -e .\n   ```\n\n#### Using Poetry\n\n1. **Clone the Repository**: Start with cloning the BirdSet repository to your local machine and navigate to the cloned directory.\n\n   ```bash\n   git clone https://github.com/DBD-research-group/BirdSet.git\n   cd BirdSet\n   ```\n\n2. **Install Dependencies and Activate Environment**: Install all the necessary dependencies with Poetry and then activate the Poetry shell environment.\n\n   ```bash\n   poetry install\n   poetry shell\n   ```","metadata":{}},{"id":"41a4f398-f18d-42e4-a6b5-02329f1f5567","cell_type":"markdown","source":"## Log in to Huggingface\n\nOur datasets are shared via HuggingFace Datasets in our [HuggingFace BirdSet repository](https://huggingface.co/datasets/DBD-research-group/birdset_v1). Huggingface is a central hub for sharing and utilizing datasets and models, particularly beneficial for machine learning and data science projects. For accessing private datasets hosted on HuggingFace, you need to be authenticated. Here's how you can log in to HuggingFace:\n\n1. **Install HuggingFace CLI**: If you haven't already, you need to install the HuggingFace CLI (Command Line Interface). This tool enables you to interact with HuggingFace services directly from your terminal. You can install it using pip:\n\n   ```bash\n   pip install huggingface_hub\n   ```\n\n2. **Login via CLI**: Once the HuggingFace CLI is installed, you can log in to your HuggingFace account directly from your terminal. This step is essential for accessing private datasets or contributing to the HuggingFace community. Use the following command:\n\n   ```bash\n   huggingface-cli login\n   ```\n\n   After executing this command, you'll be prompted to enter your HuggingFace credentials ([User Access Token](https://huggingface.co/docs/hub/security-tokens)). Once authenticated, your credentials will be saved locally, allowing seamless access to HuggingFace resources.","metadata":{}},{"id":"89c29ea2","cell_type":"markdown","source":"## TLDR;\nTo get started with the default configuration, you can use the following code snippet to set up the BirdSet pipeline to load the [High Sierras](https://zenodo.org/records/7525805) test dataset (10,296 samples) including a train set (5,197 samples) with matching bird classes from [xeno-canto](https://xeno-canto.org/). The total size of the dataset is 6.2GB. The samples will be provided as spectrograms with a resolution of `128x1024` pixels in batches of size `32`, the labels are one-hot encoded for a multilabel classification task. Down below you find further information on how to configure the pipeline to your needs.","metadata":{}},{"id":"9f345d07","cell_type":"code","source":"from birdset.datamodule import DatasetConfig\nfrom birdset.datamodule.birdset_datamodule import BirdSetDataModule\n\n# initiate the data module\ndm = BirdSetDataModule(\n    dataset=DatasetConfig(\n        data_dir=\"../../data_birdset/HSN\",\n        hf_name=\"research-group/BirdSet\",\n        hf_name=\"HSN\",\n        n_workers=3,\n        val_split=0.2,\n        task=\"multilabel\",\n        classlimit=500,\n        eventlimit=5,\n        sampling_rate=32000,\n    ),\n)\n# prepare the data (download dataset, ...)\ndm.prepare_data()\n# setup the dataloaders\ndm.setup(stage=\"fit\")\n# get the dataloaders\ntrain_loader = dm.train_dataloader()\n# get the first batch\nbatch = next(iter(train_loader))\n# get shape of the batch\nprint(batch[\"input_values\"].shape)\nprint(batch[\"labels\"].shape)\n# batch","metadata":{"ExecuteTime":{"end_time":"2024-05-27T10:25:54.876793900Z","start_time":"2024-05-27T10:25:54.663332400Z"}},"outputs":[],"execution_count":null},{"id":"5584fa7f","cell_type":"code","source":"from lightning import Trainer\n\nmin_epochs = 1\nmax_epochs = 5\ntrainer = Trainer(\n    min_epochs=min_epochs,\n    max_epochs=max_epochs,\n    accelerator=\"gpu\",\n    devices=1,\n    fast_dev_run=True,\n)","metadata":{},"outputs":[],"execution_count":null},{"id":"4481caa6","cell_type":"code","source":"from birdset.modules.base_module import BaseModule\n\nmodel = BaseModule(\n    len_trainset=dm.len_trainset,\n    task=dm.task,\n    batch_size=dm.train_batch_size,\n    num_epochs=max_epochs,\n)","metadata":{"ExecuteTime":{"end_time":"2024-05-27T10:27:01.894487400Z","start_time":"2024-05-27T10:27:01.330278300Z"}},"outputs":[],"execution_count":null},{"id":"93cc4646","cell_type":"code","source":"trainer.fit(model, dm)","metadata":{},"outputs":[],"execution_count":null},{"id":"8c79c1fe0cc397d6","cell_type":"code","source":"trainer.test(model, dm)","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"id":"73be30bd-772f-407f-9925-ef247dbc8219","cell_type":"markdown","source":"## Configuration of BirdSet Data Pipeline\n\nThe BirdSet Data Pipeline offers a robust and flexible configuration system, primarily designed to streamline the process of setting up your data processing environment. While this notebook presents hardcoded configurations for simplicity, it's important to note that these settings can be dynamically managed using advanced configuration tools like Hydra. Hydra is a powerful utility that enables flexible and scalable configuration management, allowing you to adapt the pipeline settings to various environments or use cases seamlessly. For an in-depth understanding of Hydra, consider visiting [Hydra's official documentation](https://hydra.cc/docs/intro).\n\nTipp! Detailed information is provided in the docstrings of the classes and functions. You can access them by hovering over the class or function name in your IDE or by opening the source file in a text editor.","metadata":{}},{"id":"7ad76228","cell_type":"markdown","source":"### BirdSetDataModule\nThe `BirdSetDataModule` is a [PyTorch Lightning DataModule](https://lightning.ai/docs/pytorch/stable/data/datamodule.html#lightningdatamodule) that encapsulates the entire data pipeline. It inherits from `BaseDataModuleHF` which is a base class for all DataModules that use [HuggingFace datasets libary](https://huggingface.co/docs/datasets/index).\n\nTo initialize the `BirdSetDataModule`, you need to provide the following parameters:\n\n```python\nfrom src.datamodule.birdset_datamodule import BirdSetDataModule\n\ndata_module = BirdSetDataModule(\n    dataset=dataset_config, #dataset (DatasetConfig): The configuration for the dataset.\n    loaders=loaders_config, #loaders (LoadersConfig): The configuration for the loaders.\n    transforms=transforms, #transforms (BirdSetTransformsWrapper): The transforms to be applied to the data.\n    mapper=mapper #mapper (XCEventMapping): The mapping for the events.\n)\n```\n\nWe will now walk through each of these parameters to understand their functionality and configuration.\n","metadata":{}},{"id":"5158f248-890f-47d0-8b16-68758a314685","cell_type":"markdown","source":"### 1. Dataset Configuration\n\nConfiguring the dataset is the first step in configuring the BirdSet data pipeline, here you specify which dataset you want to load, how many classes it has, and how the data is split into train, validation, and test sets. The `DatasetConfig` class is used to configure the dataset.","metadata":{}},{"id":"33cca9fe-2ac0-4418-93d1-e8ad5fd37786","cell_type":"code","source":"from birdset.datamodule import DatasetConfig\n\ndataset_config = DatasetConfig(\n    data_dir='../../data_birdset/HSN',\n    hf_name-research-group/BirdSet',\n    hf_name='HSN',\n    n_workers=1,\n    val_split=0.2,\n    task=\"multilabel\",\n    subset=None,\n    sampling_rate=32000,\n    class_weights_sampler=None,\n    classlimit=500,\n    eventlimit=5,\n)","metadata":{"ExecuteTime":{"end_time":"2024-05-23T08:55:39.449314Z","start_time":"2024-05-23T08:55:39.380976700Z"}},"outputs":[],"execution_count":null},{"id":"e8ea4694-e2e8-4b71-a2ad-6b004850f7e1","cell_type":"markdown","source":"Here's a brief overview of the parameters used in the `DatasetConfig` class:\n\n- `data_dir`: Specifies the directory where the dataset files are stored. **Important**: The dataset uses a lot of disk space, so make sure you have enough storage available.\n- `hf_path`: The path to the dataset stored on HuggingFace.\n- `hf_name`: The name of the dataset on HuggingFace.\n- `seed`: A seed value for ensuring reproducibility across runs.\n- `n_classes`: The total number of distinct classes in the dataset.\n- `n_workers`: The number of worker processes used for data loading.\n- `val_split`: The proportion of the dataset reserved for validation.\n- `task`: Defines the type of task (e.g., 'multilabel' or 'multiclass').\n- `sampling_rate`: The sampling rate for audio data processing.\n- `class_weights_sampler`: Indicates whether to use class weights in the sampler for handling imbalanced datasets.\n- `class_limit`: The maximum number of samples per class.\n- `eventlimit`: Defines the maximum number of audio events processed per audio file, capping the quantity to ensure balance across files.","metadata":{"tags":[]}},{"id":"fdc2a0b4-c98c-4e95-9b51-3e7f17d038eb","cell_type":"markdown","source":"#### Important Note:\n- The `class_weights_loss` parameter is currently deprecated and only implemented for focal loss. It's recommended to utilize the `class_weights_sampler` instead, as it has shown to yield favorable results, particularly as evidenced by the winner of the [BirdCLEF 2023](https://www.kaggle.com/competitions/birdclef-2023) challenge.","metadata":{}},{"id":"5f58a9d4-aabd-43b7-b6f6-e7acbb4cefeb","cell_type":"markdown","source":"Selecting appropriate values for these parameters is crucial, as they directly influence the efficiency of the training process and the overall performance of the model.","metadata":{}},{"id":"5734f65a-8fb3-4468-8fa2-7bccff9f8097","cell_type":"markdown","source":"### 2. Dataloader Configuration\n\nOnce the dataset is configured, the next crucial step is setting up the data loaders. Data loaders are pivotal in efficiently feeding data into the model during both the training and testing phases. They manage the data flow, ensuring that the model is supplied with a consistent stream of data batches. In this section, we'll use the `LoaderConfig` and `LoadersConfig` classes to define different configurations for the training and testing data loaders.","metadata":{}},{"id":"5a49150e-4ce1-4ed4-abc7-9d376eb9c3b4","cell_type":"code","source":"from birdset.datamodule import LoaderConfig, LoadersConfig\n\n# Configuration for the training data loader\ntrain_loader_config = LoaderConfig(\n    batch_size=32,\n    shuffle=True,\n    num_workers=8,\n    pin_memory=False,\n    drop_last=True,\n    persistent_workers=False,\n    # prefetch_factor=None,\n)\n\n# Configuration for the testing data loader\ntest_loader_config = LoaderConfig(\n    batch_size=32,\n    shuffle=False,\n    num_workers=8,\n    pin_memory=False,\n    drop_last=False,\n    persistent_workers=False,\n    # prefetch_factor=None,\n)\n\n# Aggregating the loader configurations\nloaders_config = LoadersConfig(\n    train=train_loader_config,\n    valid=test_loader_config,\n    test=test_loader_config,\n)","metadata":{"ExecuteTime":{"end_time":"2024-05-23T08:55:43.941782600Z","start_time":"2024-05-23T08:55:43.915891100Z"}},"outputs":[],"execution_count":null},{"id":"3ea42652-6857-4d0e-be29-8f50928aad0b","cell_type":"markdown","source":"Here's a brief overview of the parameters used in the `LoaderConfig` class:\n\n- `batch_size`: Specifies the number of samples contained in each batch. This is a crucial parameter as it impacts memory utilization and model performance.\n- `shuffle`: Determines whether the data is shuffled at the beginning of each epoch. Shuffling is typically used for training data to ensure model robustness and prevent overfitting.\n- `num_workers`: Sets the number of subprocesses to be used for data loading. More workers can speed up the data loading process but also increase memory consumption.\n- `pin_memory`: When set to `True`, enables the DataLoader to copy Tensors into CUDA pinned memory before returning them. This can lead to faster data transfer to CUDA-enabled GPUs.\n- `drop_last`: Determines whether to drop the last incomplete batch. Setting this to `True` is useful when the total size of the dataset is not divisible by the batch size.\n- `persistent_workers`: Indicates whether the data loader should keep the workers alive for the next epoch. This can improve performance at the cost of memory.\n- `prefetch_factor`: Defines the number of samples loaded in advance by each worker. This parameter is commented out here and can be adjusted based on specific requirements.\n\nProper configuration of the data loaders is essential as it directly influences the efficiency of the training process, hardware resource utilization, and ultimately, the performance of the model.","metadata":{}},{"id":"09a2390d-937a-46c8-8829-94ff3eca2b28","cell_type":"markdown","source":"### 3. Configuration of TransformationsWrapper\n\nTransformations play a critical role in the data preparation process within the BirdSet data pipeline. These operations, applied before the data is fed into the model, encompass a range of augmentation techniques designed to regularize the model and prevent overfitting. Properly configured transformations not only enhance the diversity and quality of the training data but also help the model generalize better to new, unseen data.\n\nIn the BirdSet framework, transformations are meticulously orchestrated through the `BirdSetTransformsWrapper` class. This wrapper acts as a comprehensive interface for defining and applying various transformations and augmentations to the data. It ensures that the data is consistently and effectively transformed, aligning with the specific requirements of the model and the inherent characteristics of the dataset.\n\nBy configuring the `transforms_wrapper` using the `BirdSetTransformsWrapper` class, you gain precise control over how the data is manipulated during the preprocessing phase.\n\nTo initialize the `BirdSetTransformsWrapper`, you need to provide the following parameters:\n\n```python\nfrom src.datamodule.components.transforms import BirdSetTransformsWrapper\n\ntransforms = BirdSetTransformsWrapper(\n    task: Literal['multiclass', 'multilabel'] = \"multilabel\",\n    sampling_rate: int = 32000,\n    model_type: Literal['vision', 'waveform'] = \"waveform\",\n    spectrogram_augmentations: DictConfig = DictConfig({}),\n    waveform_augmentations: DictConfig = DictConfig({}),\n    decoding: EventDecoding | None = None,\n    feature_extractor: DefaultFeatureExtractor = DefaultFeatureExtractor(),\n    max_length: int = 5,\n    nocall_sampler: DictConfig = DictConfig({}),\n    preprocessing: PreprocessingConfig = PreprocessingConfig()\n)\n```\n\nWe will go through each of these parameters to understand their functionality and configuration.","metadata":{}},{"id":"239b8432","cell_type":"markdown","source":"#### 3.1 Dataset and Model specific parameters\nThe following parameters ensure that the transformations are tailored to the specific requirements of the dataset and the model:\n\n- `task`: Defines the type of task (e.g., 'multilabel' or 'multiclass').\n- `sampling_rate`: The sampling rate for audio data processing.\n- `model_type`: Specifies the type of model (e.g., 'vision' or 'waveform'). In case of a vison model, the input data is expected to be a spectrogram, while for a waveform model, the input data is the raw audio waveform.\n- max_length: The maximum length of the audio files in seconds.","metadata":{}},{"id":"a16af9b0-b4da-47a3-a551-433009c4739d","cell_type":"markdown","source":"#### 3.2 Augmentations\n\nAugmentations are powerful techniques applied to the data to introduce diversity and variability. They are particularly useful in audio and signal processing to enhance the robustness of models against variations in input data. In the BirdSet framework, you can configure waveform and spectrogram augmentations as follows:","metadata":{}},{"id":"306dfe3d-b6a1-41e9-ab21-ac551e501dd9","cell_type":"markdown","source":"**Waveform Augmentations**\n\nThese augmentations are applied directly to the audio waveform. In BirdSet, you can use any waveform augmentation technique as long as it can be composed by the [torch-audiomentations Compose function](https://github.com/asteroid-team/torch-audiomentations/blob/main/torch_audiomentations/core/composition.py). You can add waveform augmentations as follows:","metadata":{"tags":[]}},{"id":"263b67d3-0a2f-4ccf-b8eb-d55af0236ac9","cell_type":"code","source":"from torch_audiomentations import AddColoredNoise, PitchShift\n\nwaveform_augmentation = {\n    \"colored_noise\": AddColoredNoise(\n        p=0.2, min_snr_in_db=3.0, max_snr_in_db=30.0, min_f_decay=-2.0, max_f_decay=2.0\n    ),\n    \"pitch_shift\": PitchShift(\n        p=0.2,\n        sample_rate=32000,\n        min_transpose_semitones=-4.0,\n        max_transpose_semitones=4.0,\n    ),\n}","metadata":{"ExecuteTime":{"end_time":"2024-05-23T08:55:48.164688100Z","start_time":"2024-05-23T08:55:48.125622700Z"}},"outputs":[],"execution_count":null},{"id":"47c94eee-4cd7-49eb-b277-1902b48ced9e","cell_type":"markdown","source":"In this example:\n- `colored_noise`: Adds colored noise to the audio signal to simulate various real-world noise conditions.\n- `pitch_shift`: Alters the pitch of the audio signal, which is useful for simulating different tonal variations.","metadata":{}},{"id":"daf77953-a58d-4e81-8e50-2d15be2ca0bf","cell_type":"markdown","source":"**Spectrogram Augmentations**\n\nThese augmentations are applied to the spectrogram representation of the audio. In BirdSet, you can use any spectrogram augmentation technique as long as it can be composed by the [torchvision Compose function](https://pytorch.org/vision/stable/generated/torchvision.transforms.Compose.html). You can add spectrogram augmentations as follows:","metadata":{}},{"id":"58e9af4b-879b-4095-92da-99a5e0992c9e","cell_type":"code","source":"from torchvision.transforms import RandomApply\nfrom torchaudio.transforms import TimeMasking, FrequencyMasking\n\nspectrogram_augmentations = {\n    \"time_masking\": RandomApply(\n        [TimeMasking(time_mask_param=100, iid_masks=True)], p=0.3\n    ),\n    \"frequency_masking\": RandomApply(\n        [FrequencyMasking(freq_mask_param=100, iid_masks=True)], p=0.5\n    ),\n}","metadata":{"ExecuteTime":{"end_time":"2024-05-23T08:55:50.790354600Z","start_time":"2024-05-23T08:55:50.746233300Z"}},"outputs":[],"execution_count":null},{"id":"e8aca7d4-609d-4519-8675-ee7a35fa52c2","cell_type":"markdown","source":"In this example:\n- `time_masking`: Randomly masks a sequence of consecutive time steps in the spectrogram, helping the model become invariant to small temporal shifts.\n- `frequency_masking`: Randomly masks a sequence of consecutive frequency channels, encouraging the model to be robust against frequency variations.","metadata":{}},{"id":"04268897-f6f4-4488-9d85-12cf23c15baf","cell_type":"markdown","source":"Configuring the augmentations correctly is crucial as they directly influence the model's ability to learn from a diverse set of data representations, ultimately leading to better generalization and performance.","metadata":{}},{"id":"08ee00cb-31db-4f66-883f-623791389e06","cell_type":"markdown","source":"#### 3.3 Decoding\n\nDecoding is a process, that converts the (compressed) data into a format that can be directly used by the model. In the BirdSet framework, we use the `EventDecoding` class by default. It is designed for preprocessing audio files in the context of event detection tasks. Its primary function is to ensure that each audio segment fed into the model is not only in the correct format, but also conditioned to improve the model's ability to identify and understand different audio events. Decoding is configured as follows:","metadata":{}},{"id":"b7e2b022-ae03-4940-99de-3bf4823ce506","cell_type":"code","source":"from birdset.datamodule.components import EventDecoding\n\ndecoding = EventDecoding(\n    min_len=1.0,\n    max_len=5.0,\n    sampling_rate=32000,\n    extension_time=8,\n    extracted_interval=5,\n)","metadata":{"ExecuteTime":{"end_time":"2024-05-23T08:55:53.127627600Z","start_time":"2024-05-23T08:55:53.096907600Z"}},"outputs":[],"execution_count":null},{"id":"8ad60adc-3a7e-4506-9c7e-529074359b9f","cell_type":"markdown","source":"Key Parameters:\n- `_target_`: Specifies the EventDecoding component to be used in the data processing pipeline.\n- `min_len` and `max_len`: Determine the minimum and maximum duration (in seconds) of the audio segments after decoding. These constraints ensure that each processed audio segment is of a suitable length for the model.\n- `sampling_rate`: Defines the sampling rate to which the audio should be resampled. This standardizes the input data's sampling rate, making it consistent for model processing.\n- `extension_time`: Refers to the time (in seconds) by which the duration of an audio event is extended. This parameter is crucial for ensuring that shorter audio events are sufficiently long for the model to process effectively.\n- `extracted_interval`: Denotes the fixed duration (in seconds) of the audio segment that is randomly extracted from the extended audio event.","metadata":{}},{"id":"1f6723e8-1b8c-4dcd-b55d-0e934f6f704f","cell_type":"markdown","source":"Decoding is performed on the fly, ensuring that the data fed into the model is always in the correct format, even when the source data comes in various encoded forms.","metadata":{}},{"id":"af655211-4a1a-4015-b40f-a6f6fbf2647c","cell_type":"markdown","source":"#### 3.4 Feature Extraction\n\nFeature extraction is a pivotal step in transforming raw data into a structured format that is suitable for model training. The `DefaultFeatureExtractor` in BirdSet is tailored for processing waveform data, providing a range of functionalities to prepare the data for model consumption.","metadata":{}},{"id":"bbf56c70-dfc1-4bd4-8b91-a0e40ac06380","cell_type":"code","source":"from birdset.datamodule.components import DefaultFeatureExtractor\n\nfeature_extractor = DefaultFeatureExtractor(\n    feature_size=1,\n    sampling_rate=32000,\n    padding_value=0.0,\n    return_attention_mask=False,\n)","metadata":{"ExecuteTime":{"end_time":"2024-05-23T08:55:55.685254700Z","start_time":"2024-05-23T08:55:55.665304600Z"}},"outputs":[],"execution_count":null},{"id":"1f84eaf7-0d06-4a5d-91d1-55ede47349f5","cell_type":"markdown","source":"Key Parameters:\n- `feature_size`: Determines the size of the extracted features.\n- `sampling_rate`: The sampling rate at which the audio data should be processed.\n- `padding_value`: The value used for padding shorter sequences to a consistent length.\n- `return_attention_mask`: Indicates whether an attention mask should be returned along with the processed features.","metadata":{}},{"id":"d6ae4db7-fa6a-4768-a00a-1e588d0534ac","cell_type":"markdown","source":"This component is crucial for ensuring that the input data to the model is in a consistent and processable format, catering to models that require structured input in the form of PyTorch tensors.","metadata":{}},{"id":"98891b2a","cell_type":"markdown","source":"#### 3.5 No-call Sampler\nYou can use the `NoCallMixer` to add no-call samples to the dataset. This is particularly useful for training models to recognize the absence of bird calls. The `NoCallMixer` is configured as follows:\n```python\nfrom src.datamodule.components.no_call_sampler import NoCallSampler\n\nnocall_sampler = NoCallMixer(\n    directory: str = \"path/to/no_call_samples\",\n    p: float = 0.075,\n    sampling_rate: int = 32000,\n    length: int = 5,\n    n_classes: int = 21,\n)\n```\n","metadata":{}},{"id":"2300978e","cell_type":"code","source":"# Since this would require to have the dataset downloaded (see `download_background_noise.ipynb`, we will not use this for now\nnocall_sampler = None","metadata":{"ExecuteTime":{"end_time":"2024-05-23T08:55:58.507189300Z","start_time":"2024-05-23T08:55:58.462284500Z"}},"outputs":[],"execution_count":null},{"id":"e75a91ff-07b7-48f2-a115-7a15de770c52","cell_type":"markdown","source":"#### 3.6 Configuration of Data Preprocessing\n\nData preprocessing is a fundamental step in the BirdSet data pipeline, ensuring that the raw data is adequately conditioned and transformed, making it suitable for model consumption. The `PreprocessingConfig` class allows for a detailed specification of various preprocessing parameters, each carefully selected to meet the unique demands of your dataset and model. Here's how you can configure the data preprocessing in the BirdSet pipeline:","metadata":{}},{"id":"235ed383-b28f-4575-9a8b-02fff555e812","cell_type":"code","source":"from torchaudio.transforms import Spectrogram\nfrom birdset.datamodule.components.resize import Resizer\nfrom birdset.datamodule.components.augmentations import PowerToDB\nfrom birdset.datamodule.components.transforms import PreprocessingConfig\n\n# Creating the preprocessing configuration\npreprocessing = PreprocessingConfig(\n    spectrogram_conversion=Spectrogram(\n        n_fft=1024,\n        hop_length=320,\n        power=2.0,\n    ),\n    resizer=Resizer(\n        db_scale=True,\n        target_height=None,\n        target_width=1024,\n    ),\n    dbscale_conversion=PowerToDB(),\n    normalize_spectrogram=True,\n    normalize_waveform=None,\n    mean=4.268,  # calculated on AudioSet\n    std=4.569,  # calculated on AudioSet\n)","metadata":{"ExecuteTime":{"end_time":"2024-05-23T08:56:00.635143200Z","start_time":"2024-05-23T08:56:00.610853400Z"}},"outputs":[],"execution_count":null},{"id":"0eff94aa-3191-41e8-a2ec-4393e8418c94","cell_type":"markdown","source":"Here's a brief overview of the parameters used in the `PreprocessingConfig` class:\n\n- `spectrogram_conversion`: This is an instance of the Spectrogram class from torchaudio.transforms. It is used to convert the audio waveform to a spectrogram. The parameters n_fft, hop_length, and power are used to configure the spectrogram conversion.\n\n    - `n_fft`: Th` size of the FFT, which will also determine the size of the window used for the STFT. It is set to 1024.\n    - `hop_length`: The number of samples between successive frames in the STFT. It is set to 320.\n    - `power`: The exponent for the magnitude spectrogram, e.g., 1 for energy, 2 for power, etc. It is set to 2.0.\n    - `resizer`: This is an instance of the Resizer class from src.datamodule.components.resize. It is used to resize the spectrogram. The parameters db_scale and target_width are used to configure the resizing.\n\n- `db_scale`: If set to True, the spectrogram is converted to dB scale. It is set to True.\n- `target_height`: The target height for the resized spectrogram. It is not set in this case.\n- target_width: The target width for the resized spectrogram. It is set to 1024.\n- `dbscale_conversion`: This is an instance of the PowerToDB class from src.datamodule.components.augmentations. It is used to convert the spectrogram to a dB scale.\n\n- `normalize_spectrogram`: If set to True, the spectrogram is normalized. It is set to True.\n\n- `normalize_waveform`: If set to a value, the audio waveform is normalized. It is not set in this case.\n\n- ``mean``: The mean value used for normalization. It is set to 4.268, which is calculated on AudioSet.\n\n- `std`: The standard deviation used for normalization. It is set to 4.569, which is calculated on AudioSet.\n\n\nBy accurately configuring these preprocessing parameters, you ensure that the input data to the model is standardized and optimized for the learning process, which is essential for achieving high performance.","metadata":{}},{"id":"d993142c-54a2-43a0-9e4e-b977571b03fd","cell_type":"markdown","source":"#### 3.7 Initiating the BirdSetTransformsWrapper\n","metadata":{}},{"id":"267876cf","cell_type":"code","source":"from birdset.datamodule.components.transforms import BirdSetTransformsWrapper\n\ntransforms = BirdSetTransformsWrapper(\n    task=\"multilabel\",\n    sampling_rate=32000,\n    model_type=\"vision\",\n    spectrogram_augmentations=spectrogram_augmentations,\n    waveform_augmentations=waveform_augmentation,\n    decoding=decoding,\n    feature_extractor=feature_extractor,\n    max_length=5,\n    nocall_sampler=nocall_sampler,\n    preprocessing=preprocessing,\n)","metadata":{"ExecuteTime":{"end_time":"2024-05-23T08:56:03.756435400Z","start_time":"2024-05-23T08:56:03.719469900Z"}},"outputs":[],"execution_count":null},{"id":"fb170e58-deed-4b52-ba18-160681b5dba0","cell_type":"markdown","source":"### 4. Configuration of Event Mappings\n\nEvent mapping plays a pivotal role in the data pipeline, serving as the bridge between raw dataset events and the structured input required by the model. This process ensures that each event in the dataset is accurately represented and can be effectively utilized during model training. By default, we use [bambird](https://www.sciencedirect.com/science/article/pii/S1574954122004022?casa_token=HEbcdB5MyRMAAAAA:saYbr1WNlJTs-kAZOtzMrNt5r1sN_69E7bMjfCJu2A4zlLLFoIt-5-Cht2Wryg59851H_PWgfHzw) for event mapping, which is implemented in the `XCEventMapping` class. Within the BirdSet framework, event mappings are configured as follows:","metadata":{}},{"id":"d4f30431-6972-4536-b323-88b7378e47c7","cell_type":"code","source":"from birdset.datamodule.components import XCEventMapping\n\n# Instantiate the event mapper\nmapper = XCEventMapping(\n    biggest_cluster=True,\n    no_call=False,\n)","metadata":{"ExecuteTime":{"end_time":"2024-05-23T08:56:06.004675200Z","start_time":"2024-05-23T08:56:05.954028700Z"}},"outputs":[],"execution_count":null},{"id":"bd405878-31de-4fbc-9729-20410747c89d","cell_type":"markdown","source":"Key Parameters in Event Mapping:\n- `biggest_cluster`: If set to `True`, the mapper focuses on the biggest cluster of events, which can be particularly useful for datasets with imbalanced event distributions.\n- ``: Specifies the maximum number of events to consider. This can be used to limit the scope of the mapping, although it's usually already managed by the `DatasetConfig`.\n- `no_call`: Indicates whether 'no-call' events should be included. In this configuration, it's set to `False` as the no-call samples are handled separately by the `nocall_sampler`.","metadata":{}},{"id":"fedc44f5-852c-4231-bafd-4757ee27841f","cell_type":"markdown","source":"Properly configuring the event mappings is essential for ensuring that the model receives accurately structured and meaningful data, which is a cornerstone for effective model training and robust performance.","metadata":{}},{"id":"704cfdb6-d25c-46b7-a1ee-47d4b834e01f","cell_type":"markdown","source":"## Creating the BirdSet Datamodule\n\nThe BirdSet Datamodule plays a central role in the BirdSet data pipeline, offering streamlined handling and preprocessing of BirdSet datasets to ensure they are primed for model training. Let's delve into the setup process:","metadata":{}},{"id":"60180aa6-285c-4879-abca-e88b29578897","cell_type":"markdown","source":"### Imports\nFirst, we import the necessary modules. `BirdSetDataModule` is responsible for managing the BirdSet datasets, while the `logging` module is used for logging information during the data processing steps.","metadata":{}},{"id":"0c76242c-928d-4404-922d-30c462681bb0","cell_type":"code","source":"import logging\nimport os\n\nfrom birdset.datamodule.birdset_datamodule import BirdSetDataModule","metadata":{"ExecuteTime":{"end_time":"2024-05-23T08:56:13.339917200Z","start_time":"2024-05-23T08:56:13.290347900Z"}},"outputs":[],"execution_count":null},{"id":"29444ab9-1b1c-4ea2-b85c-c55b280f2c16","cell_type":"markdown","source":"### Creating Cache Directory\nThe cache directory is a dedicated space for storing processed data. Utilizing a cache directory can significantly expedite subsequent data loading operations by avoiding redundant data processing. Here's how to create and manage a cache directory effectively:","metadata":{}},{"id":"8c8b7b7a-2c44-40d1-a7db-1355d2aefe82","cell_type":"code","source":"# Log the absolute path of the dataset\nlogging.info(f\"Dataset path: <{os.path.abspath(dataset_config.data_dir)}>\")\n\n# Create the dataset directory if it does not exist\nos.makedirs(dataset_config.data_dir, exist_ok=True)","metadata":{"ExecuteTime":{"end_time":"2024-05-23T08:56:15.351748900Z","start_time":"2024-05-23T08:56:15.280666900Z"}},"outputs":[],"execution_count":null},{"id":"82895c4e-93ac-4590-aeb0-b0acb1f9909b","cell_type":"markdown","source":"This approach ensures:\n- Organized data management: By maintaining a structured directory for your datasets, you facilitate easier access and management of your data assets.\n- Efficient data loading: By caching processed data, subsequent loads are much faster, which is particularly beneficial when working with large datasets.\n\nBy carefully setting up the BirdSet Datamodule and managing your cache directory, you enhance the efficiency and reliability of your data pipeline, ensuring that your datasets are always ready for model training.","metadata":{}},{"id":"d764504c-f167-40b4-9297-6ca1184c5668","cell_type":"markdown","source":"### Datamodule Initialization\n\nThe `BirdSetDataModule` class plays a pivotal role in orchestrating the data pipeline. It consolidates the dataset configuration, data loaders, transformations, and event mappings into a cohesive structure, ensuring a clean and manageable workflow. Here's how the BirdSetDataModule is initialized:","metadata":{}},{"id":"d0586ee6-9cad-4270-acb9-931cf046b6ab","cell_type":"code","source":"# Initialize the BirdSetDataModule\ndatamodule = BirdSetDataModule(\n    dataset=dataset_config,\n    loaders=loaders_config,\n    transforms=transforms,\n    mapper=mapper,\n)","metadata":{"ExecuteTime":{"end_time":"2024-05-23T08:56:18.392557400Z","start_time":"2024-05-23T08:56:18.370405800Z"}},"outputs":[],"execution_count":null},{"id":"cb0982ec-199c-43bc-a2e6-8732f2356cc9","cell_type":"markdown","source":"Here's a brief overview of the parameters used in the `BirdSetDataModule` class:\n- `dataset`: The configuration settings for the dataset. It defines how the data is structured and managed.\n- `loaders`: Configuration settings for the data loaders, determining how data is batched and fed into the model.\n- `transforms`: The set of transformations and augmentations applied to the data, ensuring that it's properly conditioned for the model.\n- `mapper`: The event mapping configuration, essential for translating raw dataset events into a structured format that the model can interpret.","metadata":{}},{"id":"88ab439e-2ee1-471f-9e38-15dfbc12cdfc","cell_type":"markdown","source":"### Data Preparation\n\nThe data preparation stage is where the actual data processing takes place. This stage is critical in ensuring that the data is correctly preprocessed, structured, and ready for model training.","metadata":{}},{"id":"ee400c19-41a3-4a5e-9011-509bdd319e7b","cell_type":"code","source":"# Prepare the data for training\ndatamodule.prepare_data()","metadata":{},"outputs":[],"execution_count":null},{"id":"eb9cf461-1fb8-4bb4-8db3-f7b4b4830036","cell_type":"markdown","source":"The `prepare_data()` method encompasses various steps, including downloading the data (if not already locally available), applying the preprocessing steps defined in the transformations, and organizing the data into a format that is compatible with the model. It's a method that encapsulates the entire data preparation workflow, ensuring that the data is optimally prepared for the training process.\n\nThis methodical approach to data preparation and modularization of the data pipeline components in the BirdSet framework contributes significantly to the efficiency, maintainability, and robustness of the machine learning lifecycle.\n\n**Hint**: If you recive an error concerning a not existing huggingface dataset, please make sure you are logged in to HuggingFace (see [Log in to Huggingface](#log-in-to-huggingface)).","metadata":{}},{"id":"932e3987-0f9a-4452-8b82-54c0ea96180e","cell_type":"markdown","source":"### Datamodule Setup for Training Phase\n\nSetting up the datamodule for the training phase is a crucial step in the BirdSet data pipeline. This setup involves initializing the training and validation dataloaders, which play a vital role in supplying the model with data during the training process. The setup is performed using the `setup(stage=\"fit\")` method:","metadata":{}},{"id":"a252adb3-539e-434c-adcb-2ec2c1f7c996","cell_type":"code","source":"# Setup the datamodule for the training phase\ndatamodule.setup(stage=\"fit\")\n\n# Retrieve the training and validation dataloaders\ntrain_loader = datamodule.train_dataloader()\nvalidation_loader = datamodule.val_dataloader()","metadata":{},"outputs":[],"execution_count":null},{"id":"2048c240-bac0-465e-836f-1d9718c3a1ce","cell_type":"code","source":"# Fetch a sample batch from the training dataloader\nbatch = next(iter(train_loader))\n\n# Inspect the keys and shapes of the data in the batch\nbatch.keys(), batch[\"input_values\"].shape, batch[\"labels\"].shape","metadata":{"ExecuteTime":{"end_time":"2024-05-23T08:59:00.296838500Z","start_time":"2024-05-23T08:58:58.460319100Z"}},"outputs":[],"execution_count":null},{"id":"00066181-de6f-4980-90dc-458d6b92f27d","cell_type":"markdown","source":"This code snippet demonstrates:\n- The initialization of the training phase.\n- The retrieval of training and validation dataloaders.\n- Fetching and inspecting a sample batch from the training dataloader.\n- The shapes of `input_values` and `labels` indicate the batch size, number of channels (if applicable), and dimensions of the input data and labels, respectively.","metadata":{}},{"id":"ed731d9c-7aea-49ac-9df9-4fc61774abbc","cell_type":"markdown","source":"### Datamodule Setup for Test Phase\n\nSimilarly, the datamodule is set up for the test phase to ensure that the model can be effectively evaluated using the test data:","metadata":{}},{"id":"5db95322-ca2b-4f8f-b419-40529d7c92b5","cell_type":"code","source":"# Setup the datamodule for the test phase\ndatamodule.setup(stage=\"test\")\n\n# Retrieve the test dataloader\ntest_loader = datamodule.test_dataloader()","metadata":{"ExecuteTime":{"end_time":"2024-05-23T08:59:05.646911900Z","start_time":"2024-05-23T08:59:05.565471400Z"}},"outputs":[],"execution_count":null},{"id":"7d5ed904-f95b-4bad-a03b-531de0a67dc7","cell_type":"markdown","source":"The `setup(stage=\"test\")` method prepares the datamodule specifically for the test phase, and `test_dataloader()` retrieves the test dataloader, which is instrumental for batching and loading the test data efficiently during the model evaluation process.\n\nBy methodically setting up the datamodule for both training and test phases, you ensure that the model has access to well-prepared data, which is essential for accurate training, validation, and testing.","metadata":{}},{"id":"2567af44-494a-4547-b57b-9b487e57a520","cell_type":"markdown","source":"### Usage in TensorFlow\n\nUtilizing the BirdSet datamodule in a TensorFlow environment involves integrating the prepared dataloaders with TensorFlow's training and evaluation workflows. This integration ensures that the data is fed into TensorFlow models efficiently and in a format that TensorFlow can process. Here's how you can set up the BirdSet datamodule for TensorFlow compatibility:","metadata":{}},{"id":"ce1c99c0-201a-4cae-9e9e-d4c6f9512fed","cell_type":"code","source":"# Setup the datamodule for the training phase\ndatamodule.setup(stage=\"fit\")\n\n# Retrieve the training and validation datasets\ntrain_loader = datamodule.train_dataset\nvalidation_loader = datamodule.val_dataset\n\n# Setup the datamodule for the test phase\ndatamodule.setup(stage=\"test\")\n\n# Retrieve the test dataset\ntest_loader = datamodule.test_dataset","metadata":{"ExecuteTime":{"end_time":"2024-05-23T08:59:08.064567700Z","start_time":"2024-05-23T08:59:07.871744300Z"}},"outputs":[],"execution_count":null},{"id":"d8bda355-34f2-42b2-b57a-b6112f48f49c","cell_type":"markdown","source":"#### Key Considerations:\n- `train_dataset`, `validation_dataset`, and `test_dataset` are the datasets prepared by the BirdSet datamodule, ready to be used in TensorFlow's training and evaluation routines.\n- It's important to ensure that these datasets are in a format compatible with TensorFlow. This might involve additional steps such as converting the data to `tf.data.Dataset` objects or applying necessary transformations to align with TensorFlow's data handling mechanisms.\n- More information on this integration process can be found in [HuggingFace's documentation](https://huggingface.co/docs/datasets/use_with_tensorflow).\n\nBy following these steps, you can leverage the robust data preprocessing and management capabilities of the BirdSet datamodule within a TensorFlow environment, facilitating an efficient and streamlined model training and evaluation process.","metadata":{}}]}