{"metadata":{"kernelspec":{"display_name":"Python3 (dbd-datapipeline-env)","language":"python","name":"dbd-datapipeline-env"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.12"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":91844,"databundleVersionId":11361821,"sourceType":"competition"}],"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# BirdSet Tutorial Augmentations [Copied from BirdSet Repository](https://github.com/DBD-research-group/BirdSet)","metadata":{}},{"cell_type":"markdown","source":"- Augmentations are done on-the-fly with **set_transform** \n- Set __getitem__ return format using this transform. The transform is applied on-the-fly on batches when __getitem__ is called. As set_format(), this can be reset using reset_format().\n- For spectrogram augmentations we use torchaudio (https://pytorch.org/audio/stable/transforms.html)\n- For waveform augmentations we use torch-audiomentations (https://github.com/asteroid-team/torch-audiomentations)","metadata":{}},{"cell_type":"markdown","source":"# Imports","metadata":{"vscode":{"languageId":"plaintext"}}},{"cell_type":"code","source":"from datasets import load_dataset\n\nimport librosa\nimport matplotlib.pyplot as plt\n\nimport torch\nfrom torchaudio import transforms\n\nimport IPython.display as ipd","metadata":{"vscode":{"languageId":"plaintext"},"pycharm":{"is_executing":true}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#### Augmentation module for our datapipeline","metadata":{}},{"cell_type":"code","source":"import sys\n\nsys.path.append(\"/mnt/home/rheinrich/deep_bird_detect/datapipeline/Bird2Vec\")\n\nfrom birdset.augmentations import AudioAugmentor","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Create a Huggingface dataset","metadata":{}},{"cell_type":"markdown","source":"#### For simplicity, we use the ESC50 dataset","metadata":{}},{"cell_type":"code","source":"esc = load_dataset(\"ashraq/esc50\", split=\"train\")","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#### Define a function for normalizing the audio signals","metadata":{}},{"cell_type":"markdown","source":"Here, however, we would first have to calculate the mean and standard deviation for the entire training data set!","metadata":{}},{"cell_type":"code","source":"def normalize_audio(audio_tensor, mean, std):\n    return (audio_tensor - mean) / std","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#### Define a general preprocessing function","metadata":{}},{"cell_type":"markdown","source":"In this function we can combine data augmentations, normalization, etc.","metadata":{}},{"cell_type":"code","source":"def preprocess(\n    waveform,\n    use_spectrogram,\n    spectrogram_augmentations=None,\n    waveform_augmentations=None,\n    n_fft=1024,\n    hop_length=512,\n    n_mels=None,\n):\n    audio_augmentor = AudioAugmentor(\n        sample_rate=waveform[\"sampling_rate\"],\n        use_spectrogram=use_spectrogram,\n        spectrogram_augmentations=spectrogram_augmentations,\n        waveform_augmentations=waveform_augmentations,\n        n_fft=n_fft,\n        hop_length=hop_length,\n        n_mels=n_mels,\n        db_scale=True,\n    )\n    audio_augmented = audio_augmentor.combined_augmentations(waveform[\"array\"])\n\n    # audio_augmented = normalize_audio(audio_augmented)\n    return audio_augmented","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Waveform data augmentations","metadata":{}},{"cell_type":"markdown","source":"#### Create a dictionary which defines the augmentation techniques to be used as well as their respective parameters","metadata":{}},{"cell_type":"code","source":"# path to the aufio files which are used for creating background noise\nspeech_command_path = (\n    \"/mnt/home/rheinrich/deep_bird_detect/datapipeline/speech_command_dataset\"\n)\n\n# dictionary which defines the waveform data augmentations to be used\nwaveform_augmentations_dict = {\n    \"colored_noise\": {\n        \"prob\": 0.5,\n        \"min_snr_in_db\": 3.0,\n        \"max_snr_in_db\": 30.0,\n        \"min_f_decay\": -2.0,\n        \"max_f_decay\": 2.0,\n    },\n    \"background_noise\": {\n        \"background_paths\": speech_command_path,\n        \"prob\": 0.5,\n        \"min_snr_in_db\": 3.0,\n        \"max_snr_in_db\": 30.0,\n    },\n    \"pitch_shift\": {\n        \"prob\": 0.5,\n        \"min_transpose_semitones\": -4.0,\n        \"max_transpose_semitones\": 4.0,\n    },\n    \"time_mask\": {\"prob\": 0.5, \"min_band_part\": 0.0, \"max_band_part\": 0.5},\n    \"time_stretch\": {\"prob\": 0.5, \"min_rate\": 0.8, \"max_rate\": 1.25},\n}","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#### Create a function that applies our prepocessing and can be passed to the set_transform function of the dataset","metadata":{}},{"cell_type":"code","source":"def train_transforms_waveform(examples):\n    examples[\"audio_array_preprocessed\"] = [\n        preprocess(\n            waveform=audio,\n            waveform_augmentations=waveform_augmentations_dict,\n            use_spectrogram=False,\n        )\n        for audio in examples[\"audio\"]\n    ]\n    return examples","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"code","source":"esc.set_transform(train_transforms_waveform)","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Results","metadata":{}},{"cell_type":"markdown","source":"#### waveform audio data without augmentations","metadata":{}},{"cell_type":"code","source":"waveform = esc[101][\"audio\"][\"array\"]\nsr = esc[101][\"audio\"][\"sampling_rate\"]\nipd.Audio(data=waveform, rate=sr)","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#### waveform audio data with augmentations - nice, the augmentations are audible!","metadata":{}},{"cell_type":"code","source":"waveform_augmented = esc[101][\"audio_array_preprocessed\"]\nsr = esc[101][\"audio\"][\"sampling_rate\"]\nipd.Audio(data=waveform_augmented, rate=sr)","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Spectrogram and waveform data augmentations combined","metadata":{}},{"cell_type":"markdown","source":"#### Create a HuggingFace dataset (ESC50)","metadata":{}},{"cell_type":"code","source":"esc = load_dataset(\"ashraq/esc50\", split=\"train\")","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#### Create dictionaries which define the data augmentation techniques to be used as well as their respective parameters","metadata":{}},{"cell_type":"code","source":"# path to the aufio files which are used for creating background noise\nspeech_command_path = (\n    \"/mnt/home/rheinrich/deep_bird_detect/datapipeline/speech_command_dataset\"\n)\n\n# dictionary which defines the waveform data augmentations to be used\nwaveform_augmentations_dict = {\n    \"colored_noise\": {\n        \"prob\": 0.5,\n        \"min_snr_in_db\": 3.0,\n        \"max_snr_in_db\": 30.0,\n        \"min_f_decay\": -2.0,\n        \"max_f_decay\": 2.0,\n    },\n    \"background_noise\": {\n        \"background_paths\": speech_command_path,\n        \"prob\": 0.5,\n        \"min_snr_in_db\": 3.0,\n        \"max_snr_in_db\": 30.0,\n    },\n    \"pitch_shift\": {\n        \"prob\": 0.5,\n        \"min_transpose_semitones\": -4.0,\n        \"max_transpose_semitones\": 4.0,\n    },\n}","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# dictionary which defines the spectrogram data augmentations to be used\nspectrogram_augmentations_dict = {\n    \"time_masking\": {\"time_mask_param\": 100, \"prob\": 0.5},\n    \"frequency_masking\": {\"freq_mask_param\": 100, \"prob\": 0.5},\n    \"time_stretch\": {\n        \"prob\": 0.33,\n        \"min_rate\": 0.8,\n        \"max_rate\": 1.25,\n    },\n}","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#### Create a function that applies our prepocessing and can be passed to the set_transform function of the dataset","metadata":{}},{"cell_type":"code","source":"def train_transforms_with_spec(examples):\n    examples[\"audio_array_preprocessed\"] = [\n        preprocess(\n            waveform=audio,\n            use_spectrogram=True,\n            spectrogram_augmentations=spectrogram_augmentations_dict,\n            waveform_augmentations=waveform_augmentations_dict,\n        )\n        for audio in examples[\"audio\"]\n    ]\n    return examples","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"code","source":"esc.set_transform(train_transforms_with_spec)","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Results","metadata":{}},{"cell_type":"code","source":"spectrogram = esc[101][\"audio_array_preprocessed\"]","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#### spectrogram with augmentations","metadata":{}},{"cell_type":"code","source":"plt.imshow(spectrogram.squeeze().numpy())","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#### Convert the spectrogram back to waveform audio data","metadata":{}},{"cell_type":"markdown","source":"The GriffinLim transformation converts power spectrograms back to waveform.","metadata":{}},{"cell_type":"markdown","source":"If Mel-scale spectrograms are used instead, the InverseMelScale transformation has to be applied first, see https://pytorch.org/audio/stable/transforms.html.","metadata":{}},{"cell_type":"code","source":"transform_to_waveform = transforms.GriffinLim(n_fft=1024, hop_length=512)","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Note: since the spectrogram was converted to decibel (dB) units in the AudioAugmentor, we also need to invert the dB-scale transformation.","metadata":{}},{"cell_type":"code","source":"spectrogram_unscaled = spectrogram.numpy()\nspectrogram_unscaled = librosa.db_to_power(spectrogram_unscaled)\nspectrogram_unscaled = torch.from_numpy(spectrogram_unscaled)\nwaveform_from_spectrogram = transform_to_waveform(spectrogram_unscaled)","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#### the waveform obtained from the augmented spectrogram.","metadata":{}},{"cell_type":"code","source":"sr = esc[101][\"audio\"][\"sampling_rate\"]\nipd.Audio(data=waveform_from_spectrogram, rate=sr)","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Create a PyTorch dataloader from the HuggingFace dataset","metadata":{}},{"cell_type":"code","source":"from torch.utils.data import DataLoader\nfrom torch.utils.data.dataloader import default_collate","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def collate_batch(batch):\n    input_features = [x[\"audio_array_preprocessed\"] for x in batch]\n    input_features = torch.cat(input_features, 0)\n    targets = [x[\"target\"] for x in batch]\n    targets = torch.Tensor(targets)\n    return input_features, targets","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"code","source":"dataloader = DataLoader(esc, batch_size=5, shuffle=True, collate_fn=collate_batch)","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"code","source":"for i in dataloader:\n    print(i)\n    break","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{},"outputs":[],"execution_count":null}]}