![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4992459%2F7d31650115ae805d4475181a370afc17%2FNot%20enough%20minerals(1).png?generation=1683972145458317&alt=media\")
\n What, you didn't hear that? But I think I did.
\n\n\n
\n What, you didn't hear that? But I think I did.
Although given a good GPU, the bottleneck on Kaggle has been the CPU since it is limited to two cores. In the following I will go through all the things that I found out that can speed up the process and some pitfalls to avoid.
\nThe GPU utilization is around 80% when training on the entire EfficientNetV2 (slightly modified). If freezing the backbone and trained only on the last few layers, it drops down to 20~30%.
\n\n\n
\n GPU utilization when trained on full model.
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4992459%2Fb092ee49871ac9063bc8fd192a13fae9%2FUntitled.png?generation=1683972435356362&alt=media\")
\nThere are three stages that the audio goes through before being fed to the model:
We'll discuss how to save time in each of these stages.
\nI thought this should be mentioned first since this seems to be the fastest approach.
\nUse another script to preprocess the entire dataset and save the computed spectrogram as an image. Although in the training process it still needs to be loaded and decoded, because of the small file size it should still be quicker than dealing with audio.
\nThe main concern I have here is that some might mix audio by summing up images. Mixing audio isn’t as simple as summing up the spectrogram so I wouldn’t say it is an ideal augmentation, but if it works then who cares?
\nMost of the other issues mentioned above can be resolved if not strictly using an image format. You can use a numpy array instead and scale it to uint8, but the speed difference still needs to be tested.
\n\n\n\n
\n A spectrogram in 4 channels(CMYK)
tl;dr: Presplit audio into segments is the fastest; If not applicable, then use num_frames and frame_offset arguments.
Often we only need a small segment of the audio but the normal torchaudio.load() just loads and decodes the entire file. Write another script to split and save the audio first.
Maybe you need every possible sample in your training data, then num_frames and frame_offset arguments are extremely helpful since it only loads and decodes a part of the data.
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4992459%2F41b9c9ab540836207676db3b7406070f%2FUntitled.png?generation=1683972254691424&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4992459%2F7e0e28bbe89108afdaf8cac7b0d72b93%2FUntitled.jpeg?generation=1683972579292442&alt=media\")
num_frames and frame_offset.\n\n\n
\n
\n Comparison of how much time needed to load and decode an audio file
tl;dr: Convert .ogg to 16-bit .wav files.
\nI found this to be the most time consuming part. Decoding takes a lot of cpu and unlike video decoding, there is no way to decode audio on the gpu. If we can save it in a decoded state, such as .wav, then it will save tons of time.
\nThe best solution here is to convert to .wav files, but you might have already tried this before and realized that the dataset is too large. The issue here is that the default of torchaudio.save() uses 32-bit bitrate which is an overkill since 16-bit is usually sufficient. The bird datasets the officials released on zenodo are also using 16-bit.
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4992459%2F0c2551105b287f850110e66c0891bd01%2FUntitled.png?generation=1683972645764760&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4992459%2Ff4ab16bf4e02a6eac74d00e2d565dd0e%2FUntitled.png?generation=1683972669076437&alt=media\")
torchaudio.save(file_path,sig,sample_rate=sr,bits_per_sample=,=)\nThe entire dataset is around 44GB which is totally acceptable and I uploaded it here so you don’t have to do it again.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4992459%2F08d373b8149d2f9a17db189dd07e7879%2FUntitled.png?generation=1683972825295270&alt=media\")
\nWhat about 8-bit .wav? Doing so does make the file size even smaller, but it comes with a drawback that it will be prone to noise. If there are noises such as rain sound in the audio, you basically can’t hear anything except huge crinkling noise.
Kaggle is quite generous with the RAM, make use of those to get a slight speed up. For example, I preloaded a set of noise files I reuse a lot in a class.
\n :\n ():\n self.noise_p = noise_p\n self.noise_list = []\n (.(base_path))\n dirname, _, filenames os.walk(base_path):\n filename tqdm(filenames):\n filename.split()[-] [,,]:\n audio_file = torchaudio.load((Path(dirname)/Path(filename)),=)\n self.noise_list.append(audio_file)\n\n ():\n \n noise_sig, noise_sr = random.choice(self.noise_list)\n ... \n sig = AudioUtil.add_noise(sig, noise_sig)\n\n sig\nWhen datasets are too large it might exceed the allocated disk size of /kaggle/working, however, you can still save up to 100GB in other directories such as /kaggle/tmp and upload it to kaggle directly in the script. A quick tutorial like this one should help.
\nLast but not least, computing spectrograms are also very cpu consuming. Fortunately, we can move it to the GPU in both tensorflow and pytorch. In pytorch you can create a mel spectorgram instance easily torchaudio.transforms.MelSpectrogram().to(device)
Just remember to send the instance to the GPU!
\nWow you made it to the end! If you found it helpful, consider adding me on linkedin to increase connections and potentially help me find a job.
", "messages": [ { "id": 2257411, "postDate": "2023-05-13T10:28:04.373Z", "content": "\n\n\n
\n What, you didn't hear that? But I think I did.
Although given a good GPU, the bottleneck on Kaggle has been the CPU since it is limited to two cores. In the following I will go through all the things that I found out that can speed up the process and some pitfalls to avoid.
\nThe GPU utilization is around 80% when training on the entire EfficientNetV2 (slightly modified). If freezing the backbone and trained only on the last few layers, it drops down to 20~30%.
\n\n\n
\n GPU utilization when trained on full model.
\nThere are three stages that the audio goes through before being fed to the model:
We'll discuss how to save time in each of these stages.
\nI thought this should be mentioned first since this seems to be the fastest approach.
\nUse another script to preprocess the entire dataset and save the computed spectrogram as an image. Although in the training process it still needs to be loaded and decoded, because of the small file size it should still be quicker than dealing with audio.
\nThe main concern I have here is that some might mix audio by summing up images. Mixing audio isn’t as simple as summing up the spectrogram so I wouldn’t say it is an ideal augmentation, but if it works then who cares?
\nMost of the other issues mentioned above can be resolved if not strictly using an image format. You can use a numpy array instead and scale it to uint8, but the speed difference still needs to be tested.
\n\n\n\n
\n A spectrogram in 4 channels(CMYK)
tl;dr: Presplit audio into segments is the fastest; If not applicable, then use num_frames and frame_offset arguments.
Often we only need a small segment of the audio but the normal torchaudio.load() just loads and decodes the entire file. Write another script to split and save the audio first.
Maybe you need every possible sample in your training data, then num_frames and frame_offset arguments are extremely helpful since it only loads and decodes a part of the data.
num_frames and frame_offset.\n\n\n
\n
\n Comparison of how much time needed to load and decode an audio file
tl;dr: Convert .ogg to 16-bit .wav files.
\nI found this to be the most time consuming part. Decoding takes a lot of cpu and unlike video decoding, there is no way to decode audio on the gpu. If we can save it in a decoded state, such as .wav, then it will save tons of time.
\nThe best solution here is to convert to .wav files, but you might have already tried this before and realized that the dataset is too large. The issue here is that the default of torchaudio.save() uses 32-bit bitrate which is an overkill since 16-bit is usually sufficient. The bird datasets the officials released on zenodo are also using 16-bit.
torchaudio.save(file_path,sig,sample_rate=sr,bits_per_sample=,=)\nThe entire dataset is around 44GB which is totally acceptable and I uploaded it here so you don’t have to do it again.
\n
\nWhat about 8-bit .wav? Doing so does make the file size even smaller, but it comes with a drawback that it will be prone to noise. If there are noises such as rain sound in the audio, you basically can’t hear anything except huge crinkling noise.
Kaggle is quite generous with the RAM, make use of those to get a slight speed up. For example, I preloaded a set of noise files I reuse a lot in a class.
\n :\n ():\n self.noise_p = noise_p\n self.noise_list = []\n (.(base_path))\n dirname, _, filenames os.walk(base_path):\n filename tqdm(filenames):\n filename.split()[-] [,,]:\n audio_file = torchaudio.load((Path(dirname)/Path(filename)),=)\n self.noise_list.append(audio_file)\n\n ():\n \n noise_sig, noise_sr = random.choice(self.noise_list)\n ... \n sig = AudioUtil.add_noise(sig, noise_sig)\n\n sig\nWhen datasets are too large it might exceed the allocated disk size of /kaggle/working, however, you can still save up to 100GB in other directories such as /kaggle/tmp and upload it to kaggle directly in the script. A quick tutorial like this one should help.
\nLast but not least, computing spectrograms are also very cpu consuming. Fortunately, we can move it to the GPU in both tensorflow and pytorch. In pytorch you can create a mel spectorgram instance easily torchaudio.transforms.MelSpectrogram().to(device)
Just remember to send the instance to the GPU!
\nWow you made it to the end! If you found it helpful, consider adding me on linkedin to increase connections and potentially help me find a job.
", "rawMarkdown": ">.png?generation=1683972145458317&alt=media)\n>*What, you didn't hear that? But I think I did.*\n\n\nAlthough given a good GPU, the bottleneck on Kaggle has been the CPU since it is limited to two cores. In the following I will go through all the things that I found out that can speed up the process and some pitfalls to avoid.\nThe GPU utilization is around 80% when training on the entire EfficientNetV2 (slightly modified). If freezing the backbone and trained only on the last few layers, it drops down to 20~30%.\n\n>\n>*GPU utilization when trained on full model.*\n\n\nThere are three stages that the audio goes through before being fed to the model:\n\n- **Load File:** Load the file from the disk to memory\n- **Decode Audio**: Audio may be in compressed format and needs to be decompressed before using. For example: .ogg is compressed, .wav is uncompressed.\n- **Compute Spectrogram**: Most models requires spectrogram as input.\n\nWe'll discuss how to save time in each of these stages.\n\n\n# 0. Another Route - save as image\n\nI thought this should be mentioned first since this seems to be the fastest approach. \n\nUse another script to preprocess the entire dataset and save the computed spectrogram as an image. Although in the training process it still needs to be loaded and decoded, because of the small file size it should still be quicker than dealing with audio. \n\n### Pros:\n\n- Dataset size is small, speeding up the process everywhere\n- Can use 4 cpu cores on Kaggle cpu notebook to preprocess\n\n### Cons:\n\n- Limits data augmentation choices\n- Quantization to uint8 loses some detail information\n- Channels limited to 1,3,4 channels\n- Hard to load segmentation of image if only a small area is needed\n\n### Thoughts:\n\nThe main concern I have here is that some might mix audio by summing up images. Mixing audio isn’t as simple as summing up the spectrogram so I wouldn’t say it is an ideal augmentation, but if it works then who cares? \n\nMost of the other issues mentioned above can be resolved if not strictly using an image format. You can use a numpy array instead and scale it to uint8, but the speed difference still needs to be tested.\n\n>\n>*A spectrogram in 4 channels(CMYK)*\n\n# 1. Load File\n\n**tl;dr:** Presplit audio into segments is the fastest; If not applicable, then use `num_frames` and `frame_offset` arguments.\n\n## Presplit into segments\n\nOften we only need a small segment of the audio but the normal `torchaudio.load()` just loads and decodes the entire file. Write another script to split and save the audio first.\n\n### Pros:\n\n- Up to 10x faster or more depending on audio length\n\n### Cons:\n\n- Slightly limits the variety of data. For example, if cut at every 5secs, you won’t have a sample that looks like the [2.5,7.5] cut.\n\n## Use num_frames and frame_offset\n\nMaybe you need every possible sample in your training data, then `num_frames` and `frame_offset` arguments are extremely helpful since it only loads and decodes a part of the data. \n\n### Pros:\n\n- Still significantly faster than loading the entire file. Speedup depends on `num_frames` and `frame_offset`.\n- No compromise on data\n\n### Cons:\n\n- Still reads all the way up to the frame_offset. So if attempting to read the last 5 secs of the audio, the load time is basically the same as loading the entire file.\n>\n>\n>*Comparison of how much time needed to load and decode an audio file*\n\n# 2. Decode Audio\n\n**tl;dr:** Convert .ogg to 16-bit .wav files.\n\nI found this to be the most time consuming part. Decoding takes a lot of cpu and unlike video decoding, there is no way to decode audio on the gpu. If we can save it in a decoded state, such as .wav, then it will save tons of time.\n\n## Convert to .wav - is it too large?\n\nThe best solution here is to convert to .wav files, but you might have already tried this before and realized that the dataset is too large. The issue here is that the default of `torchaudio.save()` uses 32-bit bitrate which is an overkill since 16-bit is usually sufficient. The bird datasets the officials released on zenodo are also using 16-bit.\n\n```python\ntorchaudio.save(file_path,sig,sample_rate=sr,bits_per_sample=16,format=\"wav\")\n```\n\nThe entire dataset is around 44GB which is totally acceptable and I [uploaded it here](https://www.kaggle.com/datasets/lhanhsin/birdclef2023-wav16) so you don’t have to do it again.\n\n\n**What about 8-bit .wav?** Doing so does make the file size even smaller, but it comes with a drawback that it will be prone to noise. If there are noises such as rain sound in the audio, you basically can’t hear anything except huge crinkling noise.\n\n## Preload to memory\n\nKaggle is quite generous with the RAM, make use of those to get a slight speed up. For example, I preloaded a set of noise files I reuse a lot in a class.\n\n```python\nclass EnvNoiseLoader:\n def __init__(self, base_path = wandb.config.noise_base_path, noise_p = wandb.config.noise_p):\n self.noise_p = noise_p\n self.noise_list = []\n print(\"Loading noise files from: {}\".format(base_path))\n for dirname, _, filenames in os.walk(base_path):\n for filename in tqdm(filenames):\n if filename.split(\".\")[-1] in [\"wav\",\"mp3\",\"ogg\"]:\n audio_file = torchaudio.load(str(Path(dirname)/Path(filename)),format=\"wav\")\n self.noise_list.append(audio_file)\n \n def add_random_noise(self,sig,verbose=False):\n # Load noise signal\n noise_sig, noise_sr = random.choice(self.noise_list)\n ... \n sig = AudioUtil.add_noise(sig, noise_sig)\n \n return sig\n```\n\n## Tip - working with large datasets\n\nWhen datasets are too large it might exceed the allocated disk size of /kaggle/working, however, you can still save up to 100GB in other directories such as /kaggle/tmp and upload it to kaggle directly in the script. A quick tutorial [like this one](https://www.kaggle.com/code/xhlulu/how-to-create-very-large-datasets-from-a-notebook/notebook) should help.\n\n# 3. Compute Spectrogram\n\nLast but not least, computing spectrograms are also very cpu consuming. Fortunately, we can move it to the GPU in both tensorflow and pytorch. In pytorch you can create a mel spectorgram instance easily `torchaudio.transforms.MelSpectrogram().to(device)`\n\nJust remember to send the instance to the GPU!\n\n# 4. Others\n\n- To utilize all CPUs, remember to set the dataloader num_workers argument to 2.\n- Set dataloader shuffle argument to False since shuffling the validation set is not needed.\n\n---\n*Wow you made it to the end! If you found it helpful, consider adding me on [linkedin](https://www.linkedin.com/in/lhanhsin/) to increase connections and potentially help me find a job.*", "votes": 3 } ], "comments": [], "raw_markdown_by_id": { "2257411": ">.png?generation=1683972145458317&alt=media)\n>*What, you didn't hear that? But I think I did.*\n\n\nAlthough given a good GPU, the bottleneck on Kaggle has been the CPU since it is limited to two cores. In the following I will go through all the things that I found out that can speed up the process and some pitfalls to avoid.\nThe GPU utilization is around 80% when training on the entire EfficientNetV2 (slightly modified). If freezing the backbone and trained only on the last few layers, it drops down to 20~30%.\n\n>\n>*GPU utilization when trained on full model.*\n\n\nThere are three stages that the audio goes through before being fed to the model:\n\n- **Load File:** Load the file from the disk to memory\n- **Decode Audio**: Audio may be in compressed format and needs to be decompressed before using. For example: .ogg is compressed, .wav is uncompressed.\n- **Compute Spectrogram**: Most models requires spectrogram as input.\n\nWe'll discuss how to save time in each of these stages.\n\n\n# 0. Another Route - save as image\n\nI thought this should be mentioned first since this seems to be the fastest approach. \n\nUse another script to preprocess the entire dataset and save the computed spectrogram as an image. Although in the training process it still needs to be loaded and decoded, because of the small file size it should still be quicker than dealing with audio. \n\n### Pros:\n\n- Dataset size is small, speeding up the process everywhere\n- Can use 4 cpu cores on Kaggle cpu notebook to preprocess\n\n### Cons:\n\n- Limits data augmentation choices\n- Quantization to uint8 loses some detail information\n- Channels limited to 1,3,4 channels\n- Hard to load segmentation of image if only a small area is needed\n\n### Thoughts:\n\nThe main concern I have here is that some might mix audio by summing up images. Mixing audio isn’t as simple as summing up the spectrogram so I wouldn’t say it is an ideal augmentation, but if it works then who cares? \n\nMost of the other issues mentioned above can be resolved if not strictly using an image format. You can use a numpy array instead and scale it to uint8, but the speed difference still needs to be tested.\n\n>\n>*A spectrogram in 4 channels(CMYK)*\n\n# 1. Load File\n\n**tl;dr:** Presplit audio into segments is the fastest; If not applicable, then use `num_frames` and `frame_offset` arguments.\n\n## Presplit into segments\n\nOften we only need a small segment of the audio but the normal `torchaudio.load()` just loads and decodes the entire file. Write another script to split and save the audio first.\n\n### Pros:\n\n- Up to 10x faster or more depending on audio length\n\n### Cons:\n\n- Slightly limits the variety of data. For example, if cut at every 5secs, you won’t have a sample that looks like the [2.5,7.5] cut.\n\n## Use num_frames and frame_offset\n\nMaybe you need every possible sample in your training data, then `num_frames` and `frame_offset` arguments are extremely helpful since it only loads and decodes a part of the data. \n\n### Pros:\n\n- Still significantly faster than loading the entire file. Speedup depends on `num_frames` and `frame_offset`.\n- No compromise on data\n\n### Cons:\n\n- Still reads all the way up to the frame_offset. So if attempting to read the last 5 secs of the audio, the load time is basically the same as loading the entire file.\n>\n>\n>*Comparison of how much time needed to load and decode an audio file*\n\n# 2. Decode Audio\n\n**tl;dr:** Convert .ogg to 16-bit .wav files.\n\nI found this to be the most time consuming part. Decoding takes a lot of cpu and unlike video decoding, there is no way to decode audio on the gpu. If we can save it in a decoded state, such as .wav, then it will save tons of time.\n\n## Convert to .wav - is it too large?\n\nThe best solution here is to convert to .wav files, but you might have already tried this before and realized that the dataset is too large. The issue here is that the default of `torchaudio.save()` uses 32-bit bitrate which is an overkill since 16-bit is usually sufficient. The bird datasets the officials released on zenodo are also using 16-bit.\n\n```python\ntorchaudio.save(file_path,sig,sample_rate=sr,bits_per_sample=16,format=\"wav\")\n```\n\nThe entire dataset is around 44GB which is totally acceptable and I [uploaded it here](https://www.kaggle.com/datasets/lhanhsin/birdclef2023-wav16) so you don’t have to do it again.\n\n\n**What about 8-bit .wav?** Doing so does make the file size even smaller, but it comes with a drawback that it will be prone to noise. If there are noises such as rain sound in the audio, you basically can’t hear anything except huge crinkling noise.\n\n## Preload to memory\n\nKaggle is quite generous with the RAM, make use of those to get a slight speed up. For example, I preloaded a set of noise files I reuse a lot in a class.\n\n```python\nclass EnvNoiseLoader:\n def __init__(self, base_path = wandb.config.noise_base_path, noise_p = wandb.config.noise_p):\n self.noise_p = noise_p\n self.noise_list = []\n print(\"Loading noise files from: {}\".format(base_path))\n for dirname, _, filenames in os.walk(base_path):\n for filename in tqdm(filenames):\n if filename.split(\".\")[-1] in [\"wav\",\"mp3\",\"ogg\"]:\n audio_file = torchaudio.load(str(Path(dirname)/Path(filename)),format=\"wav\")\n self.noise_list.append(audio_file)\n \n def add_random_noise(self,sig,verbose=False):\n # Load noise signal\n noise_sig, noise_sr = random.choice(self.noise_list)\n ... \n sig = AudioUtil.add_noise(sig, noise_sig)\n \n return sig\n```\n\n## Tip - working with large datasets\n\nWhen datasets are too large it might exceed the allocated disk size of /kaggle/working, however, you can still save up to 100GB in other directories such as /kaggle/tmp and upload it to kaggle directly in the script. A quick tutorial [like this one](https://www.kaggle.com/code/xhlulu/how-to-create-very-large-datasets-from-a-notebook/notebook) should help.\n\n# 3. Compute Spectrogram\n\nLast but not least, computing spectrograms are also very cpu consuming. Fortunately, we can move it to the GPU in both tensorflow and pytorch. In pytorch you can create a mel spectorgram instance easily `torchaudio.transforms.MelSpectrogram().to(device)`\n\nJust remember to send the instance to the GPU!\n\n# 4. Others\n\n- To utilize all CPUs, remember to set the dataloader num_workers argument to 2.\n- Set dataloader shuffle argument to False since shuffling the validation set is not needed.\n\n---\n*Wow you made it to the end! If you found it helpful, consider adding me on [linkedin](https://www.linkedin.com/in/lhanhsin/) to increase connections and potentially help me find a job.*" } }