{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"![xyzy fsadsdf](https://images.unsplash.com/photo-1511142878591-5040f0bdaadd?ixid=MnwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8&ixlib=rb-1.2.1&auto=format&fit=crop&w=1650&q=80)","metadata":{}},{"cell_type":"markdown","source":"[](https://images.unsplash.com/photo-1473448912268-2022ce9509d8?ixid=MnwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8&ixlib=rb-1.2.1&auto=format&fit=crop&w=1625&q=80)","metadata":{}},{"cell_type":"markdown","source":"## 🎯 Motivations\n**Processing spectrogram on-the-fly is slow!** And it's likely to be a bottleneck of your GPU utility rate, and hence the speed of your training loop. One simple way to speed up your data pipeline is to precompute all spectrograms into disk in prior. By doing so you can avoid repeated costly transformation. \n\nIn this notebook, I will implement this simple solution in fastai and fastaudio, using [Rainforest Connection Species Audio Detection competition](https://www.kaggle.com/c/rfcx-species-audio-detection/overview) as an example. I will show you:  \n1. How to transforme audios into mel-spectrograms and save them into disk\n2. How to include audio-domain augmentations in the precomputed mel-spectrogram\n2. How to construct `DataLoaders` from these precomputed mel-spectrograms\n3. Compare the speed of data loading with and without precomputed mel-spectrogram\n\nAt the end, I will show you how you can achieve **20x SPEED-UP** with this simple trick! (Remember to run this kernel in GPU mode to reproduce the speed-up stated)","metadata":{}},{"cell_type":"code","source":"import pkg_resources\n\ndef placeholder(x):\n    raise pkg_resources.DistributionNotFound\npkg_resources.get_distribution = placeholder\n\n\n# uninstall fastai-2.3.0 & fastcore-1.3.19 to remove constraint on torch version\n!pip uninstall fastai fastcore torchaudio -y\n!pip install torch==1.8.1 torchaudio==0.8.1 fastcore==1.3.20\n!pip install fastaudio","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-06-25T01:07:37.95961Z","iopub.execute_input":"2021-06-25T01:07:37.959996Z","iopub.status.idle":"2021-06-25T01:09:25.372102Z","shell.execute_reply.started":"2021-06-25T01:07:37.959915Z","shell.execute_reply":"2021-06-25T01:09:25.371064Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\nimport librosa\nfrom tqdm import tqdm\n\nimport pandas as pd\nfrom fastaudio.all import *\nfrom fastai.vision.all import *\n\nimport torch\nimport torchaudio\nimport fastcore\nimport fastai\nimport fastaudio\nimport torchaudio\ntorchaudio.set_audio_backend(\"sox_io\")","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-06-25T01:09:25.375606Z","iopub.execute_input":"2021-06-25T01:09:25.37591Z","iopub.status.idle":"2021-06-25T01:09:28.213707Z","shell.execute_reply.started":"2021-06-25T01:09:25.375877Z","shell.execute_reply":"2021-06-25T01:09:28.212808Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 📔 Bookkeeping","metadata":{}},{"cell_type":"code","source":"print(f'CUDA mode: {torch.cuda.is_available()}')\nfor _module in (torch, torchaudio, fastcore, fastaudio, fastai):\n    print(f'{_module.__name__} version: {_module.__version__}')","metadata":{"execution":{"iopub.status.busy":"2021-06-25T01:09:28.2155Z","iopub.execute_input":"2021-06-25T01:09:28.216042Z","iopub.status.idle":"2021-06-25T01:09:28.223097Z","shell.execute_reply.started":"2021-06-25T01:09:28.216002Z","shell.execute_reply":"2021-06-25T01:09:28.222134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# CONFIGURATIONS\nDATA_DIR = Path('../input/rfcx-species-audio-detection')\nCACHE_DIR = Path('./cached_melspectrograms')\nSR = 48000\nMELSPEC_CONFIG = {'n_fft': 2048, 'sample_rate': SR}\nCROP_INTERVAL = 10  # sec\nRADNOM_CROP_INTERVAL = 8  # sec\nRESAMPLE_N = 15\nBS = 32\nCLASS_N = 24\nCLASS_NAMES = list(map(str, range(CLASS_N)))","metadata":{"execution":{"iopub.status.busy":"2021-06-25T01:09:28.224328Z","iopub.execute_input":"2021-06-25T01:09:28.224795Z","iopub.status.idle":"2021-06-25T01:09:28.238088Z","shell.execute_reply.started":"2021-06-25T01:09:28.224757Z","shell.execute_reply":"2021-06-25T01:09:28.236962Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ☕ Prepare Our Mel-spectrogram into Disk\nFirst we build a pipeline that can transform audio into mel-spectrogram and then save them into disk. Instead of utilizing a full audio clip, here we crop a small clip of 10 seconds long around the place when there is a species identified, so a cropped clip correspond to a sample of a single species. This has been shown to be an effective approach for model to learn the classification of different species.  \n\nWe applied `ResizeSignal` and `AddNoise` as augmentations on the audio. `ResizeSignal` will randomly crop a 8-second clip again from the 10-second clip. In the pipeline, we will bootstrap the same sample many times (controlled by `RESAMPLE_N`) in order to create a variety of augmented snapshots. With such setting, we can emulate the samples being randomly augmented over iterations in downstream stage. (We will show this later)\n\nWe need an identifiable filename for each sample. Since a `recording_id` contain multiple samples, we can additionally make use of `t_min` and `t_max` (those are the columns from `train_tp.csv`) to annotate the filename of a sample, with different snapshots having different index. i.e. `{recording_id}_{t_min}_{t_max}_{index}.pt`","metadata":{}},{"cell_type":"code","source":"audio_fns = get_audio_files(DATA_DIR/'train')\nprint(f'No. of audio files in train folder: {len(audio_fns)}')\n\n# to save time, I subset training data\ndf = pd.read_csv(DATA_DIR/ 'train_tp.csv')\ndf.species_id = df.species_id.astype(str)\nprint(f'No. of samples to be processed: {df.shape[0]}')","metadata":{"execution":{"iopub.status.busy":"2021-06-24T17:08:38.23173Z","iopub.execute_input":"2021-06-24T17:08:38.232104Z","iopub.status.idle":"2021-06-24T17:08:42.546478Z","shell.execute_reply.started":"2021-06-24T17:08:38.232074Z","shell.execute_reply":"2021-06-24T17:08:42.545515Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def read_and_crop_audio(row: pd.Series) -> AudioTensor:\n    # crop the subclip responsible for the target species\n    t_min, t_max = row.t_min, row.t_max\n    center = (t_max + t_min) / 2\n    start_t = center - (CROP_INTERVAL / 2.)\n    \n    _frame_offset = int(max(0, start_t) * SR)\n    _num_frames = int(CROP_INTERVAL * SR)\n    audio_fn = DATA_DIR.resolve()/ f'train/{row.recording_id}.flac'\n    assert audio_fn.is_file()\n    audio = AudioTensor.create(audio_fn,\n                               frame_offset=_frame_offset,\n                               num_frames=_num_frames)\n    \n    # attach metadata to tensor so as to help create unique filename later\n    audio.t_min, audio.t_max = row.t_min, row.t_max\n    audio.recording_id = row.recording_id\n    return audio\n\n\ndef new_encodes(self, audio: AudioTensor):\n    self.pipe.to(audio.device)\n    self.settings.update({\"sr\": audio.sr, \"nchannels\": audio.nchannels})\n    spec = AudioSpectrogram.create(self.pipe(audio), settings=dict(self.settings))\n    \n    # propagate metadata from AdudioTensor to AudioSpectrogram\n    spec.__dict__.update(audio.__dict__)\n    return spec\n\n# overload an existing type-dispatch function\nAudioToSpec.encodes.add(new_encodes)\n\n\ndef save_spectrogram(spec: AudioSpectrogram, idx: int):\n    CACHE_DIR.mkdir(exist_ok=True)\n    spec_fn = f'{spec.recording_id}_{spec.t_min:.3f}_{spec.t_max:.3f}_{idx}.pt'\n    torch.save(spec, CACHE_DIR/ spec_fn)","metadata":{"execution":{"iopub.status.busy":"2021-06-24T17:08:42.547918Z","iopub.execute_input":"2021-06-24T17:08:42.548263Z","iopub.status.idle":"2021-06-24T17:08:42.55785Z","shell.execute_reply.started":"2021-06-24T17:08:42.548227Z","shell.execute_reply":"2021-06-24T17:08:42.556868Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"spec_tfms = AudioToSpec.from_cfg(AudioConfig.BasicMelSpectrogram(**MELSPEC_CONFIG))\ntfms_ls = [read_and_crop_audio, \n           AddNoise(noise_level=0.1), \n           ResizeSignal(RADNOM_CROP_INTERVAL*1000, AudioPadType.Repeat),\n           spec_tfms]\ndsets = Datasets(items=df, tfms=[tfms_ls])","metadata":{"execution":{"iopub.status.busy":"2021-06-24T17:08:42.560507Z","iopub.execute_input":"2021-06-24T17:08:42.56105Z","iopub.status.idle":"2021-06-24T17:08:42.751629Z","shell.execute_reply.started":"2021-06-24T17:08:42.561012Z","shell.execute_reply":"2021-06-24T17:08:42.750795Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nfor sample_i in tqdm(range(len(dsets))):\n    # create different snapshots as an augmentation in audio domain\n    for repeat_j in range(RESAMPLE_N):\n        spec, = dsets[sample_i]\n        save_spectrogram(spec, repeat_j)","metadata":{"execution":{"iopub.status.busy":"2021-06-24T17:08:42.753082Z","iopub.execute_input":"2021-06-24T17:08:42.753427Z","iopub.status.idle":"2021-06-24T17:21:30.618147Z","shell.execute_reply.started":"2021-06-24T17:08:42.75339Z","shell.execute_reply":"2021-06-24T17:21:30.616637Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# sanity check all files have been successfully saved\n!ls $CACHE_DIR | wc -l","metadata":{"execution":{"iopub.status.busy":"2021-06-24T17:21:30.620814Z","iopub.execute_input":"2021-06-24T17:21:30.621067Z","iopub.status.idle":"2021-06-24T17:21:31.448385Z","shell.execute_reply.started":"2021-06-24T17:21:30.62104Z","shell.execute_reply":"2021-06-24T17:21:31.447322Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ⏩ Build `DataLoaders` with the Precomputed Mel-spectrogram\nOnce all mel-spectrograms have been saved, we can build `DataLoaders` for loading our precomputed mel-spectrogram. For simplicity, I skipped over the part for creating validation set, but you can easily plug in those parts into the pipeline.","metadata":{}},{"cell_type":"code","source":"def load_precompute_spectrogram(row: pd.Series) -> AudioSpectrogram:\n    assert CACHE_DIR.is_dir()\n    # use random file sampling to emulate audio domain augmentation\n    random_idx = random.choice(range(RESAMPLE_N))\n    random_fn = f'{row.recording_id}_{row.t_min:.3f}_{row.t_max:.3f}_{random_idx}.pt'\n    fn = CACHE_DIR/ random_fn\n    spec = torch.load(fn)\n    return spec\n\n\ndef parse_labels_from_row(row: pd.Series):\n    _y_reader = ColReader('species_id')\n    label = _y_reader(row)\n    assert isinstance(label, str)\n    return [label]","metadata":{"execution":{"iopub.status.busy":"2021-06-24T17:26:44.091576Z","iopub.execute_input":"2021-06-24T17:26:44.091954Z","iopub.status.idle":"2021-06-24T17:26:44.097721Z","shell.execute_reply.started":"2021-06-24T17:26:44.091914Z","shell.execute_reply":"2021-06-24T17:26:44.096901Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# emulate DataLoaders for model training: with input + target\nx_tfms_ls = [load_precompute_spectrogram]\ny_tfms_ls = [parse_labels_from_row, MultiCategorize(vocab=CLASS_NAMES), OneHotEncode]\ndsets_precomp = Datasets(df, tfms=[x_tfms_ls, y_tfms_ls])\ndls_precomp = dsets_precomp.dataloaders(bs=BS, num_workers=4)\n\n# sanity check\nbatch_precomp = dls_precomp.one_batch()\nassert isinstance(batch_precomp[0], AudioSpectrogram)\nassert isinstance(batch_precomp[1], TensorMultiCategory)","metadata":{"execution":{"iopub.status.busy":"2021-06-24T17:40:35.499645Z","iopub.execute_input":"2021-06-24T17:40:35.500025Z","iopub.status.idle":"2021-06-24T17:40:35.578175Z","shell.execute_reply.started":"2021-06-24T17:40:35.499993Z","shell.execute_reply":"2021-06-24T17:40:35.577312Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 🚀 Time the Speed of Loading 1 Epoch \nTo time the speed of data loading, lets emulate an epoch of data feeding into our model. Note that as a sole purpose of illustrating the speed, I simply feed each batch to the model ResNet18, instead of constructing a full training loop with `Learner`.","metadata":{}},{"cell_type":"code","source":"device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\nmodel = create_cnn_model(resnet18, n_in=1, \n                         n_out=CLASS_N, \n                         pretrained=True)\nmodel.to(device);","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It took only **~2s to complete ~1200 samples**!","metadata":{}},{"cell_type":"code","source":"%%time\nfor _batch in tqdm(dls_precomp.train):\n    _ = model(_batch[0])","metadata":{"execution":{"iopub.status.busy":"2021-06-24T17:40:37.009554Z","iopub.execute_input":"2021-06-24T17:40:37.009911Z","iopub.status.idle":"2021-06-24T17:40:38.943223Z","shell.execute_reply.started":"2021-06-24T17:40:37.009878Z","shell.execute_reply":"2021-06-24T17:40:38.940333Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 🔥 20x Speedup Compared to On-the-fly Approach!\nTo contrast how fast it is, lets build another `DataLoaders` with the same pipeline, but this time we compute mel-spectrogram on-the-fly. ","metadata":{}},{"cell_type":"code","source":"# DataLoaders that load in melspectrogram on the fly\nonthefly_x_tfms_ls = tfms_ls[:-1]\ndsets_woprecomp = Datasets(df, tfms=[onthefly_x_tfms_ls, y_tfms_ls])\ndls_woprecomp = dsets_woprecomp.dataloaders(bs=BS, after_batch=spec_tfms, num_workers=4)\n\n# sanity check\nbatch_woprecomp = dls_woprecomp.one_batch()\nassert isinstance(batch_woprecomp[0], AudioSpectrogram)\nassert isinstance(batch_woprecomp[1], TensorMultiCategory)","metadata":{"execution":{"iopub.status.busy":"2021-06-24T17:41:15.464521Z","iopub.execute_input":"2021-06-24T17:41:15.464875Z","iopub.status.idle":"2021-06-24T17:41:16.979441Z","shell.execute_reply.started":"2021-06-24T17:41:15.464841Z","shell.execute_reply":"2021-06-24T17:41:16.978417Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This time it took **~38s (v.s. 2s)** to complete the same loading.  \n\n**So we have gained ~20x speedup simply by precomputing melspectrograms!**","metadata":{}},{"cell_type":"code","source":"%%time\nfor _batch in tqdm(dls_woprecomp.train):\n    _ = model(_batch[0])","metadata":{"execution":{"iopub.status.busy":"2021-06-24T17:41:22.625076Z","iopub.execute_input":"2021-06-24T17:41:22.625406Z","iopub.status.idle":"2021-06-24T17:41:59.912976Z","shell.execute_reply.started":"2021-06-24T17:41:22.625376Z","shell.execute_reply":"2021-06-24T17:41:59.912116Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 📝 Final Remarks\n1. Saving different (random) snapshots of the same audio could enumlate augmentations done in audio space, but shortcoming of this approach is that it could eat up a lot of disk space: the more variations you want, you more snapshots you have to save in your disk. This is certainty a issue when you have a big sample size. Such snapshots could be several times bigger than your original input.\n2. The speed of 1 epoch is mainly determined by the speed of data processing and the speed of your model, so if you plug in a heavier model in the same pipeline, probably you cant get the same extent of speedup because the speed is likely to be bounded by your model.","metadata":{}},{"cell_type":"code","source":"!du -sh $CACHE_DIR","metadata":{"execution":{"iopub.status.busy":"2021-06-24T17:48:07.137196Z","iopub.execute_input":"2021-06-24T17:48:07.137519Z","iopub.status.idle":"2021-06-24T17:48:07.929894Z","shell.execute_reply.started":"2021-06-24T17:48:07.137489Z","shell.execute_reply":"2021-06-24T17:48:07.928878Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}