{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Audio Analysis and Augmentation Demo
with Interactive Plots\n\n**Dataset:** [Cornell Birdcall Identification](https://www.kaggle.com/c/birdsong-recognition/data)\n\n\n
\n Version History\n
\n \n
\n\n---"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"import os\nimport random\nimport numpy as np\nimport pandas as pd\nimport scipy\nimport librosa\nimport librosa.display\nfrom IPython.display import Audio\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport plotly.graph_objs as go\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"# Hide unnecessary warnings when loading audio files\nimport warnings\nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data"},{"metadata":{},"cell_type":"markdown","source":"Load the data"},{"metadata":{"trusted":true},"cell_type":"code","source":"audio_path = '../input/birdsong-recognition/train_audio/'\ntest_audio_path = '../input/birdsong-recognition/example_test_audio/'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Analyse the data and the number of audio files per class"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"subFolderList = []\nfor x in os.listdir(audio_path):\n if os.path.isdir(audio_path + '/' + x):\n subFolderList.append(x)\n\nsample_audio = []\ntotal = 0\nfor x in subFolderList:\n \n # get all the wave files\n all_files = [y for y in os.listdir(audio_path+x) if '.mp3' in y]\n total += len(all_files)\n \n # collect the first file from each dir\n sample_audio.append(audio_path + x + '/'+ all_files[0])\n \n # show file counts\n print(x, len(all_files), end=' | ')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print('Number of classes:', len(subFolderList))\nprint(' Number of files:', total)\n\nprint('\\nAverage number of files per class:', round(total/len(subFolderList)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Example Audio \n\nLet us consider this audio as an example to demonstrate."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# demo_audio = '../input/birdsong-recognition/train_audio/aldfly/XC142068.mp3'\ndemo_audio = '../input/birdsong-recognition/train_audio/bewwre/XC122453.mp3' #shorter clip\ndemo_clip, demo_sample_rate = librosa.load(demo_audio, sr=None)\nprint(\"Class:\", demo_audio.split('/')[-2])\nprint(\" File:\", demo_audio)\nAudio(demo_audio)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Audio Visualizations"},{"metadata":{},"cell_type":"markdown","source":"### Raw waves of a few samples\n\nWaveform is merely a graph that displays amplitude or level changes over time. Amplitude is measured in a bipolar manner, with positive and negative values.\n\nAmazing interactive site to learn more: https://pudding.cool/2018/02/waveforms"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"trace = [go.Scatter(\n x=np.linspace(0, demo_sample_rate/len(demo_clip), len(demo_clip)), \n y=demo_clip\n)]\nlayout = go.Layout(\n title = 'Waveform
Interactive',\n yaxis = dict(title='Amplitude'),\n xaxis = dict(title='Time'),\n )\nfig = go.Figure(data=trace, layout=layout)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This visualization is called the **time-domain** representation of a given signal. This shows us the loudness (amplitude) of sound wave changing with time. Here amplitude = 0 represents silence. These amplitudes are not very informative, as they only talk about the loudness of audio recording. "},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# fig = plt.figure(figsize=(25,15))\nfig = plt.figure(figsize=(25,12))\nfor i, filepath in enumerate(sample_audio[:20]):\n # plt.subplot(5,1,i+1)\n plt.subplot(5,4,i+1)\n clip, sample_rate = librosa.load(filepath, sr=None)\n plt.title(filepath.split('/')[-2])\n plt.axis('off')\n plt.plot(clip, c='black', lw=0.5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def plot_raw_waves(label, color=None):\n same_samples = [audio_path + f'{label}/' + y for y in os.listdir(audio_path + f'{label}/')[:20]]\n\n fig = plt.figure(figsize=(25,12))\n fig.suptitle(label, fontsize=30, c=color)\n for i, filepath in enumerate(same_samples):\n plt.subplot(5,4,i+1)\n clip, sample_rate = librosa.load(filepath, sr=None)\n plt.axis('off')\n plt.plot(clip, c=color, lw=0.5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plot_raw_waves('ribgul', '#FBC02D')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plot_raw_waves('casfin', '#D81B60')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plot_raw_waves('snobun', '#F4511E')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plot_raw_waves('lazbun', '#039BE5')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Spectrograms of a few samples\n\nA spectrogram is a visual way of representing the signal strength, or “loudness”, of a signal over time at various frequencies present in a particular waveform. Not only can one see whether there is more or less energy at, for example, 2 Hz vs 10 Hz, but one can also see how energy levels vary over time.\n\nSpectrograms can be two-dimensional graphs with a third variable represented by color. One axis represents time, and the other axis represents frequency; a third dimension indicating the amplitude of a particular frequency at a particular time is represented by the intensity or color of each point in the image.\n\nAmazing spectrogram visualization for ***bird*** here: https://musiclab.chromeexperiments.com/Spectrogram/"},{"metadata":{},"cell_type":"markdown","source":"Function that calculates spectrogram.\n\nNote, that we are taking logarithm of spectrogram values. It will make our plot much more clear, moreover, it is strictly connected to the way people hear. We need to assure that there are no 0 values as input to logarithm."},{"metadata":{"trusted":true},"cell_type":"code","source":"def log_specgram(audio, sample_rate, window_size=20,\n step_size=10, eps=1e-10):\n nperseg = int(round(window_size * sample_rate / 1e3))\n noverlap = int(round(step_size * sample_rate / 1e3))\n freqs, times, spec = scipy.signal.spectrogram(audio,\n fs=sample_rate,\n window='hann',\n nperseg=nperseg,\n noverlap=noverlap,\n detrend=False)\n return freqs, times, np.log(spec.T.astype(np.float32) + eps)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"demo_freqs, demo_times, demo_spec = log_specgram(demo_clip, sample_rate)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"trace = [go.Heatmap(\n x= demo_times,\n y= demo_freqs,\n z= demo_spec.T,\n colorscale='viridis',\n )]\nlayout = go.Layout(\n title = 'Spectrogram
Interactive',\n yaxis = dict(title='Frequency'),\n xaxis = dict(title='Time'),\n )\nfig = go.Figure(data=trace, layout=layout)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = plt.figure(figsize=(25,12))\nfor i, filepath in enumerate(sample_audio[:20]):\n plt.subplot(5,4,i+1)\n label = filepath.split('/')[-2]\n plt.title(label)\n clip, sample_rate = librosa.load(filepath, sr=None)\n _, _, spectrogram = log_specgram(clip, sample_rate)\n plt.imshow(spectrogram.T, aspect='auto', origin='lower')\n plt.axis('off');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def plot_spectrograms(label):\n same_samples = [audio_path + f'{label}/' + y for y in os.listdir(audio_path + f'{label}/')[:20]]\n\n fig = plt.figure(figsize=(25,12))\n fig.suptitle(label, fontsize=30)\n for i, filepath in enumerate(same_samples):\n plt.subplot(5,4,i+1)\n clip, sample_rate = librosa.load(filepath, sr=None)\n _, _, spectrogram = log_specgram(clip, sample_rate)\n plt.imshow(spectrogram.T, aspect='auto', origin='lower')\n plt.axis('off')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plot_spectrograms('ribgul')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plot_spectrograms('casfin')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plot_spectrograms('snobun')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plot_spectrograms('lazbun')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3D Spectrogram"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"trace = [go.Surface(\n x= demo_times,\n y= demo_freqs,\n z= demo_spec.T,\n colorscale='viridis',\n)]\nlayout = go.Layout(\ntitle='3D Specgtrogram
Interactive',\nscene = dict(\n yaxis = dict(title='Frequency', range=[demo_freqs.min(),demo_freqs .max()]),\n xaxis = dict(title='Time', range=[demo_times.min(),demo_times.max()],),\n zaxis = dict(title='Log amplitude'),\n ),\n)\nfig = go.Figure(data=trace, layout=layout)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Fast Fourier Transform (FFT)\n\nAn audio signal is a complex signal composed of multiple ‘single-frequency sound waves’ which travel together as a disturbance(pressure-change) in the medium. When sound is recorded we only capture the resultant amplitudes of those multiple waves. Fourier Transform is a mathematical concept that can **decompose a signal into its constituent frequencies**. Fourier transform does not just give the frequencies present in the signal, It also gives the magnitude of each frequency present in the signal. The only difference between FT(Fourier Transform) and FFT is that FT considers a continuous signal while FFT takes a discrete signal as input."},{"metadata":{},"cell_type":"markdown","source":"Function that calculates FFT.\n\nNote, FFT is simmetrical, so we take just the first half and FFT is also complex, so we consider the real part (abs)."},{"metadata":{"trusted":true},"cell_type":"code","source":"from scipy.fftpack import fft\ndef custom_fft(y, fs):\n T = 1.0 / fs\n N = y.shape[0]\n yf = fft(y)\n xf = np.linspace(0.0, 1.0/(2.0*T), N//2)\n vals = 2.0/N * np.abs(yf[0:N//2]) \n return xf, vals","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"demo_xf, demo_vals = custom_fft(demo_clip, demo_sample_rate)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"trace = [go.Scatter(\n x=demo_xf, \n y=demo_vals,\n line_color='deeppink'\n)]\nlayout = go.Layout(\n title = 'Fast Fourier Transform (FFT)
Interactive',\n yaxis = dict(title='Magnitude'),\n xaxis = dict(title='Frequency'),\n )\nfig = go.Figure(data=trace, layout=layout)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(25,12))\nfor i, filepath in enumerate(sample_audio[:20]):\n plt.subplot(5,4,i+1)\n label = filepath.split('/')[-2]\n plt.title(label)\n clip, sample_rate = librosa.load(filepath, sr=None)\n xf, vals = custom_fft(clip, sample_rate)\n plt.plot(xf, vals, c='black')\n plt.axis('off');","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"def plot_ffts(label, color=None):\n same_samples = [audio_path + f'{label}/' + y for y in os.listdir(audio_path + f'{label}/')[:20]]\n\n fig = plt.figure(figsize=(25,12))\n fig.suptitle(label, fontsize=30, c=color)\n for i, filepath in enumerate(same_samples):\n plt.subplot(5,4,i+1)\n clip, sample_rate = librosa.load(filepath, sr=None)\n xf, vals = custom_fft(clip, sample_rate)\n plt.plot(xf, vals, c=color)\n plt.axis('off')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot_ffts('ribgul', '#FBC02D')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot_ffts('casfin', '#D81B60')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot_ffts('snobun', '#F4511E')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot_ffts('lazbun', '#039BE5')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Mel Spectrogram\n\nThe Mel Scale, mathematically speaking, is the result of some **non-linear transformation of the frequency scale**. This Mel Scale is constructed such that sounds of equal distance from each other on the Mel Scale, also _“sound”_ to humans as they are equal in distance from one another.\nIn contrast to Hz scale, where the difference between 500 and 1000 Hz is obvious, whereas the difference between 7500 and 8000 Hz is barely noticeable.\nMel Spectrogram, is, rather surprisingly, a Spectrogram with the Mel Scale as its y axis.\nMore [here](https://towardsdatascience.com/getting-to-know-the-mel-spectrogram-31bca3e2d9d0).\n"},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"demo_S = librosa.feature.melspectrogram(demo_clip, sr=demo_sample_rate, n_mels=128) \ndemo_log_S = librosa.power_to_db(demo_S, ref=np.max)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"trace = [go.Heatmap(\n x= demo_times,\n y= demo_freqs,\n z= demo_log_S,\n colorscale='magma',\n )]\nlayout = go.Layout(\n title = 'Mel Power Spectrogram
Interactive',\n yaxis = dict(title='Mel'),\n xaxis = dict(title='Time'),\n )\nfig = go.Figure(data=trace, layout=layout)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(25,12))\nfor i, filepath in enumerate(sample_audio[:20]):\n plt.subplot(5,4,i+1)\n label = filepath.split('/')[-2]\n plt.title(label)\n clip, sample_rate = librosa.load(filepath, sr=None)\n S = librosa.feature.melspectrogram(clip, sr=sample_rate, n_mels=128) \n log_S = librosa.power_to_db(S, ref=np.max) \n librosa.display.specshow(log_S, sr=sample_rate, x_axis='time', y_axis='mel')\n plt.axis('off');","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"def plot_mel_spectrograms(label):\n same_samples = [audio_path + f'{label}/' + y for y in os.listdir(audio_path + f'{label}/')[:20]]\n\n fig = plt.figure(figsize=(25,12))\n fig.suptitle(label, fontsize=30)\n for i, filepath in enumerate(same_samples):\n plt.subplot(5,4,i+1)\n clip, sample_rate = librosa.load(filepath, sr=None)\n S = librosa.feature.melspectrogram(clip, sr=sample_rate, n_mels=128) \n log_S = librosa.power_to_db(S, ref=np.max) \n librosa.display.specshow(log_S, sr=sample_rate, x_axis='time', y_axis='mel')\n plt.axis('off');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plot_mel_spectrograms('ribgul')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plot_mel_spectrograms('casfin')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plot_mel_spectrograms('snobun')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plot_mel_spectrograms('lazbun')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Mel Frequency Cepstral Coefficient (MFCC)\n\nMFCC features represent phonemes (distinct units of sound) as the shape of the vocal tract (which is responsible for sound generation) is manifest in them. The mel frequency cepstral coefficients (MFCCs) of a signal are a small set of features (usually about 10-20) which concisely **describe the overall shape of a spectral envelope**.\nMore [here](https://medium.com/prathena/the-dummys-guide-to-mfcc-aceab2450fd) & [here](http://practicalcryptography.com/miscellaneous/machine-learning/guide-mel-frequency-cepstral-coefficients-mfccs/)."},{"metadata":{},"cell_type":"markdown","source":"> In classical, but still state-of-the-art systems, MFCC or similar features are taken as the input to the system instead of spectrograms. However, in end-to-end (often neural-network based) systems, the most common input features are probably raw spectrograms, or mel power spectrograms. For example MFCC decorrelates features, but NNs deal with correlated features well."},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"demo_mfcc = librosa.feature.mfcc(S=demo_log_S, n_mfcc=13)\ndemo_delta2_mfcc = librosa.feature.delta(demo_mfcc, order=2)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"trace = [go.Heatmap(\n x= demo_times,\n y= demo_freqs,\n z= demo_delta2_mfcc,\n colorscale='RdBu_r',\n )]\nlayout = go.Layout(\n title = 'MFCC
Interactive',\n yaxis = dict(title='MFCC coeffs'),\n xaxis = dict(title='Time'),\n )\nfig = go.Figure(data=trace, layout=layout)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(25,12))\nfor i, filepath in enumerate(sample_audio[:20]):\n plt.subplot(5,4,i+1)\n label = filepath.split('/')[-2]\n plt.title(label)\n clip, sample_rate = librosa.load(filepath, sr=None)\n S = librosa.feature.melspectrogram(clip, sr=sample_rate, n_mels=128) \n log_S = librosa.power_to_db(S, ref=np.max) \n mfcc = librosa.feature.mfcc(S=log_S, n_mfcc=13)\n delta2_mfcc = librosa.feature.delta(mfcc, order=2)\n librosa.display.specshow(delta2_mfcc, cmap='bwr')\n plt.axis('off');","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"def plot_mmfcs(label):\n same_samples = [audio_path + f'{label}/' + y for y in os.listdir(audio_path + f'{label}/')[:20]]\n\n fig = plt.figure(figsize=(25,12))\n fig.suptitle(label, fontsize=30)\n for i, filepath in enumerate(same_samples):\n plt.subplot(5,4,i+1)\n clip, sample_rate = librosa.load(filepath, sr=None)\n S = librosa.feature.melspectrogram(clip, sr=sample_rate, n_mels=128) \n log_S = librosa.power_to_db(S, ref=np.max) \n mfcc = librosa.feature.mfcc(S=log_S, n_mfcc=13)\n delta2_mfcc = librosa.feature.delta(mfcc, order=2)\n librosa.display.specshow(delta2_mfcc, cmap='bwr')\n plt.axis('off');","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot_mmfcs('ribgul')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot_mmfcs('casfin')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot_mmfcs('snobun')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot_mmfcs('lazbun')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## In-depth per sample analysis\n\nLet's analyse individual audio files with all the above visualizations to get a better understanding."},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"def audio_plots(filename):\n clip, sample_rate = librosa.load(filename, sr=None)\n freqs, times, spectrogram = log_specgram(clip, sample_rate)\n label = filename.split('/')[-2]\n \n # # Normalize (if needed)\n # mean = np.mean(spectrogram, axis=0)\n # std = np.std(spectrogram, axis=0)\n # spectrogram = (spectrogram - mean) / std\n\n fig = plt.figure(figsize=(20,15), dpi=200)\n \n # Raw wave\n ax1 = fig.add_subplot(511)\n ax1.set_title('Raw wave of ' + label)\n ax1.set_ylabel('Amplitude')\n librosa.display.waveplot(clip.astype('float'), sr=sample_rate)\n\n # FFT \n ax2 = fig.add_subplot(512)\n xf, vals = custom_fft(clip, sample_rate)\n ax2.set_title('FFT of ' + label + ' with ' + str(sample_rate) + ' Hz')\n ax2.plot(xf, vals, 'm')\n ax2.set_xlabel('Frequency')\n ax2.set_ylabel('Magnitude')\n ax2.grid()\n \n # Spectrogram \n ax3 = fig.add_subplot(513)\n ax3.set_title('Spectrogram of ' + label)\n ax3.set_ylabel('Freqs in Hz')\n ax3.set_xlabel('Seconds')\n ax3.imshow(spectrogram.T, aspect='auto', origin='lower', \n extent=[times.min(), times.max(), freqs.min(), freqs.max()])\n \n # Mel power spectrogram\n ax4 = fig.add_subplot(514) \n S = librosa.feature.melspectrogram(clip, sr=sample_rate, n_mels=128) \n log_S = librosa.power_to_db(S, ref=np.max) \n librosa.display.specshow(log_S, sr=sample_rate, x_axis='time', y_axis='mel')\n ax4.set_title('Mel power spectrogram of ' + label) \n \n # MFCC\n ax5 = fig.add_subplot(515)\n mfcc = librosa.feature.mfcc(S=log_S, n_mfcc=13)\n delta2_mfcc = librosa.feature.delta(mfcc, order=2)\n librosa.display.specshow(delta2_mfcc, cmap='bwr')\n ax5.set_title('MFCC of ' + label)\n ax5.set_ylabel('MFCC coeffs')\n ax5.set_xlabel('Time')\n\n plt.tight_layout()\n print(f\"Class: {label}\\n File: {filename}, \")\n return Audio(filename)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"audio_plots(demo_audio)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"audio_plots(sample_audio[1])","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"audio_plots(sample_audio[50])","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"audio_plots(sample_audio[100])","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"audio_plots(sample_audio[150])","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"audio_plots(sample_audio[200])","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"audio_plots(sample_audio[250])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Audio Augmentations\n\nLet us consider this *(same)* audio as an example to demonstrate.\n\n> ***NOTE:*** Augmentation values are boosted to showcase their effect"},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"def load_audio_file(filename):\n data, sr = librosa.load(filename, sr=None)\n return data, sr\n\nsns.set(style='darkgrid')\ndef plot_time_series(data, sr, alpha=0.5, color=None, label=None):\n# plt.figure(figsize=(20,5))\n plt.title('Raw wave')\n plt.ylabel('Amplitude')\n plt.plot(np.linspace(0, sr/len(data), len(data)), data, alpha=alpha, c=color, label=label)\n plt.legend(loc='lower left', ncol=2, frameon=False)\n return Audio(data, rate=sr)\n\ndef plot_spectrogram(data, sr, label=None):\n S = librosa.feature.melspectrogram(data, sr, n_mels=128) \n log_S = librosa.power_to_db(S, ref=np.max)\n plt.figure(figsize=(20,3), dpi=200)\n plt.title('Mel power spectrogram')\n plt.text(0.05, 200, label, fontsize=12, c='w')\n librosa.display.specshow(log_S, sr=sr, x_axis='time', y_axis='mel') \n# plt.axis('off')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"org_data, sr = load_audio_file(demo_audio)\nprint(\"Class:\", demo_audio.split('/')[-2])\nprint(\" File:\", demo_audio)\n\nplt.figure(figsize=(20,5))\nplot_time_series(org_data, sr, 0.67, 'blue', 'Actual')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"BLUE is the actual audio clip and RED is for the augmented audio clip."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot_spectrogram(org_data, sr, 'Actual')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Normalize\n\nLet's first normalize the audio clip before applying augmentations."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Use mean & std of individual data or of all data as custom params\ndef normalize(X, mean=None, std=None):\n mean = mean or X.mean()\n std = std or (X-X.mean()).std()\n return ((X - mean)/std).astype(np.float64)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data = normalize(org_data)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(20,5))\nplot_time_series(org_data, sr, 0.7, 'blue', 'Actual')\nplot_time_series(data, sr, 0.6, 'red', 'Normalized')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot_spectrogram(org_data, sr, 'Actual')\nplot_spectrogram(data, sr, 'Normalized')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Adding white noise "},{"metadata":{"trusted":true},"cell_type":"code","source":"wn = np.random.randn(len(data))\ndata_wn = data + 0.1 * wn","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(20,5))\nplot_time_series(data, sr, 0.7, 'blue', 'Actual')\nplot_time_series(data_wn, sr, 0.6, 'red', 'White Noise')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot_spectrogram(data, sr, 'Actual')\nplot_spectrogram(data_wn, sr, 'White Noise')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Shifting the sound"},{"metadata":{"trusted":true},"cell_type":"code","source":"data_roll = np.roll(data, 5000)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(20,5))\nplot_time_series(data, sr, 0.5, 'blue', 'Actual')\nplot_time_series(data_roll, sr, 0.7, 'red', 'Shifted')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot_spectrogram(data, sr, 'Actual')\nplot_spectrogram(data_roll, sr, 'Shifted')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Stretching the sound"},{"metadata":{"trusted":true},"cell_type":"code","source":"# larger value\ndata_stretch = librosa.effects.time_stretch(data, 1.2)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(20,5))\nplot_time_series(data, sr, 0.3, 'blue', 'Actual')\nplot_time_series(data_stretch, sr, 0.8, 'red', 'Stretched')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot_spectrogram(data, sr, 'Actual')\nplot_spectrogram(data_stretch, sr, 'Stretched')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# smaller value\ndata_stretch = librosa.effects.time_stretch(data, 0.8)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(20,5))\nplot_time_series(data, sr, 0.3, 'blue', 'Actual')\nplot_time_series(data_stretch, sr, 0.8, 'red', 'Stretched')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot_spectrogram(data, sr, 'Actual')\nplot_spectrogram(data_stretch, sr, 'Stretched')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Changing the Pitch"},{"metadata":{"trusted":true},"cell_type":"code","source":"data_pitch = librosa.effects.pitch_shift(data, sr, n_steps=-10)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(20,5))\nplot_time_series(data, sr, 0.5, 'blue', 'Actual')\nplot_time_series(data_pitch, sr, 0.8, 'red', 'Pitch changed')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot_spectrogram(data, sr, 'Actual')\nplot_spectrogram(data_pitch, sr, 'Pitch changed')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Inverting the Polarity"},{"metadata":{"trusted":true},"cell_type":"code","source":"data_invert = -data","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(20,5))\nplot_time_series(data, sr, 1, 'blue', 'Actual')\nplot_time_series(data_invert, sr, 0.7, 'red', 'Inverted')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot_spectrogram(data, sr, 'Actual')\nplot_spectrogram(data_invert, sr, 'Inverted')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### All Augmentations"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"x = np.linspace(0, sr/len(data), len(data))\ntrace = [\n go.Scatter(x=x, y=org_data, name='Actual', opacity=0.5),\n go.Scatter(x=x, y=data, name='Normalized', opacity=0.5),\n go.Scatter(x=x, y=data_wn, name='Noise', opacity=0.5),\n go.Scatter(x=x, y=data_roll, name='Shift', opacity=0.5),\n go.Scatter(x=x, y=data_stretch, name='Stretch', opacity=0.5),\n go.Scatter(x=x, y=data_pitch, name='Pitch', opacity=0.5),\n go.Scatter(x=x, y=data_invert, name='Invert', opacity=0.5),\n]\n\nlayout = go.Layout(\n title = 'Augmentations
Interactive',\n yaxis = dict(title='Amplitude'),\n xaxis = dict(title='Time'),\n legend_title_text='Augmentations
Toggle augmentations on/off'\n )\n\nfig = go.Figure(data=trace, layout=layout)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Combining all in one"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(\"ACTUAL:\")\norg_data, sr = load_audio_file(demo_audio)\nAudio(org_data, rate=sr)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"data = normalize(org_data)\ndata_wn = data + 0.1 * wn\ndata_roll = np.roll(data_wn, 5000)\ndata_stretch = librosa.effects.time_stretch(data_roll, 1)\ndata_pitch = librosa.effects.pitch_shift(data_stretch, sr, n_steps=-10)\ndata_invert = -data_pitch\ndata_aug = data_invert\n\nplt.figure(figsize=(20,5))\nplot_time_series(data, sr, 0.3, 'blue', 'Actual')\nprint(\"AUGMENTED:\")\nplot_time_series(data_aug, sr, 0.7, 'red', 'Augmented')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot_spectrogram(data, sr, 'Actual')\nplot_spectrogram(data_aug, sr, 'Augmented')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Mel Spec Augmentations\n\nAs mentioned earlier, we can convert this time-series problem into a computer vision problem by converting the audio data into mel spectrograms *(or MFCC)*.\n\nThis is similar to any other image classification problem, where all the common augmentations can be done. Here are a few most common augmentation methods for mel spectrograms.\n\nLet us consider this *(same)* audio as an example to demonstrate."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"clip, sample_rate = librosa.load(demo_audio, sr=None)\nprint(\"Class:\", demo_audio.split('/')[-2])\nprint(\" File:\", demo_audio)\nAudio(demo_audio)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"S = librosa.feature.melspectrogram(clip, sr=sample_rate, n_mels=256) \nlog_S = librosa.power_to_db(S, ref=np.max)\n\nplt.figure(figsize=(20,4))\nplt.title('Mel power spectrogram')\nplt.text(0.05, 200, 'original audio', fontsize=12, c='w')\nlibrosa.display.specshow(log_S, sr=sample_rate, x_axis='time', y_axis='mel');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Spec Dropout\n\nWe augment spectrograms by hiding random time & frequency intervals."},{"metadata":{"trusted":true},"cell_type":"code","source":"def spec_augment(spec: np.ndarray, num_mask=2, freq_masking_max_percentage=0.15, time_masking_max_percentage=0.3):\n spec = spec.copy()\n for i in range(num_mask):\n num_freqs, num_frames = spec.shape\n freq_percentage = random.uniform(0.0, freq_masking_max_percentage)\n time_percentage = random.uniform(0.0, time_masking_max_percentage)\n \n num_freqs_to_mask = int(freq_percentage * num_freqs)\n num_frames_to_mask = int(time_percentage * num_frames)\n \n t0 = int(np.random.uniform(low=0.0, high=num_frames - num_frames_to_mask))\n f0 = int(np.random.uniform(low=0.0, high=num_freqs - num_freqs_to_mask))\n \n spec[:, t0:t0 + num_frames_to_mask] = 0 \n spec[f0:f0 + num_freqs_to_mask, :] = 0 \n \n return spec","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"for i in range(4):\n plt.figure(figsize=(20,3))\n librosa.display.specshow(spec_augment(log_S), sr=sample_rate, x_axis='time', y_axis='mel');\n# plt.axis('off')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Pad to size\n\nIf the spectrogram is shorter than the required image size, it can be padded."},{"metadata":{"trusted":true},"cell_type":"code","source":"def pad_to_size(signal, size, mode):\n if signal.shape[1] < size:\n padding = size - signal.shape[1]\n offset = padding // 2\n pad_width = ((0, 0), (offset, padding - offset))\n if mode == 'constant':\n signal = np.pad(signal, pad_width, 'constant', constant_values=signal.min())\n elif mode == 'wrap':\n signal = np.pad(signal, pad_width, 'wrap')\n return signal ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"img_size = 400\n\nplt.figure(figsize=(20,4))\nplt.title('Mel power spectrogram')\nplt.text(0.05, 200, 'constant', fontsize=12, c='w')\nlibrosa.display.specshow(pad_to_size(log_S, img_size, 'constant'), sr=sample_rate, x_axis='time', y_axis='mel');\n\nplt.figure(figsize=(20,4))\nplt.title('Mel power spectrogram')\nplt.text(0.05, 200, 'wrap', fontsize=12, c='w')\nlibrosa.display.specshow(pad_to_size(log_S, img_size, 'wrap'), sr=sample_rate, x_axis='time', y_axis='mel');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Acknowledgements and References:\n\n- https://www.kaggle.com/davids1992/speech-representation-and-data-exploration\n- https://www.kaggle.com/timolee/audio-data-conversion-to-images-eda\n- https://www.kaggle.com/CVxTz/audio-data-augmentation\n\n---"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}