{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Introduction !\n\nIn [this kernel](https://www.kaggle.com/davids1992/data-visualization-and-investigation) it was proposed to shrink the samples to those parts where speech could be identified (to speed up training) using webrtcvad\n\nAs opposed to the original Kernel posted by *ANDRE HOLZNER* ; this is fully functional one after multiple modifications on the [original kernel](https://www.kaggle.com/holzner/voice-activity-detection-example).\nI have done so many modification in the above two kernels. As, some of them were not working now or not appropriate to my use-case. I tweaked the code and comprehensions somewhere, also. I hope you would love this and appreciate for the effort put into it.\n\n**Note :** You have to keep your internet toggle **ON** to successfully execute all cells of this notebook."},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"## Table of Contents:\n\n\n\n\n- 0. Read the Audio Files\n\n
\n- 1. Visualization\n\n
\n- 2. Voice Activity Detection ( VAD )\n\n
\n- 3. Listen to the speech only segments\n\n
\n- 4. Frequency components across the words\n"},{"metadata":{},"cell_type":"markdown","source":"### Importing necessary Libraries"},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nfrom os.path import isdir, join\nfrom scipy.io import wavfile\nfrom subprocess import check_output\nfrom pathlib import Path\nimport pandas as pd\n\n\n# Math\nimport numpy as np\nfrom scipy.fftpack import fft\nfrom scipy import signal\nfrom scipy.io import wavfile\nimport librosa\n\nfrom sklearn.decomposition import PCA\n\n# Visualization\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport IPython.display as ipd\nimport librosa.display\n\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nimport plotly.tools as tls\nimport pandas as pd\n\n%matplotlib inline\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 0. Read the Audio Files\n\n### Here's abig deal. Earlier versions of this kernel could read .7z archive files directly. So, It was smooth like butter to run the kernel.\n### But, recently kaggle is not able to read .7z archives directly after their config updates for the Notebook Environment. So, given below is a passage to overcome this hurdle. \n### We need to extract the 7z archive into a directory and work on it. Later, we need to delete it especially, if no. of files > 500. Read the instruction given below in three steps."},{"metadata":{},"cell_type":"markdown","source":"### Step [#1]: Install Python packages in an internet-enabled notebook"},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install pyunpack\n!pip install patool","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Step [#2]: Unpack your .7z file\n"},{"metadata":{"trusted":true,"_kg_hide-output":true,"_kg_hide-input":false},"cell_type":"code","source":"from pyunpack import Archive\nimport shutil\nif not os.path.exists('/kaggle/working/train/'):\n os.makedirs('/kaggle/working/train/')\nArchive('/kaggle/input/train.7z').extractall('/kaggle/working/train/')\nfor dirname, _, filenames in os.walk('/kaggle/working/train/'):\n for filename in filenames:\n print(os.path.join(dirname, filename))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### ALL archived files have been extracted"},{"metadata":{},"cell_type":"markdown","source":"### Step [#3]: Then after you are finished working with the images you can delete them so that your commit will succeed (max number of files in working directory for a commit = 500)"},{"metadata":{"trusted":true},"cell_type":"code","source":"shutil.make_archive('train/', 'zip', 'train')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Extracted files Zipped again in .zip format (readeable for kaggle kernels)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# deleting unwanted extracted files to avoid memory overflow (maxlimit files = 500) while commiting.\n!rm -rf kaggle/working/train/*","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Loading the trainig Input file.\ntrain_audio_path = \"/kaggle/working/train/train/audio\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"\"\"\"\n# It is just a checker code to validate the presence of file.\n\nprint(check_output([\"ls\", \"../input/train/audio\"]).decode(\"utf8\"))\nprint(os.listdir(\"../input/train\"))\n\n\"\"\"\n\n\nprint(check_output([\"ls\", \"/kaggle/working/train/train/audio\"]).decode(\"utf8\"))\nprint(os.listdir(\"/kaggle/working/train/train/audio/yes\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Example input file to be used here...\nfilename = '/yes/00f0204f_nohash_0.wav'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dirs = [f for f in os.listdir(train_audio_path) if isdir(join(train_audio_path, f))]\ndirs.sort()\nprint('Number of labels: ' + str(len(dirs)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"# Audio Features.\n\n#### sample_rate, samples = wavfile.read(str(train_audio_path) + filename) \n\n#### The above code line works fine for everything except **Librosa** library MFCC functionality. So, we'll read wave files using librosa only."},{"metadata":{},"cell_type":"markdown","source":"### Must to read samples in librosa format. Other wise \"librosa\" error: data must be in floating format"},{"metadata":{"trusted":true},"cell_type":"code","source":"samples, sample_rate = librosa.load(str(train_audio_path)+filename)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 1. Visualization\n\nThere are two theories of a [human hearing - place](https://en.wikipedia.org/wiki/Place_theory_(hearing) (frequency-based) and [temporal](https://en.wikipedia.org/wiki/Temporal_theory_(hearing) In speech recognition, I see two main tendencies - to input spectrogram (frequencies), and more sophisticated features MFCC - Mel-Frequency Cepstral Coefficients, PLP. You rarely work with raw, temporal data.\n\nLet's visualize some recordings!"},{"metadata":{},"cell_type":"markdown","source":"### 1.1 Spectogram \n\nDefine a function that calculates spectrogram.\n\nNote, that we are taking logarithm of spectrogram values. It will make our plot much more clear, moreover, it is strictly connected to the way people hear. We need to assure that there are no 0 values as input to logarithm.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"def log_specgram(audio, sample_rate, window_size=20,\n step_size=10, eps=1e-10):\n nperseg = int(round(window_size * sample_rate / 1e3))\n noverlap = int(round(step_size * sample_rate / 1e3))\n freqs, times, spec = signal.spectrogram(audio,\n fs=sample_rate,\n window='hann',\n nperseg=nperseg,\n noverlap=noverlap,\n detrend=False)\n return freqs, times, np.log(spec.T.astype(np.float32) + eps)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Frequencies are in range (0, 8000) according to Nyquist theorem.\n\nLet's plot it:"},{"metadata":{"trusted":true},"cell_type":"code","source":"freqs, times, spectrogram = log_specgram(samples, sample_rate)\n\nfig = plt.figure(figsize=(14, 8))\nax1 = fig.add_subplot(211)\nax1.set_title('Raw wave of ' + filename)\nax1.set_ylabel('Amplitude')\nax1.plot(np.linspace(0, sample_rate/len(samples), sample_rate), samples)\n\nax2 = fig.add_subplot(212)\nax2.imshow(spectrogram.T, aspect='auto', origin='lower', \n extent=[times.min(), times.max(), freqs.min(), freqs.max()])\nax2.set_yticks(freqs[::16])\nax2.set_xticks(times[::16])\nax2.set_title('Spectrogram of ' + filename)\nax2.set_ylabel('Freqs in Hz')\nax2.set_xlabel('Seconds')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"normalizing the audio data. Always a good plan if we gonna feed it into NN."},{"metadata":{"trusted":true},"cell_type":"code","source":"mean = np.mean(spectrogram, axis=0)\nstd = np.std(spectrogram, axis=0)\nspectrogram = (spectrogram - mean) / std","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There is an interesting fact to point out. We have ~160 features for each frame, frequencies are between 0 and 8000. It means, that one feature corresponds to 50 Hz. However, frequency resolution of the ear is 3.6 Hz within the octave of 1000 – 2000 Hz It means, that people are far more precise and can hear much smaller details than those represented by spectrograms like above."},{"metadata":{},"cell_type":"markdown","source":"### 1.2 MFCC\n\nIf you want to get to know some details about MFCC take a look at this great tutorial. MFCC explained You can see, that it is well prepared to imitate human hearing properties.\n\nYou can calculate Mel power spectrogram and MFCC using for example librosa python package.\n"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# From this tutorial\n# https://github.com/librosa/librosa/blob/master/examples/LibROSA%20demo.ipynb\nS = librosa.feature.melspectrogram(samples, sr=sample_rate, n_mels=128)\n\n# Convert to log scale (dB). We'll use the peak power (max) as reference.\nlog_S = librosa.power_to_db(S, ref=np.max)\n\nplt.figure(figsize=(12, 4))\nlibrosa.display.specshow(log_S, sr=sample_rate, x_axis='time', y_axis='mel')\nplt.title('Mel power spectrogram ')\nplt.colorbar(format='%+02.0f dB')\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Now delta- mfcc"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"mfcc = librosa.feature.mfcc(S=log_S, n_mfcc=13)\n\n# Let's pad on the first and second deltas while we're at it\ndelta2_mfcc = librosa.feature.delta(mfcc, order=2)\n\nplt.figure(figsize=(12, 4))\nlibrosa.display.specshow(delta2_mfcc)\nplt.ylabel('MFCC coeffs')\nplt.xlabel('Time')\nplt.title('MFCC')\nplt.colorbar()\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In classical, but still state-of-the-art systems, MFCC or similar features are taken as the input to the system instead of spectrograms.\n\nHowever, in end-to-end (often neural-network based) systems, the most common input features are probably raw spectrograms, or mel power spectrograms. For example MFCC decorrelates features, but NNs deal with correlated features well. Also, if you'll understand mel filters, you may consider their usage sensible.a\n\nIt is your decision which to choose!"},{"metadata":{},"cell_type":"markdown","source":"### 1.3 Silence Removal"},{"metadata":{"trusted":true},"cell_type":"code","source":"samples_cut = samples[4000:13000]\nipd.Audio(samples_cut, rate=sample_rate)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### We can agree that the entire word can be heard. It is impossible to cut all the files manually and do this basing on the simple plot. But you can use for example webrtcvad package to have a good VAD.\n\nLet's plot it again, together with guessed alignment of 'y' 'e' 's' graphems"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"freqs, times, spectrogram_cut = log_specgram(samples_cut, sample_rate)\n\nfig = plt.figure(figsize=(14, 8))\nax1 = fig.add_subplot(211)\nax1.set_title('Raw wave of ' + filename)\nax1.set_ylabel('Amplitude')\nax1.plot(samples_cut)\n\nax2 = fig.add_subplot(212)\nax2.set_title('Spectrogram of ' + filename)\nax2.set_ylabel('Frequencies * 0.1')\nax2.set_xlabel('Samples')\nax2.imshow(spectrogram_cut.T, aspect='auto', origin='lower', \n extent=[times.min(), times.max(), freqs.min(), freqs.max()])\nax2.set_yticks(freqs[::16])\nax2.set_xticks(times[::16])\nax2.text(0.06, 1000, 'Y', fontsize=18)\nax2.text(0.17, 1000, 'E', fontsize=18)\nax2.text(0.36, 1000, 'S', fontsize=18)\n\nxcoords = [0.025, 0.11, 0.23, 0.49]\nfor xc in xcoords:\n ax1.axvline(x=xc*16000, c='r')\n ax2.axvline(x=xc, c='r')\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 1.4 Features extraction steps\n\nA generalized feature extraction algorithm for an audio data sample be like that:\n\n 1. Resampling\n 2. VAD\n 3. Maybe padding with 0 to make signals be equal length\n 4. Log spectrogram (or MFCC, or PLP)\n 5. Features normalization with mean and std\n 6. Stacking of a given number of frames to get temporal information\n\nIt's a pity it can't be done in notebook. It has not much sense to write things from zero, and everything is ready to take, but in packages, that can not be imported in Kernels.\n"},{"metadata":{},"cell_type":"markdown","source":"## 2. Voice Activity Detection ( VAD )"},{"metadata":{},"cell_type":"markdown","source":"### use the webrtcvad library to identify segments as speech or not"},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install webrtcvad","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import webrtcvad","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### reading the samples and sample_rate feature again to make them compatible with the webrtcvad library. ( it reads at sample_rate = 16000, 32000, 48000; but we had sample_rate = 22050 with librosa)"},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_rate, samples = wavfile.read(str(train_audio_path) + filename)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"vad = webrtcvad.Vad()\n\n# set aggressiveness from 0 to 3\nvad.set_mode(3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### convert samples to raw 16 bit per sample stream needed by webrtcvad( there are other options available too , like 32 )"},{"metadata":{"trusted":true},"cell_type":"code","source":"import struct\nraw_samples = struct.pack(\"%dh\" % len(samples), *samples)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.1 run the detector on windows of 30 ms \n[example here](https://github.com/wiseman/py-webrtcvad/blob/master/example.py)"},{"metadata":{"trusted":true},"cell_type":"code","source":"window_duration = 0.03 # duration in seconds\n\nsamples_per_window = int(window_duration * sample_rate + 0.5)\n\nbytes_per_sample = 2","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.2 Detect Speech instances in an audio"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"segments = []\n\nfor start in np.arange(0, len(samples), samples_per_window):\n stop = min(start + samples_per_window, len(samples))\n \n is_speech = vad.is_speech(raw_samples[start * bytes_per_sample: stop * bytes_per_sample], \n sample_rate = sample_rate)\n\n segments.append(dict(\n start = start,\n stop = stop,\n is_speech = is_speech))\n \n ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"### 2.3 plot the range of samples identified as speech in orange"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize = (10,7))\nplt.plot(samples)\n\nymax = max(samples)\n\n\n# plot segment identifed as speech\nfor segment in segments:\n if segment['is_speech']:\n plt.plot([ segment['start'], segment['stop'] - 1], [ymax * 1.1, ymax * 1.1], color = 'orange')\n\nplt.xlabel('sample')\nplt.grid()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3. Listen to the speech only segments"},{"metadata":{"trusted":true},"cell_type":"code","source":"speech_samples = np.concatenate([ samples[segment['start']:segment['stop']] for segment in segments if segment['is_speech']])\n\nimport IPython.display as ipd\nipd.Audio(speech_samples, rate=sample_rate)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Till now we have processed for a single audio of any one word : YES here.\n\n#### Now, its time to have an overall view on other words also. So, lets visualize frequency components for other words as well."},{"metadata":{},"cell_type":"markdown","source":"## 4. Frequency components across the words\n\n### plotting for first 8 words only to avoid clumsy tight plots."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def violinplot_frequency(dirs, freq_ind):\n \"\"\" Plot violinplots for given words (waves in dirs) and frequency freq_ind\n from all frequencies freqs.\"\"\"\n\n spec_all = [] # Contain spectrograms\n ind = 0\n # taking first 8 words only to keep the plots clean and unclumsy.\n for direct in dirs[:8]:\n spec_all.append([])\n\n waves = [f for f in os.listdir(join(train_audio_path, direct)) if\n f.endswith('.wav')]\n for wav in waves[:100]:\n sample_rate, samples = wavfile.read(\n train_audio_path + '/' + direct + '/' + wav)\n freqs, times, spec = log_specgram(samples, sample_rate)\n spec_all[ind].extend(spec[:, freq_ind])\n ind += 1\n\n # Different lengths = different num of frames. Make number equal\n minimum = min([len(spec) for spec in spec_all])\n spec_all = np.array([spec[:minimum] for spec in spec_all])\n\n plt.figure(figsize=(13,7))\n plt.title('Frequency ' + str(freqs[freq_ind]) + ' Hz')\n plt.ylabel('Amount of frequency in a word')\n plt.xlabel('Words')\n sns.violinplot(data=pd.DataFrame(spec_all.T, columns=dirs[:8]))\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"violinplot_frequency(dirs, 20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"violinplot_frequency(dirs, 50)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"violinplot_frequency(dirs, 120)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# I hope you liked this kernel. Please UPVOTE to show your LOVE.\n\n# Your feedbacks are always heartly welcomed."}],"metadata":{"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","nbconvert_exporter":"python","mimetype":"text/x-python","name":"python","pygments_lexer":"ipython3","version":"3.4.5"},"kernelspec":{"language":"python","name":"python3","display_name":"Python 3"}},"nbformat":4,"nbformat_minor":1}