{"metadata": {"language_info": {"file_extension": ".py", "version": "3.6.3", "name": "python", "nbconvert_exporter": "python", "mimetype": "text/x-python", "pygments_lexer": "ipython3", "codemirror_mode": {"version": 3, "name": "ipython"}}, "kernelspec": {"name": "conda-env-python3-py", "language": "python", "display_name": "Python [conda env:python3]"}}, "cells": [{"metadata": {}, "cell_type": "markdown", "source": ["# Explore Sound Visualization of Low Amplitude Ranges\n", "##  Using \"seven\" utterances\n", "\n", "Earl F Glynn\n", "\n", "2017-12-10"]}, {"metadata": {}, "cell_type": "markdown", "source": ["# Purpose"]}, {"metadata": {}, "cell_type": "markdown", "source": ["Explore \n", "\n", " * What is \"silence\"?  \n", "  \n", " * How do spectrograms change when the amplitude range changes?\n", " \n", " * Is there an amplitude range below which wav files can be deemed to be \"silence.\"\n", "\n", "Based on [Sound-Visualization notebook](https://github.com/EarlGlynn/kaggle-speech-recognition/tree/master/Jupyter/01-Sound-WAV-visualization), which was adapted from Kaggle [Data visualization and investigation](https://www.kaggle.com/davids1992/data-visualization-and-investigation) by *DavisS*.\n", "\n", "Find files of given amplitude min, max, range in the train/test WAV-File-Inventory files from [this Kaggle discussion thread](https://www.kaggle.com/c/tensorflow-speech-recognition-challenge/discussion/44687)."]}, {"metadata": {}, "cell_type": "markdown", "source": ["# Conclusions\n", "\n", "* An amplitude range cutoff of perhaps 500 or less could be used to label a wav file as \"silence.\"  This needs additional validation.\n", "\n", "* Some noise filtering may be necessary.\n", "\n", "* Time alignment at the beginning of an utternace seems necessary."]}, {"metadata": {}, "cell_type": "markdown", "source": ["# Setup"]}, {"metadata": {"collapsed": true, "scrolled": false}, "outputs": [], "source": ["import pandas as pd"], "cell_type": "code", "execution_count": 1}, {"metadata": {}, "cell_type": "markdown", "source": ["Math"]}, {"metadata": {"collapsed": true, "scrolled": false}, "outputs": [], "source": ["import librosa\n", "import numpy    as np\n", "\n", "from scipy.fftpack         import fft\n", "from scipy                 import signal\n", "from scipy.io              import wavfile"], "cell_type": "code", "execution_count": 2}, {"metadata": {}, "cell_type": "markdown", "source": ["Visualization"]}, {"metadata": {"scrolled": false}, "outputs": [], "source": ["import matplotlib.pyplot as plt\n", "import pandas            as pd\n", "\n", "import IPython.display   as ipd\n", "import librosa.display\n", "\n", "import plotly.offline    as py\n", "\n", "py.init_notebook_mode(connected=True)\n", "\n", "%matplotlib inline"], "cell_type": "code", "execution_count": 3}, {"metadata": {}, "cell_type": "markdown", "source": ["Location of train audio files"]}, {"metadata": {"collapsed": true, "scrolled": true}, "outputs": [], "source": ["trainAudioPath = '../input/train/audio/'"], "cell_type": "code", "execution_count": 4}, {"metadata": {}, "cell_type": "markdown", "source": ["## Helper Functions"]}, {"metadata": {}, "cell_type": "markdown", "source": ["[Format output of code cell with Markdown](https://stackoverflow.com/questions/32026727/format-output-of-code-cell-with-markdown)"]}, {"metadata": {"collapsed": true, "scrolled": true}, "outputs": [], "source": ["from IPython.display import Markdown, display\n", "def printMarkdown(string):\n", "    display(Markdown(string))"], "cell_type": "code", "execution_count": 5}, {"metadata": {}, "cell_type": "markdown", "source": ["Raw Wave File"]}, {"metadata": {"collapsed": true, "scrolled": false}, "outputs": [], "source": ["def plotRawWave(plotTitle, sampleRate, samples, figWidth=14, figHeight=4):\n", "    plt.figure(figsize=(figWidth, figHeight))\n", "    plt.plot(np.linspace(0, sampleRate/len(samples), sampleRate), samples)\n", "    plt.title(\"Raw sound wave of \" + plotTitle)\n", "    plt.ylabel(\"Amplitude\")\n", "    plt.xlabel(\"Time [sec]\")\n", "    plt.show()  # force display while in for loop\n", "    return None"], "cell_type": "code", "execution_count": 6}, {"metadata": {}, "cell_type": "markdown", "source": ["Spectrogram"]}, {"metadata": {"collapsed": true, "scrolled": false}, "outputs": [], "source": ["def computeLogSpectrogram(audio, sampleRate, windowSize=20, stepSize=10, epsilon=1e-10):\n", "    nperseg  = int(round(windowSize * sampleRate / 1000))\n", "    noverlap = int(round(stepSize   * sampleRate / 1000))\n", "    freqs, times, spec = signal.spectrogram(audio,\n", "                                            fs=sampleRate,\n", "                                            window='hann',\n", "                                            nperseg=nperseg,\n", "                                            noverlap=noverlap,\n", "                                            detrend=False)\n", "    return freqs, times, np.log(spec.T.astype(np.float32) + epsilon)"], "cell_type": "code", "execution_count": 7}, {"metadata": {"collapsed": true, "scrolled": false}, "outputs": [], "source": ["def plotLogSpectrogram(plotTitle, freqs, times, spectrogram, figWidth=14, figHeight=4):\n", "    fig = plt.figure(figsize=(figWidth, figHeight))\n", "    plt.imshow(spectrogram.T, aspect='auto', origin='lower', \n", "               cmap=\"inferno\",   #  default was \"viridis\"  (perceptually uniform)\n", "               extent=[times.min(), times.max(), freqs.min(), freqs.max()])\n", "    plt.colorbar(pad=0.01)\n", "    plt.title('Spectrogram of ' + plotTitle)\n", "    plt.ylabel(\"Frequency [Hz]\")\n", "    plt.xlabel(\"Time [sec]\")\n", "    fig.tight_layout()\n", "    plt.show()  # force display while in for loop\n", "    return None"], "cell_type": "code", "execution_count": 8}, {"metadata": {}, "cell_type": "markdown", "source": ["Mel Spectrogram"]}, {"metadata": {"collapsed": true, "scrolled": false}, "outputs": [], "source": ["def computeLogMelSpectrogram(samples, sampleRate, nMels=128):\n", "    melSpectrum = librosa.feature.melspectrogram(samples, sr=sampleRate, n_mels=nMels)\n", "    \n", "    # Convert to dB, which is a log scale.  Use peak power as reference.\n", "    logMelSpectrogram = librosa.power_to_db(melSpectrum, ref=np.max)\n", "    \n", "    return logMelSpectrogram"], "cell_type": "code", "execution_count": 9}, {"metadata": {"collapsed": true, "scrolled": false}, "outputs": [], "source": ["def plotLogMelSpectrogram(plotTitle, sampleRate, logMelSpectrum, figWidth=14, figHeight=4):\n", "    fig = plt.figure(figsize=(figWidth, figHeight))\n", "    librosa.display.specshow(logMelSpectrum, sr=sampleRate, x_axis='time', y_axis='mel')\n", "    plt.title('Mel log-frequency power spectrogram: ' + plotTitle)\n", "    plt.colorbar(pad=0.01, format='%+02.0f dB')\n", "    plt.tight_layout()  \n", "    plt.show()  # force display while in for loop\n", "    return None"], "cell_type": "code", "execution_count": 10}, {"metadata": {}, "cell_type": "markdown", "source": ["Compute and plot MFCC"]}, {"metadata": {"collapsed": true, "scrolled": false}, "outputs": [], "source": ["def computeMFCC(samples, sampleRate, nFFT=512, hopLength=256, nMFCC=40):\n", "    mfcc = librosa.feature.mfcc(y=samples, sr=sampleRate, \n", "                                n_fft=nFFT, hop_length=hopLength, n_mfcc=nMFCC)\n", "    \n", "    # Let's add on the first and second deltas  (what is this really doing?)\n", "    #mfcc = librosa.feature.delta(mfcc, order=2)\n", "    return mfcc"], "cell_type": "code", "execution_count": 11}, {"metadata": {"collapsed": true, "scrolled": false}, "outputs": [], "source": ["def plotMFCC(plotTitle, sampleRate, mfcc, figWidth=14, figHeight=4):\n", "    fig = plt.figure(figsize=(figWidth, figHeight))\n", "    librosa.display.specshow(mfcc, sr=sampleRate, x_axis='time', y_axis='mel')\n", "    plt.colorbar(pad=0.01)\n", "    plt.title(\"Mel-frequency cepstral coefficients (MFCC): \" + plotTitle)\n", "    plt.tight_layout()\n", "    plt.show()  # force display while in for loop\n", "    return None"], "cell_type": "code", "execution_count": 12}, {"metadata": {}, "cell_type": "markdown", "source": ["Plot waveform along with the above three spectrum alternatives"]}, {"metadata": {"collapsed": true, "scrolled": false}, "outputs": [], "source": ["def showWavefile(filename):\n", "    sampleRate, samples = wavfile.read(filename)  \n", "    plotRawWave(filename, sampleRate, samples)\n", "    \n", "    freqs, times, logSpectrogram = computeLogSpectrogram(samples, sampleRate)\n", "    plotLogSpectrogram(filename, freqs, times, logSpectrogram)\n", "    \n", "    logMelSpectrogram = computeLogMelSpectrogram(samples, sampleRate)\n", "    plotLogMelSpectrogram(filename, sampleRate, logMelSpectrogram)\n", "    \n", "    mfcc = computeMFCC(samples, sampleRate)\n", "    #print(mfcc.shape)\n", "    plotMFCC(filename, sampleRate, mfcc)\n", "    \n", "    return sampleRate, samples, logSpectrogram, logMelSpectrogram, mfcc"], "cell_type": "code", "execution_count": 13}, {"metadata": {}, "cell_type": "markdown", "source": ["# Low Amplitude Waveforms\n", "\n", "... and a some high amplitude waveforms for comparison.\n", "\n", "List files to explore in data structure .\n", "\n", "Here, we look for amplitude ranges near powers of 2.\n", "\n", "For now, only works with files with exactly 16000 samples."]}, {"metadata": {"scrolled": false}, "outputs": [], "source": ["labels = ['filename', 'comments']\n", "waves  = [\n", "          ('seven/712e4d58_nohash_2.wav',  # 10\n", "           'Noise.'),\n", "    \n", "          ('seven/099d52ad_nohash_4.wav',  # 29\n", "           'Noise.'),\n", "\n", "          ('seven/ced835d3_nohash_3.wav',  # 132\n", "           'Unintelligible, noise.'),\n", "\n", "          ('seven/7846fd85_nohash_0.wav',  # 265\n", "           'Unintelligible, noise.'),\n", "\n", "          ('seven/aff582a1_nohash_0.wav',  # 591\n", "           'clicks with noise.' ),\n", " \n", "          ('seven/3c165869_nohash_0.wav',  # 1049\n", "           '\"seven\"'),  \n", "    \n", "          ('seven/e82914c0_nohash_0.wav',  # 2084\n", "           '\"seven\"'),\n", "\n", "          ('seven/fb7cfe0e_nohash_0.wav',  # 4114\n", "           '\"seven\"'),\n", "                                                                                                    \n", "          ('seven/9ff1b8b6_nohash_1.wav',  # 8193\n", "           'Muffled \"seven\"'),\n", "    \n", "          ('seven/28ed6bc9_nohash_4.wav',  # 16440\n", "           '\"seven\"'),\n", "\n", "           ('seven/471a0925_nohash_3.wav',  # 32800\n", "           'noisy \"seven\"'),\n", "                                                                                  \n", "          ('seven/facd97c0_nohash_0.wav',  # 65535\n", "           'LOUD \"seven\"')\n", "         ]                  \n", "\n", "wavedf = pd.DataFrame.from_records(waves, columns=labels)\n", "wavedf"], "cell_type": "code", "execution_count": 14}, {"metadata": {}, "cell_type": "markdown", "source": ["# Display waveforms and related spectrograms"]}, {"metadata": {}, "cell_type": "markdown", "source": ["Need this trick below:  [More than one Audio object in a Jupyter Notebook cell](https://stackoverflow.com/questions/33048353/more-than-one-audio-object-in-a-jupyter-ipython-notebook-cell)."]}, {"metadata": {"scrolled": false}, "outputs": [], "source": ["for i in range(wavedf.shape[0]):\n", "    \n", "    filenameShort = wavedf.loc[i, 'filename']\n", "    word, wavename = filenameShort.split(\"/\")\n", "    \n", "    filename = trainAudioPath + filenameShort\n", "    \n", "    sampleRate, samples = wavfile.read(filename)  \n", "    caption = '\"' + word  + \\\n", "    '\" [amplitude from ' + str(min(samples)) +  ' to ' + str(max(samples)) + \\\n", "    ', range = ' + str(int(max(samples)) - int(min(samples))) + ']'\n", "    \n", "    printMarkdown(\"# \" + caption)\n", "    print(wavedf.loc[i, 'comments'])\n", "    \n", "    ipd.display( ipd.Audio(filename) )\n", "    sampleRate, samples, logSpectrogram, logMelSpectrogram, mfcc = showWavefile(filename)"], "cell_type": "code", "execution_count": 15}, {"metadata": {"collapsed": true, "scrolled": true}, "outputs": [], "source": [], "cell_type": "code", "execution_count": null}], "nbformat": 4, "nbformat_minor": 1}