{"cells":[{"metadata":{},"cell_type":"markdown","source":"To classify audio, you first need present it somehow to the classifier. You may notice everyone is talking about **spectrogram, FFT, STFT, MFCC**, but why don't we **just use audio?** What does it all stand for?\n\n### **Here comes a little explanation!**\n\ntip: most interesting things are marked as **QUESTION**"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import os\nfrom os.path import isdir, join\nfrom pathlib import Path\nimport pandas as pd\n\n# Math\nimport numpy as np\nfrom scipy.fftpack import fft\nfrom scipy import signal\nfrom scipy.io import wavfile\nimport librosa\n\nfrom sklearn.decomposition import PCA\n\n# Visualization\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport IPython.display as ipd\nimport librosa.display\n\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nimport plotly.tools as tls\nimport pandas as pd\n\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Samples\n\nWe can load the first two samples we'll be working on"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"train_audio_path = '../input/train_curated/' \n# 8a8110c2 c2aff189 d7d25898 0a2895b8 6459fc05 54940c5c 024e0fbe c6f8f09e f46cc65b  \n# 1acaf122 a0a85eae da3a5cd5 412c28dd 0f301184 2ce5262c\nsample_rate, samples1 = wavfile.read(os.path.join(train_audio_path, '98b0df76.wav'))\nsample_rate, samples2 = wavfile.read(os.path.join(train_audio_path, 'd7d25898.wav'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"How do they sound?"},{"metadata":{"trusted":true},"cell_type":"code","source":"ipd.Audio(samples1, rate=sample_rate)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ipd.Audio(samples2, rate=sample_rate)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_raw_wave(samples):\n    plt.figure(figsize=(14, 3))\n    plt.title('Raw wave')\n    plt.ylabel('Amplitude')\n    # ax1.plot(np.linspace(0, sample_rate/len(samples1), sample_rate), samples1)\n    plt.plot(samples)\n    plt.show()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_raw_wave(samples1)\nplot_raw_wave(samples2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"You can easily SEE 2 beeps in the second wave, and constant noise in first one. The problem is that it is impossible to understand or interpret the pitch of a sound watching physical illustration as above."},{"metadata":{},"cell_type":"markdown","source":"The first thing we need to understand: the sample rate. yYou can read about it [here](https://en.wikipedia.org/wiki/Sampling_(signal_processing))\n\n## Sampling rate: Intuition\n\nSo wait, does it mean one second of a recording has 44100 samples ('features'), so the longest recordings will have _30*44100 = 1323000_ elements? Yes. That's one of the reasons why we need some different representation of an audio signal.\n\n\nBut hey, if this fluctuations are what directly come into the human ear, shouldn't we use it directly? Not really. Human hearing is a tough topic, not well understood, but we can assume that our brain hears rather something like frequencies ( [Reference](https://en.wikipedia.org/wiki/Place_theory_(hearing)) )\n\n\n## What is frequency?\nAn explanation [here](http://www.indiana.edu/~emusic/etext/acoustics/chapter1_frequency.shtml) and a nice tutorial about calculating the frequencies here: [link](https://www.youtube.com/watch?v=r18Gi8lSkfM). The mathematical tool for that is Fast Fourier Transform (FFT)\n\nBriefly: \n- low sounds are long waves\n- higher sounds are shorter waves\n\n**To calculate frequencies, we calculate the amount of long waves and the amount of short waves ** \n\nWe analyze our two examples: the one with \"low\" sounds, and one with high pitches."},{"metadata":{"trusted":true},"cell_type":"code","source":"def custom_fft(y, fs):\n    T = 1.0 / fs\n    N = y.shape[0]\n    yf = fft(y)\n    xf = np.linspace(0.0, 1.0/(2.0*T), N//2)\n    vals = 2.0/N * np.abs(yf[0:N//2])  # FFT is simmetrical, so we take just the first half\n    # FFT is also complex, to we take just the real part (abs)\n    return xf, vals\n\ndef plot_custom_fft(samples, sample_rate):\n    xf, vals = custom_fft(samples, sample_rate)\n    plt.figure(figsize=(12, 4))\n    plt.title('FFT of recording sampled with ' + str(sample_rate) + ' Hz')\n    plt.plot(xf, vals)\n    plt.xlabel('Frequency')\n    plt.grid()\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_custom_fft(samples1, sample_rate)\nplot_custom_fft(samples2, sample_rate)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Great to see that FFT really shows bigger amplitude in low freqs in machine noise and big amplitude in high freqs for beeps."},{"metadata":{},"cell_type":"markdown","source":"**FFT is not enough**\n\nWe can indeed calculate all the frequencies in audio. But sound differs in time, so maybe we should calculate the frequencies for a small part of a signal, to show the time dependencies?\n\nWe can easily do that. We call the result **spectrogram**\n\nHow to create spectrogram in 3 steps:\n1. Cut the signal by overlapping windows 20 ms windows\n2. Find short and long waves (frequencies in this window) using FFT\n3. Concatenate the calculated frequencies. \n\n\nLet's implement it:"},{"metadata":{"trusted":true},"cell_type":"code","source":"def log_specgram(audio, sample_rate, window_size=20,\n                 step_size=10, eps=1e-10):\n    nperseg = int(round(window_size * sample_rate / 1e3))\n    noverlap = int(round(step_size * sample_rate / 1e3))\n    freqs, times, spec = signal.spectrogram(audio,\n                                    fs=sample_rate,\n                                    window='hann',\n                                    nperseg=nperseg,\n                                    noverlap=noverlap,\n                                    detrend=False)\n    return freqs, times, np.log(spec.T.astype(np.float32) + eps)\n\ndef plot_log_specgram(audio, sample_rate, window_size=20, step_size=10, eps=1e-10):\n    \n    fig = plt.figure(figsize=(14, 3))\n    freqs, times, spectrogram = log_specgram(audio, sample_rate)\n    plt.imshow(spectrogram.T, aspect='auto', origin='lower', \n               extent=[times.min(), times.max(), freqs.min(), freqs.max()])\n    plt.yticks(freqs[::16])\n    plt.xticks(times[::16])\n    plt.title('Spectrogram')\n    plt.ylabel('Freqs in Hz')\n    plt.xlabel('Seconds')\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And now we visualize our two examples using the spectrogram"},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_log_specgram(samples1, sample_rate)\nplot_log_specgram(samples2, sample_rate)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Important:\n\nThis is really a representation that **reflects and takes into account the audible properties of a signal**. Do you agree?"},{"metadata":{},"cell_type":"markdown","source":"## **Nyquist theorem**\n\nInteresting property: according to [Nyquist theorem](https://en.wikipedia.org/wiki/Nyquist_rate) we can hear only frequencies twice lower than the sampling rate.\n\nI not going to present this fact here, but what I want to say is:\n\n#### ** QUESTION!!! **\n\n**Do we really need 44100 samples per sec (frequencies between 0 and 22.1k?)**\n\nEven your laptop speakers can't pass these freqs, but you can crearly classify the stuff well, so maybe downsampling is a good idea?"},{"metadata":{},"cell_type":"markdown","source":"#### ** QUESTION!!! **\n\n**Why we split signal in 20 ms parts?**\n\n20 ms is a mean time of the shortest speech p**art, phonem. But wait, it's not a speech recognition competition, so maybe we should use longer windows? I think we should"},{"metadata":{},"cell_type":"markdown","source":"## MFCC\n<a id=\"mfcc\"></a> \n\nIf you want to get to know some details about *MFCC* take a look at this great tutorial. [MFCC explained](http://practicalcryptography.com/miscellaneous/machine-learning/guide-mel-frequency-cepstral-coefficients-mfccs/) You can see, that it is well prepared to imitate human hearing properties.\n\nYou can calculate *Mel power spectrogram* and *MFCC* using, for example, *librosa* python package.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# From this tutorial\n# https://github.com/librosa/librosa/blob/master/examples/LibROSA%20demo.ipynb\nS = librosa.feature.melspectrogram(samples1.astype(float), sr=sample_rate, n_mels=128)\n\n# Convert to log scale (dB). We'll use the peak power (max) as reference.\nlog_S = librosa.power_to_db(S, ref=np.max)\n\nplt.figure(figsize=(12, 4))\nlibrosa.display.specshow(log_S, sr=sample_rate, x_axis='time', y_axis='mel')\nplt.title('Mel power spectrogram ')\nplt.colorbar(format='%+02.0f dB')\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mfcc = librosa.feature.mfcc(S=log_S, n_mfcc=13)\n\n# Let's pad on the first and second deltas while we're at it\ndelta2_mfcc = librosa.feature.delta(mfcc, order=2)\n\nplt.figure(figsize=(12, 4))\nlibrosa.display.specshow(delta2_mfcc)\nplt.ylabel('MFCC coeffs')\nplt.xlabel('Time')\nplt.title('MFCC')\nplt.colorbar()\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Explanation: \n\nHuman hearing is complex - we hear some frequencies more than other - and we can imitate it using the above techniques. \n\n\n#### ** QUESTION!!! **\n\n**Should we use MFCC? **\n\nWe evolved to here the environment and the speech, so I think we may not have the best detectors for some artificial signals, but I feel good at the classification with my natural filters, so I would USE MFCC.\n\nAnd MFCC decorrelates the features! It's a nice property because the features are more distinct and tell more clear things."},{"metadata":{},"cell_type":"markdown","source":"## Bonus - spectrogram in 3d"},{"metadata":{"trusted":true},"cell_type":"code","source":"freqs, times, spectrogram = log_specgram(samples2, sample_rate)\ndata = [go.Surface(z=spectrogram.T)]\nlayout = go.Layout(\n    title='Specgtrogram of \"yes\" in 3d',\n#     scene = dict(\n#     yaxis = dict(title='Frequencies', range=freqs),\n#     xaxis = dict(title='Time', range=times),\n#     zaxis = dict(title='Log amplitude'),\n#     ),\n)\nfig = go.Figure(data=data, layout=layout)\npy.iplot(fig)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"---\n\n**If you like my work please upvote.**\n\nLeave a feedback that will let me improve! "}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}