{"cells": [{"source": ["## Introduction \n", "\n", "Hello! This is my very first Kernel. I'd like to take a look on the dataset. It is meant to give a grasp of a problem of speech representation.\n", "\n", "1. [Visualization of the recordings - input features](#visualization)<br>\n", "    1.1. [Wave and spectrogram](#waveandspectrogram)<br>\n", "    1.2. [Silence removal](#silenceremoval)<br>\n", "    1.3. [MFCC](#mfcc)<br>\n", "    1.4. [Features extraction steps](#featuresextractionsteps)<br>\n", "2. [Dataset investigation](#investigations)<br>\n", "    2.1. [Number of files](#numberoffiles)<br>\n", "    2.2. [Mean spectrograms and fft](#meanspectrogramsandfft)<br>\n", "    2.3. [Deeper into recordings](#deeper)<br>\n", "    2.4. [Length of recordings](#len)<br>\n", "    2.5. [Note on Gaussian Mixtures modeling](#gmms)<br>\n", "    2.6. [Frequency components across the words](#components)<br>\n", "    2.7. [Anomaly detection](#anomaly)<br>\n", "3. [Where to look for the inspiration](#wheretostart)<br>\n", "\n", "All we need is here:"], "metadata": {"_cell_guid": "cec0e376-8e36-4751-8b53-2f298fda3a47", "_uuid": "cf808b6b464476e44c8dda4beada9884cf91adca"}, "cell_type": "markdown"}, {"metadata": {"_cell_guid": "8fd82027-7be0-4a4e-a921-b8acacaaf077", "_uuid": "d9596d80cb6445d4214dda15e40d777cadbd4669"}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["import os\n", "from os.path import isdir, join\n", "from pathlib import Path\n", "import pandas as pd\n", "\n", "# Math\n", "import numpy as np\n", "from scipy.fftpack import fft\n", "from scipy import signal\n", "from scipy.io import wavfile\n", "\n", "from sklearn.decomposition import PCA\n", "\n", "# Visualization\n", "import matplotlib.pyplot as plt\n", "import seaborn as sns\n", "import IPython.display as ipd\n", "\n", "import plotly.offline as py\n", "py.init_notebook_mode(connected=True)\n", "import plotly.graph_objs as go\n", "import plotly.tools as tls\n", "import pandas as pd\n", "\n", "%matplotlib inline"]}, {"source": ["\n", "# 1. Visualization \n", "\n", "There are two theories of a human hearing - [place](https://en.wikipedia.org/wiki/Place_theory_(hearing) (frequency-based) and [temporal](https://en.wikipedia.org/wiki/Temporal_theory_(hearing))\n", "In speech recognition, I see two main tendencies - to input [spectrogram](https://en.wikipedia.org/wiki/Spectrogram) (frequencies), and more sophisticated features MFCC - Mel-Frequency Cepstral Coefficients, PLP. You rarely work with raw, temporal data.\n", "\n", "Let's visualize some recordings!\n", "\n", "## 1.1. Wave and spectrogram:\n", "\n", "Choose and read some file:"], "metadata": {"_cell_guid": "7f050711-6810-4aac-a306-da82fddb5579", "_uuid": "95fabaca63ab1a486bcc1f6824b26919ef325ff4"}, "cell_type": "markdown"}, {"metadata": {"collapsed": true, "_cell_guid": "02126a6d-dd84-4f0a-88eb-ed9ff46a9bdf", "_uuid": "76266716e7df45a83073fb2964218c85b36d31cb"}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["train_audio_path = '../input/train/audio/'\n", "filename = '/yes/0a7c2a8d_nohash_0.wav'\n", "sample_rate, samples = wavfile.read(str(train_audio_path) + filename)"]}, {"source": ["Define a function that calculates spectrogram.\n", "\n", "Note, that we are taking logarithm of spectrogram values. It will make our plot much more clear, moreover, it is strictly connected to the way people hear.\n", "We need to assure that there are no 0 values as input to logarithm."], "metadata": {"_cell_guid": "a7715152-3866-48dd-8bbb-31a72e9aa9bf", "_uuid": "3bc26d76ea9f627c4d476ff8e9523f37d0668bbf"}, "cell_type": "markdown"}, {"metadata": {"collapsed": true, "_cell_guid": "e464fe63-138e-4c66-a1f7-3ad3a81daa38", "_uuid": "a3569f66d5bbbdcf338eaa121328a507f3a7b431"}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["def log_specgram(audio, sample_rate, window_size=20,\n", "                 step_size=10, eps=1e-10):\n", "    nperseg = int(round(window_size * sample_rate / 1e3))\n", "    noverlap = int(round(step_size * sample_rate / 1e3))\n", "    freqs, times, spec = signal.spectrogram(audio,\n", "                                    fs=sample_rate,\n", "                                    window='hann',\n", "                                    nperseg=nperseg,\n", "                                    noverlap=noverlap,\n", "                                    detrend=False)\n", "    return freqs, times, np.log(spec.T.astype(np.float32) + eps)"]}, {"source": ["Frequencies are in range (0, 8000) according to [Niquist theorem](https://en.wikipedia.org/wiki/Nyquist_rate).\n", "\n", "Let's plot it:"], "metadata": {"_cell_guid": "625dcb59-00ec-4b3f-97d5-f8adc12ac61a", "_uuid": "4fd53946fd96b09765a267231ea5a66b313c2d4e"}, "cell_type": "markdown"}, {"metadata": {"_cell_guid": "4f77267a-1720-439b-9ef9-90e60f4446e1", "_uuid": "ec1d065704c51f7f2b5b49f00da64809257815ab"}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["freqs, times, spectrogram = log_specgram(samples, sample_rate)\n", "\n", "fig = plt.figure(figsize=(14, 8))\n", "ax1 = fig.add_subplot(211)\n", "ax1.set_title('Raw wave of ' + filename)\n", "ax1.set_ylabel('Amplitude')\n", "ax1.plot(np.linspace(0, sample_rate/len(samples), sample_rate), samples)\n", "\n", "ax2 = fig.add_subplot(212)\n", "ax2.imshow(spectrogram.T, aspect='auto', origin='lower', \n", "           extent=[times.min(), times.max(), freqs.min(), freqs.max()])\n", "ax2.set_yticks(freqs[::16])\n", "ax2.set_xticks(times[::16])\n", "ax2.set_title('Spectrogram of ' + filename)\n", "ax2.set_ylabel('Freqs in Hz')\n", "ax2.set_xlabel('Seconds')"]}, {"source": ["If we use spectrogram as an input features for NN, we have to remember to normalize features. (We need to normalize over all the dataset, here's example just for one, which doesn't give good *mean* and *std*!)"], "metadata": {"_cell_guid": "013846a9-a929-45d9-97f5-98c59c6b2f23", "_uuid": "8f36fd74c9ad998d71b3a8838347b1bdbe8c82a7"}, "cell_type": "markdown"}, {"metadata": {"collapsed": true, "_cell_guid": "9572b5e1-0b0f-42aa-934e-a1e313c21f46", "_uuid": "b3d09cf8bd1e91f54774f84dc952508d8a8a4eb8"}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["mean = np.mean(spectrogram, axis=0)\n", "std = np.std(spectrogram, axis=0)\n", "spectrogram = (spectrogram - mean) / std"]}, {"source": ["## 1.2. Silence removal\n", "Let's listen to that file"], "metadata": {"_cell_guid": "a2ad2019-f402-4226-9bad-65fb400aa8b1", "_uuid": "769e6738c4dae9923b9c0b0a99981bce8b443030"}, "cell_type": "markdown"}, {"metadata": {"_cell_guid": "f49b916e-53a2-4dbe-bd03-3d8d93bf25a6", "scrolled": false, "_uuid": "ab0145dc0c8efdc08b4153b136c2b78634f6ed07"}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["ipd.Audio(samples, rate=sample_rate)"]}, {"source": ["I consider that some *VAD* (Voice Activity Detection) will be really useful here. Although the words are short, there is a lot of silence in them. A decent *VAD* can reduce training size a lot, accelerating training speed significantly.\n", "Let's cut a bit of the file from the beginning and from the end. and listen to it again (based on a plot above, we take from 4000 to 13000):"], "metadata": {"_cell_guid": "4c23e8b3-0c8f-4eda-8f35-7486bdecfd9d", "_uuid": "9745fb19ce26c85c312a20e7fa19d98e672ceb64"}, "cell_type": "markdown"}, {"metadata": {"_cell_guid": "2c85e04d-cbd8-4702-bd50-7340497e800d", "_uuid": "539d123d84ac0181b820cca82c4098ab0ca54116"}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["samples_cut = samples[4000:13000]\n", "ipd.Audio(samples_cut, rate=sample_rate)"]}, {"source": ["We can agree that the entire word can be heard. It is impossible to cut all the files manually and do this basing on the simple plot. But you can use for example *webrtcvad* package to have a good *VAD*.\n", "\n", "Let's plot it again, together with guessed alignment of* 'y' 'e' 's'* graphems"], "metadata": {"_cell_guid": "45898236-4528-4e21-86dd-55abcf4f639f", "_uuid": "e9ceecbabecc11f789b3de382ee4c909186e6d22"}, "cell_type": "markdown"}, {"metadata": {"_cell_guid": "038fe488-4f25-42bd-af11-108f5ecbb1e7", "_uuid": "6831fa9311397dc8bca4192f657767d36c5c1a38"}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["freqs, times, spectrogram_cut = log_specgram(samples_cut, sample_rate)\n", "\n", "fig = plt.figure(figsize=(14, 8))\n", "ax1 = fig.add_subplot(211)\n", "ax1.set_title('Raw wave of ' + filename)\n", "ax1.set_ylabel('Amplitude')\n", "ax1.plot(samples_cut)\n", "\n", "ax2 = fig.add_subplot(212)\n", "ax2.set_title('Spectrogram of ' + filename)\n", "ax2.set_ylabel('Frequencies * 0.1')\n", "ax2.set_xlabel('Samples')\n", "ax2.imshow(spectrogram_cut.T, aspect='auto', origin='lower', \n", "           extent=[times.min(), times.max(), freqs.min(), freqs.max()])\n", "ax2.set_yticks(freqs[::16])\n", "ax2.set_xticks(times[::16])\n", "ax2.text(0.06, 1000, 'Y', fontsize=18)\n", "ax2.text(0.17, 1000, 'E', fontsize=18)\n", "ax2.text(0.36, 1000, 'S', fontsize=18)\n", "\n", "xcoords = [0.025, 0.11, 0.23, 0.49]\n", "for xc in xcoords:\n", "    ax1.axvline(x=xc*16000, c='r')\n", "    ax2.axvline(x=xc, c='r')"]}, {"source": ["## 1.3. MFCC\n", "\n", "If you want to get to know some details about *MFCC* take a look at this great tutorial. [MFCC explained](http://practicalcryptography.com/miscellaneous/machine-learning/guide-mel-frequency-cepstral-coefficients-mfccs/)\n", "\n", "You can calculate *MFCC* using for example *librosa* python package.\n", "\n", "It is worth nothing, that in classical, but still state-of-the-art systems, features similar to *MFCC* are taken as the input to the system instead of spectrograms.\n", "However, in end-to-end (often neural-network based) systems, the most common input features are raw spectrograms."], "metadata": {"_cell_guid": "11be3aca-81ec-4a52-8d48-7b646520773e", "_uuid": "bca8a80aba92207d7f1d6abfc69daf7ad9be05f4"}, "cell_type": "markdown"}, {"source": ["## 1.4. Features extraction steps\n", "\n", "I would propose the feature extraction algorithm like that:\n", "1. *VAD*\n", "2. Maybe padding with 0 to make signals be equal length\n", "3. Log spectrogram (or *MFCC*, or *PLP*)\n", "4. Features normalization with *mean* and *std*\n", "5. Stacking of a given number of frames to get temporal information\n", "\n", "It's a pity it can't be done in notebook. It has not much sense to write things from zero, and everything is ready to take, but in packages, that can not be imported in Kernels."], "metadata": {"_cell_guid": "f98fe35d-2d56-4153-b054-0882bd2e58ce", "_uuid": "57fe8c6a25753e2eb46285bc8d725d20182c1421"}, "cell_type": "markdown"}, {"source": ["\n", "# 2. Dataset investigation\n", "\n", "Some usuall investgation of dataset.\n", "\n", "## 2.1. Number of records\n", "\n"], "metadata": {"_cell_guid": "3d36bac6-eb6f-4a53-b148-805493e39052", "_uuid": "caf345ca07983f1e1d4f8a05f6f74859554289db"}, "cell_type": "markdown"}, {"metadata": {"_cell_guid": "3c24fbdd-e50e-47a1-8c44-1894bec7f043", "_uuid": "59826a3eb0f60439d5beee06781193bc67cc53f7"}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["dirs = [f for f in os.listdir(train_audio_path) if isdir(join(train_audio_path, f))]\n", "dirs.sort()\n", "print('Number of labels: ' + str(len(dirs)))"]}, {"metadata": {"_cell_guid": "a6b82ced-df8c-4c7a-8d4c-ed32bf9f60f6", "_uuid": "ea30edeaf3d8020bf55ee2a57af230bded9732e2"}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["number_of_recordings = []\n", "for direct in dirs:\n", "    waves = [f for f in os.listdir(join(train_audio_path, direct)) if f.endswith('.wav')]\n", "    number_of_recordings.append(len(waves))\n", "\n", "plt.figure(figsize=(14,8))\n", "plt.bar(dirs, number_of_recordings)\n", "plt.title('Number of recordings in given label')\n", "plt.xticks(rotation='vertical')\n", "plt.ylabel('Number of recordings')\n", "plt.xlabel('Words')\n", "plt.show()\n"]}, {"source": ["Dataset is balanced except of background_noise, but that's the different thing."], "metadata": {"_cell_guid": "c4b1d377-d84e-4601-97fa-2c989023c400", "_uuid": "928d2933df5e0a37c7dc40f2ec50b9d10423d533"}, "cell_type": "markdown"}, {"source": ["## 2.2. Deeper into recordings\n", "\n", "We'll need to calculate FFT (Fast Fourier Transform). Definition:"], "metadata": {"_cell_guid": "b9b6b43c-96fa-489d-8a49-b77dabd41705", "_uuid": "0847bb69f23ce4c8153f869222c03731fd62a22e"}, "cell_type": "markdown"}, {"metadata": {"collapsed": true, "_cell_guid": "501a0c7e-a117-4075-8b55-b1c838cd6b37", "_uuid": "04fa77faeff538b9a4bcadf932c56c0b128d24e2"}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["def custom_fft(y, fs):\n", "    T = 1.0 / fs\n", "    N = y.shape[0]\n", "    yf = fft(y)\n", "    xf = np.linspace(0.0, 1.0/(2.0*T), N//2)\n", "    vals = 2.0/N * np.abs(yf[0:N//2])\n", "    return xf, vals"]}, {"source": ["There's a very important fact. Recordings come from very different sources. As far as I can tell, some of them can come from mobile GSM channel.\n", "\n", "Nevertheless,** it is extremely important to split the dataset in a way that one speaker doesn't occur in both train and test sets.**\n", "Just take a look and listen to this two examlpes:"], "metadata": {"_cell_guid": "766c2e18-2aff-43c1-9672-bd51d4348867", "_uuid": "d4b8b90afc03493f93babb7b9d401ebd0caa1c18"}, "cell_type": "markdown"}, {"metadata": {"_cell_guid": "d29ea00a-54eb-474a-8770-fbd269bbd21e", "_uuid": "de451cfb747953d60d9b6982c046b417fdc1ab9b"}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["filenames = ['on/004ae714_nohash_0.wav', 'on/004ae714_nohash_1.wav', 'on/0137b3f4_nohash_0.wav', 'on/0137b3f4_nohash_1.wav']\n", "for filename in filenames:\n", "    sample_rate, samples = wavfile.read(str(train_audio_path) + filename)\n", "    xf, vals = custom_fft(samples, sample_rate)\n", "    plt.figure(figsize=(12, 4))\n", "    plt.title('FFT of speaker ' + filename[4:11])\n", "    plt.plot(xf, vals)\n", "    plt.xlabel('Frequency')\n", "    plt.grid()\n", "    plt.show()"]}, {"source": ["Even better to listen:"], "metadata": {"_cell_guid": "508c79b0-6beb-43cb-9092-9e0c5719e12e", "_uuid": "ae779083e33d29a22bcf972542d9911a7b9d64de"}, "cell_type": "markdown"}, {"metadata": {"_cell_guid": "63e5851f-b462-4f9c-8e17-a7e8e301d616", "_uuid": "4e2d4b6d2c6e6806a2b0b0d4554e49b080003a62"}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["print('Speaker ' + filenames[0][4:11])\n", "ipd.Audio(join(train_audio_path, filenames[0]))"]}, {"metadata": {"_cell_guid": "fd2eb518-df30-44a0-89b3-9c4611beb50d", "_uuid": "c335e31977a721831149579da9105cc2b664b40d"}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["print('Speaker ' + filenames[1][4:11])\n", "ipd.Audio(join(train_audio_path, filenames[1]))"]}, {"metadata": {"_cell_guid": "ab14e599-9f7e-4645-b802-5bf80d722c10", "_uuid": "f004ad084642da537aaed6d197a77cb445aa2b42"}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["print('Speaker ' + filenames[2][4:11])\n", "ipd.Audio(join(train_audio_path, filenames[2]))"]}, {"metadata": {"_cell_guid": "156663a3-3968-4c64-b3e5-a4b944d53a25", "_uuid": "09e25d6c4c19a013d1a987ff49e94742d4e8ec8f"}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["print('Speaker ' + filenames[3][4:11])\n", "ipd.Audio(join(train_audio_path, filenames[3]))"]}, {"source": ["There are also recordings with some weird silence (some compression?):\n"], "metadata": {"_cell_guid": "7c2f7df0-d062-46d6-9507-5ab548db14bb", "_uuid": "63555f55e906409e008d9c0a988a03b26cbb8983"}, "cell_type": "markdown"}, {"metadata": {"_cell_guid": "bc6075ee-fc43-4d04-b400-64896ac8450d", "_uuid": "ba2e76ebe90eb9c4fed6617d1948e0c71d88c54e"}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["filename = '/yes/01bb6a2a_nohash_1.wav'\n", "sample_rate, samples = wavfile.read(str(train_audio_path) + filename)\n", "freqs, times, spectrogram = log_specgram(samples, sample_rate)\n", "\n", "plt.figure(figsize=(10, 7))\n", "plt.title('Spectrogram of ' + filename)\n", "plt.ylabel('Freqs')\n", "plt.xlabel('Time')\n", "plt.imshow(spectrogram.T, aspect='auto', origin='lower', \n", "           extent=[times.min(), times.max(), freqs.min(), freqs.max()])\n", "plt.yticks(freqs[::16])\n", "plt.xticks(times[::16])\n", "plt.show()"]}, {"source": ["It means, that we have to prevent overfitting to the very specific acoustical environments.\n", "\n", "## 2.3. Recordings length\n", "Find if all the files have 1 second duration:"], "metadata": {"_cell_guid": "fb57dc50-0510-4dcf-ba07-7999daf7349e", "_uuid": "cbf303188e33f325691aa9447f18586788bf110a"}, "cell_type": "markdown"}, {"metadata": {"_cell_guid": "23be7e5e-e4b4-40a0-b9a3-4bc850571a28", "_uuid": "16a2e2c908235a99f64024abab272c65d3d99c65"}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["num_of_shorter = 0\n", "for direct in dirs:\n", "    waves = [f for f in os.listdir(join(train_audio_path, direct)) if f.endswith('.wav')]\n", "    for wav in waves:\n", "        sample_rate, samples = wavfile.read(train_audio_path + direct + '/' + wav)\n", "        if samples.shape[0] < sample_rate:\n", "            num_of_shorter += 1\n", "print('Number of recordings shorter than 1 second: ' + str(num_of_shorter))"]}, {"source": ["That's suprising, and there is a lot of them. We can pad them with zeros."], "metadata": {"_cell_guid": "57b26071-e603-4ee7-b1d5-5bd2bd46e438", "_uuid": "c035e161e9c9622fa96f9589ffbfe826e01c5658"}, "cell_type": "markdown"}, {"source": ["## 2.4. Mean spectrograms and FFT"], "metadata": {"_cell_guid": "771ff180-73dd-4be8-aa56-ccecb3586416", "_uuid": "96655097f9173d34d529a0626446194473cebf69"}, "cell_type": "markdown"}, {"source": ["Let's plot mean FFT for every word"], "metadata": {"_cell_guid": "a621a03b-4812-4d7c-93dd-6b0bf7f10572", "_uuid": "a3f64232afa284102e8ffdcdbe0db509f4a78a7e"}, "cell_type": "markdown"}, {"metadata": {"_cell_guid": "dc065096-6888-4a38-9c66-0729bfa858f6", "_uuid": "e8fc86e267bb48d3ae35f8e2d85db070f28889c9"}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["to_keep = 'yes no up down left right on off stop go silence unknown'.split()\n", "dirs = [d for d in dirs if d in to_keep]\n", "\n", "print(dirs)\n", "\n", "for direct in dirs:\n", "    vals_all = []\n", "    spec_all = []\n", "\n", "    waves = [f for f in os.listdir(join(train_audio_path, direct)) if f.endswith('.wav')]\n", "    for wav in waves:\n", "        sample_rate, samples = wavfile.read(train_audio_path + direct + '/' + wav)\n", "        if samples.shape[0] != 16000:\n", "            continue\n", "        xf, vals = custom_fft(samples, 16000)\n", "        vals_all.append(vals)\n", "        freqs, times, spec = log_specgram(samples, 16000)\n", "        spec_all.append(spec)\n", "\n", "    plt.figure(figsize=(14, 5))\n", "    plt.subplot(121)\n", "    plt.title('Mean fft of ' + direct)\n", "    plt.plot(np.mean(np.array(vals_all), axis=0))\n", "    plt.grid()\n", "    plt.subplot(122)\n", "    plt.title('Mean specgram of ' + direct)\n", "    plt.imshow(np.mean(np.array(spec_all), axis=0).T, aspect='auto', origin='lower', \n", "               extent=[times.min(), times.max(), freqs.min(), freqs.max()])\n", "    plt.yticks(freqs[::16])\n", "    plt.xticks(times[::16])\n", "    plt.show()"]}, {"source": ["## 2.5. Gaussian Mixtures modeling\n", "\n", "We can see that mean FFT looks different for every word. We could model each FFT with a mixture of Gaussian distributions. Some of them however, look almost identical on FFT, like *stop* and *up*... But wait, they are still distinguishable when we look at spectrograms! High frequencies are earlier than low at the beginning of *stop* (probably *s*).\n", "\n", "That's why temporal component is also necessary. There is a [Kaldi](http://kaldi-asr.org/) library, that can model words (or smaller parts of words) with GMMs and model temporal dependencies with [Hidden Markov Models](https://github.com/danijel3/ASRDemos/blob/master/notebooks/HMM_FST.ipynb).\n", "\n", "We could use simple GMMs for words to check what can we model and how hard it is to distinguish the words. We can use [Scikit-learn](http://scikit-learn.org/) for that, however it is not straightforward and lasts very long here, so I abandon this idea for now."], "metadata": {"_cell_guid": "1089a473-7fe5-40a0-9e43-8f3cbf1901c3", "_uuid": "931aceb2fb6ac23defc699b3d423b510171b1626"}, "cell_type": "markdown"}, {"source": ["## 2.6. Frequency components across the words"], "metadata": {"_cell_guid": "341fd72e-750e-4d78-a7b7-1d347bcad4e8", "_uuid": "0c89774ecfd33f29c10cba58f1fb1c12647b0928"}, "cell_type": "markdown"}, {"metadata": {"collapsed": true, "_cell_guid": "7a883de7-1fa7-428d-bd92-0fb0d3f75aed", "_uuid": "0c3e4ecb1eb87bd8c86547df68db35ef9a84a2ce"}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["def violinplot_frequency(dirs, freq_ind):\n", "    \"\"\" Plot violinplots for given words (waves in dirs) and frequency freq_ind\n", "    from all frequencies freqs.\"\"\"\n", "\n", "    spec_all = []  # Contain spectrograms\n", "    ind = 0\n", "    for direct in dirs:\n", "        spec_all.append([])\n", "\n", "        waves = [f for f in os.listdir(join(train_audio_path, direct)) if\n", "                 f.endswith('.wav')]\n", "        for wav in waves[:100]:\n", "            sample_rate, samples = wavfile.read(\n", "                train_audio_path + direct + '/' + wav)\n", "            freqs, times, spec = log_specgram(samples, sample_rate)\n", "            spec_all[ind].extend(spec[:, freq_ind])\n", "        ind += 1\n", "\n", "    # Different lengths = different num of frames. Make number equal\n", "    minimum = min([len(spec) for spec in spec_all])\n", "    spec_all = np.array([spec[:minimum] for spec in spec_all])\n", "\n", "    plt.figure(figsize=(13,8))\n", "    plt.title('Frequency ' + str(freqs[freq_ind]) + ' Hz')\n", "    plt.ylabel('Amount of frequency in a word')\n", "    plt.xlabel('Words')\n", "    sns.violinplot(data=pd.DataFrame(spec_all.T, columns=dirs))\n", "    plt.show()"]}, {"metadata": {"_cell_guid": "08aae780-53a3-4047-b548-46b8893aaed9", "_uuid": "8932034e13e12d7ff50bc6663c03153ae1052378"}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["violinplot_frequency(dirs, 20)"]}, {"metadata": {"collapsed": true, "_cell_guid": "ea7ea51e-3a38-46db-891d-4c3aef8fd810", "_uuid": "30e1bf01aa6431fd883530beabb4da5d9c8518dc"}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["violinplot_frequency(dirs, 50)"]}, {"metadata": {"collapsed": true, "_cell_guid": "a84e722a-3aaa-42dd-868b-6eb629abf40d", "_uuid": "fb2a917be56bc19c256ad52e257122a7f1b5dd8b"}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["violinplot_frequency(dirs, 120)"]}, {"source": ["## 2.7. Anomaly detection\n", "\n", "We should check if there are any recordings that somehow stand out from the rest. We can lower the dimensionality of the dataset and interactively check for any anomaly.\n", "We'll use PCA for dimensionality reduction:"], "metadata": {"_cell_guid": "7e06a571-5ca6-4a50-8bf1-c037d9399df3", "_uuid": "f10445f9398dcd57b404591c69075a23ab3d5115"}, "cell_type": "markdown"}, {"metadata": {"collapsed": true, "_cell_guid": "8ffe76a4-7e56-481b-8406-22520b573edc", "_uuid": "5a52c1fbea1b68c78502ac9954bd7ecae19149d6"}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["fft_all = []\n", "names = []\n", "for direct in dirs:\n", "    waves = [f for f in os.listdir(join(train_audio_path, direct)) if f.endswith('.wav')]\n", "    for wav in waves:\n", "        sample_rate, samples = wavfile.read(train_audio_path + direct + '/' + wav)\n", "        if samples.shape[0] != sample_rate:\n", "            samples = np.append(samples, np.zeros((sample_rate - samples.shape[0], )))\n", "        x, val = custom_fft(samples, sample_rate)\n", "        fft_all.append(val)\n", "        names.append(direct + '/' + wav)\n", "\n", "fft_all = np.array(fft_all)\n", "\n", "# Normalization\n", "fft_all = (fft_all - np.mean(fft_all, axis=0)) / np.std(fft_all, axis=0)\n", "\n", "# Dim reduction\n", "pca = PCA(n_components=3)\n", "fft_all = pca.fit_transform(fft_all)\n", "\n", "def interactive_3d_plot(data, names):\n", "    scatt = go.Scatter3d(x=data[:, 0], y=data[:, 1], z=data[:, 2], mode='markers', text=names)\n", "    data = go.Data([scatt])\n", "    layout = go.Layout(title=\"Anomaly detection\")\n", "    figure = go.Figure(data=data, layout=layout)\n", "    py.iplot(figure)\n", "    \n", "interactive_3d_plot(fft_all, names)"]}, {"source": ["Notice that there are *yes/e4b02540_nohash_0.wav*, *go/0487ba9b_nohash_0.wav* and more points, that lie far away from the rest. Let's listen to them."], "metadata": {"_cell_guid": "7c239d25-c761-4c54-87ed-c47c3f641fdd", "_uuid": "f0807b3e37e3ef955abb1995bfa8cccf2d979fd2"}, "cell_type": "markdown"}, {"metadata": {"collapsed": true, "_cell_guid": "70c6eba5-0621-4292-b9bd-b92bc5066ce3", "_uuid": "8f2b077bda2f985ee5dabfa66778d1c3001cb24b"}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["print('Recording go/0487ba9b_nohash_0.wav')\n", "ipd.Audio(join(train_audio_path, 'go/0487ba9b_nohash_0.wav'))"]}, {"metadata": {"collapsed": true, "_cell_guid": "1be3b686-d99d-4a1b-aa23-165ad4e5c67e", "_uuid": "4116f932a79f3e2e9f745ec4e441c0529ccf66fa"}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["print('Recording yes/e4b02540_nohash_0.wav')\n", "ipd.Audio(join(train_audio_path, 'yes/e4b02540_nohash_0.wav'))"]}, {"source": ["If you will look for anomalies for individual words, you can find for example this file for *seven*:"], "metadata": {"_cell_guid": "755472e7-6c19-4c2b-bb8a-d161a501c80a", "_uuid": "f1252a107c341e3592a301398be2a56202b9f2a5"}, "cell_type": "markdown"}, {"metadata": {"collapsed": true, "_cell_guid": "45552683-e626-4143-88c2-b22f9fa00737", "_uuid": "542a9b0b0c61e2420af77d0f0196178475952681"}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["print('Recording seven/e4b02540_nohash_0.wav')\n", "ipd.Audio(join(train_audio_path, 'seven/b1114e4f_nohash_0.wav'))"]}, {"source": ["That's nothing obviously important. Usually you can find some distortions using this method. Data seems to contain what it should."], "metadata": {"_cell_guid": "67b4915a-7459-4863-a696-ac8220da5a29", "_uuid": "8e532f1e54b04ba5850a163428d1fd94c47bee37"}, "cell_type": "markdown"}, {"source": ["## 3. Where to look for the inspiration\n", "\n", "You can take many different approches for the competition. I can't really advice any of that. I'd like to share my initial thoughts.\n", "\n", "There is a trend in recent years to propose solutions based on neural networks. Usually there are two architectures. My ideas are here.\n", "\n", "1. Encoder-decoder: https://arxiv.org/abs/1508.01211\n", "2. RNNs with CTC loss: https://arxiv.org/abs/1412.5567<br>\n", "For me, 1 and 2  are a sensible choice for this competition, especially if you do not have background in SR field. They try to be end-to-end solutions. Speech recognition is a really big topic and it would be hard to get to know important things in short time.\n", "\n", "3. Classic speech recognition is described here: http://www.ece.ucsb.edu/Faculty/Rabiner/ece259/Reprints/tutorial%20on%20hmm%20and%20applications.pdf\n", "\n", "4. Very deep CNN - Don't know if it is used for SR. However, most papers concern Large Vocabulary Continuous Speech Recognition Systems (LVCSR). We got different task here - a very small vocabulary, and recordings with only one word in it, with a (mostly) given length. I suppose such approach can win the competition. \n"], "metadata": {"_cell_guid": "8e1ba6de-b802-417a-b98d-72d0fed93296", "_uuid": "a50ecff1ab0f1629bbd069c5e325e09035bc2778"}, "cell_type": "markdown"}, {"source": ["**If you like my work please upvote.**\n", "\n", "Leave a feedback that will let me improve! "], "metadata": {"_cell_guid": "9192984a-7307-41e5-bb02-9f46f14822c7", "_uuid": "f4862ca5ef5f190c0593ff4bd165acc74588c373"}, "cell_type": "markdown"}], "metadata": {"kernelspec": {"language": "python", "display_name": "Python 3", "name": "python3"}, "language_info": {"mimetype": "text/x-python", "file_extension": ".py", "codemirror_mode": {"name": "ipython", "version": 3}, "name": "python", "pygments_lexer": "ipython3", "version": "3.6.3", "nbconvert_exporter": "python"}}, "nbformat": 4, "nbformat_minor": 1}