{"metadata": {"kernelspec": {"name": "python3", "display_name": "Python 3", "language": "python"}, "language_info": {"name": "python", "pygments_lexer": "ipython3", "file_extension": ".py", "version": "3.6.3", "codemirror_mode": {"name": "ipython", "version": 3}, "nbconvert_exporter": "python", "mimetype": "text/x-python"}}, "nbformat": 4, "nbformat_minor": 1, "cells": [{"metadata": {"_uuid": "b400f28dc99c0b70f9e00f55a942dfd8a557208c", "_cell_guid": "eb1a2eb8-b977-4574-a114-5a6cad1cf1b4"}, "cell_type": "markdown", "source": ["\n", "# Audio Data Conversion to Images + EDA\n", "\n", "**The dataset**: To help with this, TensorFlow recently released the Speech Commands Datasets. It includes 65,000 one-second long utterances of 30 short words, by thousands of different people.\n", "\n", "| dataset | file count | zipped space | expanded space | \n", "|-------|-------|-------|--------|\n", "|train|64,727 files| 1.1 GB | 2.1 GB |\n", "|test|158,538 files | 2.6 GB | 5.2 GB |\n", "\n", "**The Goal**: In this competition, you're challenged to use the Speech Commands Dataset to build an algorithm that understands simple spoken commands. \n", "\n", "### About this Kernel\n", "Hi all,  if you are considering doing image recognition on these images, I've put together a starter kit that is designed to convert all the wav files into pictures with the goal of running image recognition on the source files. Hope the below is helpful!\n", "\n", "**Version improvement 11-17-17**: swapped libraries to `scipy` instead of using `soundfile`. It will run on kaggle kernel, but handicapped the actual file processing. Credit to https://www.kaggle.com/davids1992/data-visualization-and-investigation for the log_spectogram transformation.\n", "\n", "**initial Version improvement 11-16-17**: first submission, requires `soundfile` library\n", "\n", "![](http://www1.icsi.berkeley.edu/Speech/mr/images/PZM_spectrogram.gif)\n"], "attachments": {}}, {"metadata": {}, "cell_type": "markdown", "source": ["\n"]}, {"metadata": {}, "cell_type": "markdown", "source": ["## 1. Load libraries"]}, {"metadata": {"_uuid": "38022b12c61c8906297b702514e0654e29ff359b", "_cell_guid": "9b2fb550-845b-475b-80b8-908c71f9fbd0", "collapsed": true}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["import matplotlib.pyplot as plt\n", "from matplotlib.backend_bases import RendererBase\n", "from scipy import signal\n", "from scipy.io import wavfile\n", "#import soundfile as sf\n", "import os\n", "import numpy as np\n", "from PIL import Image\n", "from scipy.fftpack import fft\n", "\n", "%matplotlib inline"]}, {"metadata": {}, "cell_type": "markdown", "source": ["## 2. Set your file path"]}, {"metadata": {"collapsed": true}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["audio_path = '../input/train/audio/'\n", "pict_Path = '../input/picts/train/'\n", "test_pict_Path = '../input/picts/test/'\n", "test_audio_path = '../input/test/audio/'\n", "samples = []"]}, {"metadata": {}, "cell_type": "markdown", "source": ["#### Kaggle Version: Identify all the subdirectories in the training directory****"]}, {"metadata": {}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["subFolderList = []\n", "for x in os.listdir(audio_path):\n", "    if os.path.isdir(audio_path + '/' + x):\n", "        subFolderList.append(x)"]}, {"metadata": {}, "cell_type": "markdown", "source": ["#### Local Version: create local directories and identify subdirectories\n", "\n", "```python\n", "if not os.path.exists(pict_Path):\n", "    os.makedirs(pict_Path)\n", "\n", "if not os.path.exists(test_pict_Path):\n", "    os.makedirs(test_pict_Path)\n", "\n", "    \n", "subFolderList = []\n", "for x in os.listdir(audio_path):\n", "    if os.path.isdir(audio_path + '/' + x):\n", "        subFolderList.append(x)\n", "        if not os.path.exists(pict_Path + '/' + x):\n", "            os.makedirs(pict_Path +'/'+ x)\n", "```"]}, {"metadata": {}, "cell_type": "markdown", "source": ["## 3. Pull an audio sample from each word\u00b6\n"]}, {"metadata": {}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["sample_audio = []\n", "total = 0\n", "for x in subFolderList:\n", "    \n", "    # get all the wave files\n", "    all_files = [y for y in os.listdir(audio_path + x) if '.wav' in y]\n", "    total += len(all_files)\n", "    # collect the first file from each dir\n", "    sample_audio.append(audio_path  + x + '/'+ all_files[0])\n", "    \n", "    # show file counts\n", "    print('count: %d : %s' % (len(all_files), x ))\n", "print(total)"]}, {"metadata": {}, "cell_type": "markdown", "source": ["> ## 4. Spectrograms\n", "Sample File Path\n"]}, {"metadata": {}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["sample_audio[0]"]}, {"metadata": {}, "cell_type": "markdown", "source": ["### 4.1 Preview of Spectograms across different words\n", "\n", "Borrowing log spec function from\n", "\n", "https://www.kaggle.com/davids1992/data-visualization-and-investigation"]}, {"metadata": {"collapsed": true}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["def log_specgram(audio, sample_rate, window_size=20,\n", "                 step_size=10, eps=1e-10):\n", "    nperseg = int(round(window_size * sample_rate / 1e3))\n", "    noverlap = int(round(step_size * sample_rate / 1e3))\n", "    freqs, _, spec = signal.spectrogram(audio,\n", "                                    fs=sample_rate,\n", "                                    window='hann',\n", "                                    nperseg=nperseg,\n", "                                    noverlap=noverlap,\n", "                                    detrend=False)\n", "    return freqs, np.log(spec.T.astype(np.float32) + eps)"]}, {"metadata": {}, "cell_type": "markdown", "source": ["#### Looking at the top 9 different words in Spectrogram format"]}, {"metadata": {}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["fig = plt.figure(figsize=(10,10))\n", "\n", "# for each of the samples\n", "for i, filepath in enumerate(sample_audio[:9]):\n", "    # Make subplots\n", "    plt.subplot(3,3,i+1)\n", "    \n", "    # pull the labels\n", "    label = filepath.split('/')[-2]\n", "    plt.title(label)\n", "    \n", "    # create spectogram\n", "    samplerate, test_sound  = wavfile.read(filepath)\n", "    _, spectrogram = log_specgram(test_sound, samplerate)\n", "    \n", "    plt.imshow(spectrogram.T, aspect='auto', origin='lower')\n", "    plt.axis('off')"]}, {"metadata": {}, "cell_type": "markdown", "source": ["### 4.2 Spectograms within the same category, look at \"five\"\n"]}, {"metadata": {}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["five_samples = [audio_path + 'five/' + y for y in os.listdir(audio_path + 'five/')[:6]]\n", "\n", "fig = plt.figure(figsize=(10,10))\n", "\n", "for i, filepath in enumerate(five_samples):\n", "    # Make subplots\n", "    plt.subplot(3,3,i+1)\n", "    \n", "    # pull the labels\n", "    label = filepath.split('/')[-1]\n", "    plt.title('\"five\": '+label)\n", "    \n", "    # create spectogram\n", "    # create spectogram\n", "    samplerate, test_sound  = wavfile.read(filepath)\n", "    _, spectrogram = log_specgram(test_sound, samplerate)\n", "    \n", "    plt.imshow(spectrogram.T, aspect='auto', origin='lower')\n", "    plt.axis('off')"]}, {"metadata": {}, "cell_type": "markdown", "source": ["## 5 Waveforms\n", "### 5.1 Waveforms across different Words "]}, {"metadata": {}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["fig = plt.figure(figsize=(8,20))\n", "for i, filepath in enumerate(sample_audio[:6]):\n", "    plt.subplot(9,1,i+1)\n", "    samplerate, test_sound  = wavfile.read(filepath)\n", "    plt.title(filepath.split('/')[-2])\n", "    plt.axis('off')\n", "    plt.plot(test_sound)"]}, {"metadata": {}, "cell_type": "markdown", "source": ["### 5.2 Waveforms within the Same Word "]}, {"metadata": {}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["fig = plt.figure(figsize=(8,20))\n", "for i, filepath in enumerate(five_samples):\n", "    plt.subplot(9,1,i+1)\n", "    samplerate, test_sound = wavfile.read(filepath)\n", "    plt.title(filepath.split('/')[-2])\n", "    plt.axis('off')\n", "    plt.plot(test_sound)"]}, {"metadata": {}, "cell_type": "markdown", "source": ["## 6. Save Figures as images\n", "#### Function: convert audio to spectogram images"]}, {"metadata": {"collapsed": true}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["def wav2img(wav_path, targetdir='', figsize=(4,4)):\n", "    \"\"\"\n", "    takes in wave file path\n", "    and the fig size. Default 4,4 will make images 288 x 288\n", "    \"\"\"\n", "\n", "    fig = plt.figure(figsize=figsize)    \n", "    # use soundfile library to read in the wave files\n", "    samplerate, test_sound  = wavfile.read(filepath)\n", "    _, spectrogram = log_specgram(test_sound, samplerate)\n", "    \n", "    ## create output path\n", "    output_file = wav_path.split('/')[-1].split('.wav')[0]\n", "    output_file = targetdir +'/'+ output_file\n", "    #plt.imshow(spectrogram.T, aspect='auto', origin='lower')\n", "    plt.imsave('%s.png' % output_file, spectrogram)\n", "    plt.close()"]}, {"metadata": {}, "cell_type": "markdown", "source": ["#### Function: convert audio to waveform images"]}, {"metadata": {}, "cell_type": "code", "execution_count": null, "outputs": [], "source": ["def wav2img_waveform(wav_path, targetdir='', figsize=(4,4)):\n", "    samplerate,test_sound  = wavfile.read(sample_audio[0])\n", "    fig = plt.figure(figsize=figsize)\n", "    plt.plot(test_sound)\n", "    plt.axis('off')\n", "    output_file = wav_path.split('/')[-1].split('.wav')[0]\n", "    output_file = targetdir +'/'+ output_file\n", "    plt.savefig('%s.png' % output_file)\n", "    plt.close()"]}, {"metadata": {}, "cell_type": "markdown", "source": ["## 6.1 Loops to iterate through directories and save\n", "\n", "#### These are 100% markdown to avoid running on the kaggle site. \n", "\n", "Copy and run locally in your instance. I've capped the number of folders and images to 3 and 5 respectively. Remove the list indexing to run through all directories and all images. Becareful, there's 64k training images!\n", "\n", "### Convert Training Audio\n", "\n", "```python\n", "for i, x in enumerate(subFolderList[:3]):\n", "    print(i, ':', x)\n", "    # get all the wave files\n", "    all_files = [y for y in os.listdir(audio_path + x) if '.wav' in y]\n", "    for file in all_files[:10]:\n", "        wav2img(audio_path + x + '/' + file, pict_Path + x)\n", "```\n", "\n", "### Convert Testing Audio\n", "```python\n", "# get all the wave files\n", "all_files = [y for y in os.listdir(test_audio_path + x) if '.wav' in y]\n", "for file in all_files:\n", "    wav2img(test_audio_path + x + '/' + file, test_pict_Path + x)\n", "```"]}, {"metadata": {"collapsed": true}, "cell_type": "code", "execution_count": null, "outputs": [], "source": []}]}