{"cells":[{"metadata":{"_uuid":"b400f28dc99c0b70f9e00f55a942dfd8a557208c","_cell_guid":"eb1a2eb8-b977-4574-a114-5a6cad1cf1b4"},"cell_type":"markdown","source":"\n# Audio Data Conversion to Images + EDA\n\n**The dataset**: To help with this, TensorFlow recently released the Speech Commands Datasets. It includes 65,000 one-second long utterances of 30 short words, by thousands of different people.\n\n| dataset | file count | zipped space | expanded space | \n|-------|-------|-------|--------|\n|train|64,727 files| 1.1 GB | 2.1 GB |\n|test|158,538 files | 2.6 GB | 5.2 GB |\n\n**The Goal**: In this competition, you're challenged to use the Speech Commands Dataset to build an algorithm that understands simple spoken commands. \n\n### About this Kernel\nHi all,  if you are considering doing image recognition on these images, I've put together a starter kit that is designed to convert all the wav files into pictures with the goal of running image recognition on the source files. Hope the below is helpful!\n\n**Version improvement 11-17-17**: swapped libraries to `scipy` instead of using `soundfile`. It will run on kaggle kernel, but handicapped the actual file processing. Credit to https://www.kaggle.com/davids1992/data-visualization-and-investigation for the log_spectogram transformation.\n\n**initial Version improvement 11-16-17**: first submission, requires `soundfile` library\n\n![](http://www1.icsi.berkeley.edu/Speech/mr/images/PZM_spectrogram.gif)\n"},{"metadata":{},"cell_type":"markdown","source":"\n"},{"metadata":{},"cell_type":"markdown","source":"## 1. Load libraries"},{"metadata":{"_uuid":"38022b12c61c8906297b702514e0654e29ff359b","_cell_guid":"9b2fb550-845b-475b-80b8-908c71f9fbd0","collapsed":true,"trusted":false},"cell_type":"code","source":"import matplotlib.pyplot as plt\nfrom matplotlib.backend_bases import RendererBase\nfrom scipy import signal\nfrom scipy.io import wavfile\n#import soundfile as sf\nimport os\nimport numpy as np\nfrom PIL import Image\nfrom scipy.fftpack import fft\n\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2. Set your file path"},{"metadata":{"collapsed":true,"trusted":false},"cell_type":"code","source":"audio_path = '../input/train/audio/'\npict_Path = '../input/picts/train/'\ntest_pict_Path = '../input/picts/test/'\ntest_audio_path = '../input/test/audio/'\nsamples = []","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Kaggle Version: Identify all the subdirectories in the training directory****"},{"metadata":{"trusted":false},"cell_type":"code","source":"subFolderList = []\nfor x in os.listdir(audio_path):\n    if os.path.isdir(audio_path + '/' + x):\n        subFolderList.append(x)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Local Version: create local directories and identify subdirectories\n\n```python\nif not os.path.exists(pict_Path):\n    os.makedirs(pict_Path)\n\nif not os.path.exists(test_pict_Path):\n    os.makedirs(test_pict_Path)\n\n    \nsubFolderList = []\nfor x in os.listdir(audio_path):\n    if os.path.isdir(audio_path + '/' + x):\n        subFolderList.append(x)\n        if not os.path.exists(pict_Path + '/' + x):\n            os.makedirs(pict_Path +'/'+ x)\n```"},{"metadata":{},"cell_type":"markdown","source":"## 3. Pull an audio sample from each word¶\n"},{"metadata":{"trusted":false},"cell_type":"code","source":"sample_audio = []\ntotal = 0\nfor x in subFolderList:\n    \n    # get all the wave files\n    all_files = [y for y in os.listdir(audio_path + x) if '.wav' in y]\n    total += len(all_files)\n    # collect the first file from each dir\n    sample_audio.append(audio_path  + x + '/'+ all_files[0])\n    \n    # show file counts\n    print('count: %d : %s' % (len(all_files), x ))\nprint(total)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> ## 4. Spectrograms\nSample File Path\n"},{"metadata":{"trusted":false},"cell_type":"code","source":"sample_audio[0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 4.1 Preview of Spectograms across different words\n\nBorrowing log spec function from\n\nhttps://www.kaggle.com/davids1992/data-visualization-and-investigation"},{"metadata":{"collapsed":true,"trusted":false},"cell_type":"code","source":"def log_specgram(audio, sample_rate, window_size=20,\n                 step_size=10, eps=1e-10):\n    nperseg = int(round(window_size * sample_rate / 1e3))\n    noverlap = int(round(step_size * sample_rate / 1e3))\n    freqs, _, spec = signal.spectrogram(audio,\n                                    fs=sample_rate,\n                                    window='hann',\n                                    nperseg=nperseg,\n                                    noverlap=noverlap,\n                                    detrend=False)\n    return freqs, np.log(spec.T.astype(np.float32) + eps)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Looking at the top 9 different words in Spectrogram format"},{"metadata":{"trusted":false},"cell_type":"code","source":"fig = plt.figure(figsize=(10,10))\n\n# for each of the samples\nfor i, filepath in enumerate(sample_audio[:9]):\n    # Make subplots\n    plt.subplot(3,3,i+1)\n    \n    # pull the labels\n    label = filepath.split('/')[-2]\n    plt.title(label)\n    \n    # create spectogram\n    samplerate, test_sound  = wavfile.read(filepath)\n    _, spectrogram = log_specgram(test_sound, samplerate)\n    \n    plt.imshow(spectrogram.T, aspect='auto', origin='lower')\n    plt.axis('off')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 4.2 Spectograms within the same category, look at \"five\"\n"},{"metadata":{"trusted":false},"cell_type":"code","source":"five_samples = [audio_path + 'five/' + y for y in os.listdir(audio_path + 'five/')[:6]]\n\nfig = plt.figure(figsize=(10,10))\n\nfor i, filepath in enumerate(five_samples):\n    # Make subplots\n    plt.subplot(3,3,i+1)\n    \n    # pull the labels\n    label = filepath.split('/')[-1]\n    plt.title('\"five\": '+label)\n    \n    # create spectogram\n    # create spectogram\n    samplerate, test_sound  = wavfile.read(filepath)\n    _, spectrogram = log_specgram(test_sound, samplerate)\n    \n    plt.imshow(spectrogram.T, aspect='auto', origin='lower')\n    plt.axis('off')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 5 Waveforms\n### 5.1 Waveforms across different Words "},{"metadata":{"trusted":false},"cell_type":"code","source":"fig = plt.figure(figsize=(8,20))\nfor i, filepath in enumerate(sample_audio[:6]):\n    plt.subplot(9,1,i+1)\n    samplerate, test_sound  = wavfile.read(filepath)\n    plt.title(filepath.split('/')[-2])\n    plt.axis('off')\n    plt.plot(test_sound)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 5.2 Waveforms within the Same Word "},{"metadata":{"trusted":false},"cell_type":"code","source":"fig = plt.figure(figsize=(8,20))\nfor i, filepath in enumerate(five_samples):\n    plt.subplot(9,1,i+1)\n    samplerate, test_sound = wavfile.read(filepath)\n    plt.title(filepath.split('/')[-2])\n    plt.axis('off')\n    plt.plot(test_sound)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6. Save Figures as images\n#### Function: convert audio to spectogram images"},{"metadata":{"collapsed":true,"trusted":false},"cell_type":"code","source":"def wav2img(wav_path, targetdir='', figsize=(4,4)):\n    \"\"\"\n    takes in wave file path\n    and the fig size. Default 4,4 will make images 288 x 288\n    \"\"\"\n\n    fig = plt.figure(figsize=figsize)    \n    # use soundfile library to read in the wave files\n    samplerate, test_sound  = wavfile.read(filepath)\n    _, spectrogram = log_specgram(test_sound, samplerate)\n    \n    ## create output path\n    output_file = wav_path.split('/')[-1].split('.wav')[0]\n    output_file = targetdir +'/'+ output_file\n    #plt.imshow(spectrogram.T, aspect='auto', origin='lower')\n    plt.imsave('%s.png' % output_file, spectrogram)\n    plt.close()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Function: convert audio to waveform images"},{"metadata":{"trusted":false},"cell_type":"code","source":"def wav2img_waveform(wav_path, targetdir='', figsize=(4,4)):\n    samplerate,test_sound  = wavfile.read(sample_audio[0])\n    fig = plt.figure(figsize=figsize)\n    plt.plot(test_sound)\n    plt.axis('off')\n    output_file = wav_path.split('/')[-1].split('.wav')[0]\n    output_file = targetdir +'/'+ output_file\n    plt.savefig('%s.png' % output_file)\n    plt.close()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6.1 Loops to iterate through directories and save\n\n#### These are 100% markdown to avoid running on the kaggle site. \n\nCopy and run locally in your instance. I've capped the number of folders and images to 3 and 5 respectively. Remove the list indexing to run through all directories and all images. Becareful, there's 64k training images!\n\n### Convert Training Audio\n\n```python\nfor i, x in enumerate(subFolderList[:3]):\n    print(i, ':', x)\n    # get all the wave files\n    all_files = [y for y in os.listdir(audio_path + x) if '.wav' in y]\n    for file in all_files[:10]:\n        wav2img(audio_path + x + '/' + file, pict_Path + x)\n```\n\n### Convert Testing Audio\n```python\n# get all the wave files\nall_files = [y for y in os.listdir(test_audio_path + x) if '.wav' in y]\nfor file in all_files:\n    wav2img(test_audio_path + x + '/' + file, test_pict_Path + x)\n```"},{"metadata":{"collapsed":true,"trusted":false},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","pygments_lexer":"ipython3","file_extension":".py","version":"3.6.3","codemirror_mode":{"name":"ipython","version":3},"nbconvert_exporter":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}