{"cells":[{"metadata":{},"cell_type":"markdown","source":"<h2>Converting sounds into images: a general guide</h2>\n\nAn easy way to feed sounds into a neural network is to first converting the sounds to images. In order to do so, there exists several ways. The most famous is by creating a spectogram of the sound. This transformation is made possible thanks to a powerful mathematical object: **the Fourier transforms**. We will see in this guide the code to general a mel-spectogram to convert a sound into an image: https://en.wikipedia.org/wiki/Mel-frequency_cepstrum.\n\nAlso if you are interested by Fourier transforms: https://www.youtube.com/watch?v=spUNpyF58BY."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Importing the basic libraries that we need\n\nimport matplotlib.pyplot as plt\nfrom scipy import signal\nfrom scipy.io import wavfile as wav\nimport numpy as np\nfrom numpy.lib import stride_tricks","execution_count":1,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**1st solution**"},{"metadata":{"trusted":false},"cell_type":"code","source":"# Please visit the original version of the code created by Daisukelab: https://www.kaggle.com/daisukelab/cnn-2d-basic-solution-powered-by-fast-ai\n\nimport librosa\nimport librosa.display\n\n# Reading the audio file and applying some transformations (trimming, padding...) to \"clean\" the sound file\n\ndef read_audio(conf, pathname, trim_long_data):\n    y, sr = librosa.load(pathname, sr=conf.sampling_rate)\n    # trim silence\n    if 0 < len(y): # workaround: 0 length causes error\n        y, _ = librosa.effects.trim(y) # trim, top_db=default(60)\n    # make it unified length to conf.samples\n    if len(y) > conf.samples: # long enough\n        if trim_long_data:\n            y = y[0:0+conf.samples]\n    else: # pad blank\n        padding = conf.samples - len(y)    # add padding at both ends\n        offset = padding // 2\n        y = np.pad(y, (offset, conf.samples - len(y) - offset), 'constant')\n    return y\n\n# Thanks to the librosa library, generating the mel-spectogram from the audio file\n\ndef audio_to_melspectrogram(conf, audio):\n    spectrogram = librosa.feature.melspectrogram(audio, \n                                                 sr=conf.sampling_rate,\n                                                 n_mels=conf.n_mels,\n                                                 hop_length=conf.hop_length,\n                                                 n_fft=conf.n_fft,\n                                                 fmin=conf.fmin,\n                                                 fmax=conf.fmax)\n    spectrogram = librosa.power_to_db(spectrogram)\n    spectrogram = spectrogram.astype(np.float32)\n    return spectrogram\n\n# Adding both previous function together\n\ndef read_as_melspectrogram(conf, pathname, trim_long_data, debug_display=False):\n    x = read_audio(conf, pathname, trim_long_data)\n    mels = audio_to_melspectrogram(conf, x)\n    return mels\n\n# A set of settings that you can adapt to fit your audio files (frequency, average duration, number of Fourier transforms...)\n\nclass conf:\n    # Preprocessing settings\n    sampling_rate = 44100\n    duration = 2\n    hop_length = 347*duration # to make time steps 128\n    fmin = 20\n    fmax = sampling_rate // 2\n    n_mels = 128\n    n_fft = n_mels * 20\n    samples = sampling_rate * duration","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def mono_to_color(X, mean=None, std=None, norm_max=None, norm_min=None, eps=1e-6):\n    # Stack X as [X,X,X]\n    X = np.stack([X, X, X], axis=-1)\n\n    # Standardize\n    mean = mean or X.mean()\n    std = std or X.std()\n    Xstd = (X - mean) / (std + eps)\n    _min, _max = Xstd.min(), Xstd.max()\n    norm_max = norm_max or _max\n    norm_min = norm_min or _min\n    if (_max - _min) > eps:\n        # Scale to [0, 255]\n        V = Xstd\n        V[V < norm_min] = norm_min\n        V[V > norm_max] = norm_max\n        V = 255 * (V - norm_min) / (norm_max - norm_min)\n        V = V.astype(np.uint8)\n    else:\n        # Just zero\n        V = np.zeros_like(Xstd, dtype=np.uint8)\n    return V","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# To generate the image dataset, we first need to remove the .wav extension and add the .jpg\n\ndef rename_file(img_name):\n    img_name = img_name.split(\"/\")[2]\n    img_name = img_name[:-4]\n    img_name += \".jpg\"\n    return img_name","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Converting sounds into images is a memory-consuming task, that takes a lot of RAM, especially if you are using Kaggle. In order to deallocate some memory from your kernel, you can call the collect() method from the gc library. Also don't forget to **delete your images** to free some memory."},{"metadata":{"trusted":false},"cell_type":"code","source":"import gc\n\ndef save_image_from_sound(img_path):\n    filename = rename_file(img_path)\n    x = read_as_melspectrogram(conf, img_path, trim_long_data=False, debug_display=True)\n    #x_color = mono_to_color(x)\n    \n    plt.imshow(x, interpolation='nearest')\n    plt.savefig('..input/freesound_audio_tagging/img_noisy_solution2/' + filename)\n    plt.show()\n    \n    plt.close()\n    del x\n    gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"\"\"\"\n\nfor i, fn in enumerate(os.listdir('..input/freesound_audio_tagging/train_noisy')):\n    print(i)\n    path = '..input/freesound_audio_tagging/train_noisy/' + fn\n    save_image_from_sound(path)\n    \n\"\"\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**2nd solution**"},{"metadata":{"trusted":false},"cell_type":"code","source":"import random\nfrom fastai import *\nfrom fastai.vision import *\nfrom fastai.vision.data import *\n\ndef open_fat2019_image(img)->Image:\n    # open\n    x = PIL.Image.fromarray(img)\n    # crop\n    time_dim, base_dim = x.size\n    crop_x = random.randint(0, time_dim - base_dim)\n    x = x.crop([crop_x, 0, crop_x+base_dim, base_dim])    \n    # standardize\n    return Image(pil2tensor(x, np.float32).div_(255))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def save_image_from_sound_2(img_path):\n    filename = rename_file(img_path)\n    x = read_as_melspectrogram(conf, img_path, trim_long_data=False, debug_display=True)\n    x_color = mono_to_color(x)\n    img = open_fat2019_image(x_color)\n    \n    img.show()\n    img.save('..input/freesound_audio_tagging/img_noisy_solution2/' + filename)\n    \n    plt.close()\n    del img\n    gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Since converting those sound files can be time-consuming, you might want to do it by batches, in which case you don't want to generate again the images you had previously. That's why we create an *exists* variable that verifies that the image file already exists or not."},{"metadata":{"scrolled":true,"trusted":false},"cell_type":"code","source":"import os\n\n\"\"\"\nfor i, fn in enumerate(os.listdir('freesound_audio_tagging/train_noisy')):\n    print(i)\n    \n    path = 'freesound_audio_tagging/train_noisy/' + fn\n    exists = os.path.isfile('freesound_audio_tagging/img_noisy_solution2/' + rename_file(path))\n    \n    if not exists:\n        save_image_from_sound_2(path)\n\"\"\"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.7"}},"nbformat":4,"nbformat_minor":1}