{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Basics of Audio Analysis"},{"metadata":{},"cell_type":"markdown","source":"## What is sound?\n* Audio signals are **transverse pressure fluctuations** (compressions and rarefactions of air pressure).\n* When someone talks, it generates **air pressure signals**.\n* The number of times the compression/rarefaction happens is known as the **frequency** of sound wave. \n\n* Below figure shows the pressure signal changes"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nimport librosa ## for working with audio data (https://librosa.org/doc/latest/tutorial.html)\nimport librosa.display ## display functions with matplotlib\n\nfrom pathlib import Path ## for reading the files into the notebook\nfrom IPython.display import Audio ## play audio file\n\nimport matplotlib.pyplot as plt ## visualization","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"## path for training files\nTRAIN_DIR = Path(\"../input/rfcx-species-audio-detection/train\") # files read in\ntrainfiles = list(TRAIN_DIR.glob(\"*.flac\"))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"## check if the list is proper\ntrainfiles[1]","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def visualize_sound_data_waveplot(i, num_mfcc):\n    \n    try:\n             \n        ## read the audio data\n        audio, sr = librosa.load(trainfiles[i]) \n\n        waveplot = librosa.display.waveplot(audio,sr=sr)\n        \n    except Exception as e:\n        print(\"Error encountered while parsing file: \", i)\n        return None ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_sound_data_waveplot(3,13)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## How do we hear sound?\n* Signal processing happens via **cochlea**, a fluid-filled part of the ear with thousands of tiny hairs that are connected to nerves. \n* Some of the hairs are short, and some are relatively longer. \n* The shorter hairs resonate with higher sound frequencies, and the longer hairs resonate with lower sound frequencies. \n* Therefore, *the ear is like a natural Fourier transform analyzer*!\n\n* Power spectogram is a common representation of showing the fourier transform as shown in below figure"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def visualize_sound_data_fft(i):\n    \n    try:\n        \n        fig, (ax1, ax2) = plt.subplots(2, 1)\n        fig.suptitle('Sound Data Waveform and Power spectogram (fourier transform in DB plotted)')\n        \n        ## read the audio data\n        audio, sr = librosa.load(trainfiles[i]) \n\n        waveplot = librosa.display.waveplot(audio,sr=sr, ax=ax1)\n        \n        D = np.abs(librosa.stft(audio))\n        db = librosa.amplitude_to_db(D,ref=np.max)\n        librosa.display.specshow(db, sr=sr, y_axis='log', x_axis='time', ax=ax2)\n          \n        \n        \n        #return img\n    \n    except Exception as e:\n        print(\"Error encountered while parsing file: \", i)\n        return None ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_sound_data_fft(3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## How does the brain perceive sound?\n* The signal from cochnea helps brain recognize that the signal is speech and memory enables to understand what someone/something is saying.\n* Human ear has a preferable range of frequencies it can pick out. This forms a filter on the signal. \n* **GFCC (or MFCC)** - gamma tone frequency cepstral coefficient (Mel Bank Frequency Cepstral Coefficient) is a close simulation of how the ear perceives sound. \n\n\n* Typically 13 cepstral coefficients are sufficient to capture the phenomes. See below figure for the filtered spectral representation. "},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def visualize_sound_data_waveplot_mfcc(i, num_mfcc):\n    \n    try:\n        \n        fig, (ax1, ax2) = plt.subplots(2, 1)\n        fig.suptitle('Sound Data Waveform and MFCC spectogram')\n        \n        ## read the audio data\n        audio, sr = librosa.load(trainfiles[i]) \n\n        waveplot = librosa.display.waveplot(audio,sr=sr, ax=ax1)\n        \n        mfcc = librosa.feature.mfcc(audio,\n                                    sr = sr,\n                                    n_mfcc=num_mfcc)\n            \n        mfccplot = librosa.display.specshow(mfcc, x_axis='time', y_axis='log', ax=ax2)\n          \n        \n        \n        #return img\n    \n    except Exception as e:\n        print(\"Error encountered while parsing file: \", i)\n        return None ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_sound_data_waveplot_mfcc(3,13)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## How do you mathematically transform sound data?\n\nSince sound is basically high frequency time domain data in its raw form, time series features can be used for it's assessment. Moreover, it can be converted to frequency domain for spectral analysis. This is not unlike time series characterisation, although a bit more specialized for the audio signal.\n\n### Time series features\n* Amplitude derived features - sum, length, mean, median etc. \n* Power - transformed amplitude with time weightage\n* Auto regression features\n\n"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"import tsfresh","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Minimal features selected currently for demonstration, a wider list can be selected as well and added based on iterations. "},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def extract_ts_features(i):\n    # read file\n    audio, sr = librosa.load(trainfiles[i])\n    \n    df = pd.DataFrame(audio).reset_index()\n    df.columns = ['time','amplitude']\n    df['col_id']=i\n    \n    settings = tsfresh.feature_extraction.MinimalFCParameters()\n    return tsfresh.feature_extraction.extract_features(df, default_fc_parameters=settings, column_id='col_id', column_sort='time')\n        ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"extract_ts_features(3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Spectral features\n* Zero crossing rate - number of times crossing through 0-axis\n* Spectral roll off - is the frequency below which 85% of accumulated spectral magnitude is concentrated\n* Spectral flux & its derivatives, \n* Spectral bandwidth and its derivaties\n\n"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def extract_spectral_features(i): \n    try:\n             \n        audio, sr = librosa.load(trainfiles[i])\n        #read audio file\n         \n        zcr = sum(librosa.zero_crossings(audio))/len(audio)\n        ## 1. zero crossings rate \n        \n        spectral_centroids = librosa.feature.spectral_centroid(audio, sr=sr)[0]\n        spectral_centroids_delta = librosa.feature.delta(spectral_centroids)\n        spectral_centroids_accelerate = librosa.feature.delta(spectral_centroids, order=2)\n        ## 2. spectral centroids and derivatives\n        \n        spectral_rolloff = librosa.feature.spectral_rolloff(audio, sr=sr)[0]\n        ## 3. spectral roll off\n        \n        onset_env = librosa.onset.onset_strength(y=audio, sr=sr)\n        ## 4. spectral bandwidth\n\n        spectral_bandwidth_2 = librosa.feature.spectral_bandwidth(audio, sr=sr)[0]\n        spectral_bandwidth_3 = librosa.feature.spectral_bandwidth(audio, sr=sr, p=3)[0]\n        spectral_bandwidth_4 = librosa.feature.spectral_bandwidth(audio, sr=sr, p=4)[0]\n        ## 5. spectral bandwidth\n        \n        ## data frame structure definition\n        spectral_features = {\n        \"zero_crossing_rate\": zcr,\n        \"spectral_centroids\": np.mean(spectral_centroids),\n        \"spectral_centroids_delta\": np.mean(spectral_centroids_delta),\n        \"spectral_centroids_accelerate\": np.mean(spectral_centroids_accelerate),\n        \"spectral_rolloff\": np.mean(spectral_rolloff),\n        \"spectral_flux\": np.mean(onset_env),\n        \"spectral_bandwidth_2\": np.mean(spectral_bandwidth_2),\n        \"spectral_bandwidth_3\": np.mean(spectral_bandwidth_3),\n        \"spectral_bandwidth_4\": np.mean(spectral_bandwidth_4),\n        }\n        \n        df = pd.DataFrame.from_records(data=[spectral_features])\n        \n        \n        return df\n    \n    except:\n        print(\"Error encountered while parsing file: \", i)\n        return None","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"extract_spectral_features(3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Chroma and spectogram features\n* Pitch class profiles - used in melody and harmony characteristics\n* Normalized amplitude power spectograms\n\n"},{"metadata":{},"cell_type":"markdown","source":"### Cepstral coefficients\n#### What is cepstrum?\n* The starting point is a fourier transform of the signal.\n* After that, a **cepstrum** is formed by taking the log magnitude of the spectrum \n* followed by an inverse Fourier transform. \n* This results in a signal that's neither in the frequency domain (because we took an inverse Fourier transform) \n* nor in the time domain (because we took the log magnitude prior to the inverse Fourier transform). \n* The domain of the resulting signal is called the quefrency.\n\n#### Cepstral features\n* Mel bank frequency cepstral coefficients\n* Gamma tonal frequency cepstral coefficients\n\n"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def extract_cepstral_features(i, n_fft = 2048, hop_length = 512, num_mfcc = 13): \n    try:\n             \n        audio, sr = librosa.load(trainfiles[i])\n        #read audio file\n         \n        d_audio = np.abs(librosa.stft(audio, n_fft=n_fft, hop_length=hop_length))\n        ## 1. short term fourier transform\n        \n        db_audio = librosa.amplitude_to_db(d_audio, ref=np.max)\n        ## 2. spectogram\n        \n        s_audio = librosa.feature.melspectrogram(audio, sr=sr)\n        s_db_audio = librosa.amplitude_to_db(s_audio, ref=np.max)\n        ## 3. Mel Spectogram\n        \n        y_harm, y_perc = librosa.effects.hpss(audio)\n        ## 4. harmonic and percussion effects\n        \n        chromagram = librosa.feature.chroma_stft(audio, sr=sr, hop_length=hop_length)\n        ## 5. chromagram\n        \n        tempo_y, _ = librosa.beat.beat_track(audio, sr=sr)\n        ## 6. tempo BPM variable\n        \n        mfcc_alt = librosa.feature.mfcc(y=audio, sr=sr, n_mfcc=num_mfcc)\n        delta = librosa.feature.delta(mfcc_alt)\n        accelerate = librosa.feature.delta(mfcc_alt, order=2)\n        ## 7. mfcc features\n        \n        ## data frame structure definition\n        cepstral_features = {\n        \"spectrogram\": np.mean(db_audio[0]),\n        \"mel_spectrogram\": np.mean(s_db_audio[0]),\n        \"harmonics\": np.mean(y_harm),\n        \"perceptual_shock_wave\": np.mean(y_perc),\n        \"chroma1\": np.mean(chromagram[0]),\n        \"chroma2\": np.mean(chromagram[1]),\n        \"chroma3\": np.mean(chromagram[2]),\n        \"chroma4\": np.mean(chromagram[3]),\n        \"chroma5\": np.mean(chromagram[4]),\n        \"chroma6\": np.mean(chromagram[5]),\n        \"chroma7\": np.mean(chromagram[6]),\n        \"chroma8\": np.mean(chromagram[7]),\n        \"chroma9\": np.mean(chromagram[8]),\n        \"chroma10\": np.mean(chromagram[9]),\n        \"chroma11\": np.mean(chromagram[10]),\n        \"chroma12\": np.mean(chromagram[11]),\n        \"tempo_bpm\": tempo_y,\n        }\n        \n        for i in range(0, num_mfcc):\n            key_name = \"\".join(['mfcc', str(i)])\n            mfcc_value = np.mean(mfcc_alt[i])\n            cepstral_features.update({key_name: mfcc_value})\n\n            # mfcc delta coefficient\n            key_name = \"\".join(['mfcc_delta_', str(i)])\n            mfcc_value = np.mean(delta[i])\n            cepstral_features.update({key_name: mfcc_value})\n\n            # mfcc accelerate coefficient\n            key_name = \"\".join(['mfcc_accelerate_', str(i)])\n            mfcc_value = np.mean(accelerate[i])\n            cepstral_features.update({key_name: mfcc_value})\n        \n        df = pd.DataFrame.from_records(data=[cepstral_features])\n        \n        \n        return df\n    \n    except:\n        print(\"Error encountered while parsing file: \", i)\n        return None","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"extract_cepstral_features(3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Deep learning approach\n* Spectogram, chromagraph of mel spectograms converted to images and fed to a neural network. \n\n"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def generate_image_data(i, num_mfcc = 13):\n    \n    try:\n        \n        y, sr = librosa.load(trainfiles[i])\n        #read audio file\n        \n        D = librosa.amplitude_to_db(librosa.stft(y), ref=np.max)\n        plt.subplot(4, 2, 1)\n        librosa.display.specshow(D, y_axis='linear')\n        plt.axis('off')\n        \n        plt.subplot(4, 2, 2)\n        librosa.display.specshow(D, y_axis='log')\n        plt.axis('off')\n\n        CQT = librosa.amplitude_to_db(librosa.cqt(y, sr=sr), ref=np.max)\n        plt.subplot(4, 2, 3)\n        librosa.display.specshow(CQT, y_axis='cqt_note')\n        plt.axis('off')\n        \n        plt.subplot(4, 2, 4)\n        librosa.display.specshow(CQT, y_axis='cqt_hz')\n        plt.axis('off')\n\n        C = librosa.feature.chroma_cqt(y=y, sr=sr)\n        plt.subplot(4, 2, 5)\n        librosa.display.specshow(C, y_axis='chroma')\n        plt.axis('off')\n\n        plt.subplot(4, 2, 6)\n        librosa.display.specshow(D, cmap='gray_r', y_axis='linear')\n        plt.axis('off')\n\n        plt.subplot(4, 2, 7)\n        librosa.display.specshow(D, x_axis='time', y_axis='log')\n        plt.axis('off')\n\n        plt.subplot(4, 2, 8)\n        Tgram = librosa.feature.tempogram(y=y, sr=sr)\n        librosa.display.specshow(Tgram, x_axis='time', y_axis='tempo')\n        plt.axis('off')\n        \n        plt.tight_layout()\n\n        # Draw beat-synchronous chroma in natural time\n\n        plt.figure()\n        tempo, beat_f = librosa.beat.beat_track(y=y, sr=sr, trim=False)\n        beat_f = librosa.util.fix_frames(beat_f, x_max=C.shape[1])\n        Csync = librosa.util.sync(C, beat_f, aggregate=np.median)\n        beat_t = librosa.frames_to_time(beat_f, sr=sr)\n        ax1 = plt.subplot(3,1,1)\n        librosa.display.specshow(C, y_axis='chroma', x_axis='time')\n        plt.axis('off')\n        \n        ax2 = plt.subplot(3,1,2, sharex=ax1)\n        librosa.display.specshow(Csync, y_axis='chroma', x_axis='time',\n                         x_coords=beat_t)\n        plt.axis('off')\n        \n        ax3 = plt.subplot(3,1,3)\n        mfcc = librosa.feature.mfcc(y,sr = sr,n_mfcc=13)\n            \n        mfccplot = librosa.display.specshow(mfcc, x_axis='time', y_axis='log', ax=ax3)\n        \n        plt.axis('off')  \n        \n        plt.tight_layout()\n\n        #return img\n    \n    except Exception as e:\n        print(\"Error encountered while parsing file: \", i)\n        return None ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"generate_image_data(3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Exploration in progress \n* Parallel processing of features to ensure optimal use of Kaggle resources.\n* Does tsfresh with its extensive array of timeseries features capture the essence of audio processing as well? \n* Would a combination of meta data derived from features extracted with images of spectograms lead a better result?\n* Can multiple spectograms be combined together?\n\n### Next steps\n* Feed features to a modeling pipeline and tune\n* Understand image data compatibility to existing networks"},{"metadata":{},"cell_type":"markdown","source":"### Reference for reading \n\nGiven that we as a team don't normally deal with audio data, a reference list is below which has been used to learn some the facts listed above. \n\n\n* https://wiki.aalto.fi/display/ITSP/Waveform\n\n* https://pyvideo.org/pybay-2019/audio-processing-and-ml-using-python.html\n\n* https://heartbeat.fritz.ai/working-with-audio-signals-in-python-6c2bd63b2daf\n\n* https://www.kdnuggets.com/2020/02/audio-data-analysis-deep-learning-python-part-1.html\n\n* https://medium.com/@makcedward/data-augmentation-for-audio-76912b01fdf6\n\n* https://www.preprints.org/manuscript/201804.0258/v2\n\n* https://github.com/mikesmales/Udacity-ML-Capstone/tree/master/Notebooks\n\n* https://www.cambridge.org/core/services/aop-cambridge-core/content/view/S2048770314000122\n\n* https://dropsofai.com/sound-wave-basics-every-data-scientist-must-know-before-starting-analysis-on-audio-data/\n\n* https://towardsdatascience.com/understanding-audio-data-fourier-transform-fft-spectrogram-and-speech-recognition-a4072d228520\n\n* https://towardsdatascience.com/how-i-understood-what-features-to-consider-while-training-audio-files-eedfb6e9002b\n\n* https://www.kaggle.com/gvyshnya/parallel-audio-feature-extraction-with-mp/\n\n"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}