{"cells":[{"metadata":{},"cell_type":"markdown","source":"### INTRODUCTION\n* Extraction of features is a very important part in analyzing and finding relations between different things. The data provided of audio cannot be understood by the models directly to convert them into an understandable format feature extraction is used. It is a process that explains most of the data but in an understandable way. Feature extraction is required for classification, prediction and recommendation algorithms.\n\n### PACKAGES TO BE USED\n* We’ll be using librosa for analyzing and extracting features of an audio signal. For playing audio we will use pyAudio so that we can play music on jupyter directly.","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport random\n\nimport pandas as pd\nimport numpy as np\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport matplotlib.image as mpimg\nplt.style.use('fivethirtyeight')\nsns.set_style('whitegrid')\n\nimport librosa\nimport librosa.display\nimport IPython.display as ipd\n\nimport sklearn\nfrom sklearn.preprocessing import minmax_scale\n\nimport warnings\nwarnings.filterwarnings('ignore')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train_folder = '../input/birdsong-recognition/train_audio'\nbirds = [path for path in os.listdir(os.path.join(train_folder))][-5:]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_sound = {}\n\nfor i,bird in enumerate(birds):\n    folder = os.path.join(train_folder, bird)\n    for path in os.listdir(os.path.join(folder)):\n        #get 1 sample sound per bird\n        sample_sound[bird] = os.path.join(folder, path)\n        break","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_sound","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### LET'S HEAR SOME BIRDS","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(birds[0], ' sample sound.')\nipd.Audio(sample_sound[birds[0]])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(birds[1], ' sample sound.')\nipd.Audio(sample_sound[birds[1]])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(birds[2], ' sample sound.')\nipd.Audio(sample_sound[birds[2]])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(birds[3], ' sample sound.')\nipd.Audio(sample_sound[birds[3]])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(birds[4], ' sample sound.')\nipd.Audio(sample_sound[birds[4]])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### LOAD THE SAMPLES AND CHECK INFO.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sound_data = {}\n\nfor i, val in enumerate(sample_sound.values()):\n    y, sr = librosa.load(val)\n    sound_data[birds[i]] = {'y':y, 'sr': sr}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sound_data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in range(len(birds)):\n    print('X shape: ', sound_data[birds[i]]['y'].shape)\n    print('Sampling Rate (KHz): ', sound_data[birds[i]]['sr'])\n    print('='*50)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### FEATURE EXTRACTION AND VISUALIZATION\n---","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### SOUNDWAVES\n* Waveplots let us know the loudness of the audio at a given time.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#function to generate random color\ndef gen_color():\n    color = \"%06x\" % random.randint(0, 0xFFFFFF)\n    color = '#'+ color\n    return color","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(len(birds),1,figsize=(14,10))\nplt.tight_layout(3)\n\nfor i in range(len(birds)):\n    librosa.display.waveplot(y = sound_data[birds[i]]['y'],\n                             sr = sound_data[birds[i]]['sr'],\n                             ax = ax[i], color = gen_color())\n    ax[i].set_title(birds[i].capitalize())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n### SPECTOGRAM\n* A spectrogram is a visual representation of the spectrum of frequencies of sound or other signals as they vary with time. It’s a representation of frequencies changing with respect to time for given music signals.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in range(len(birds)):\n    #perform a short fourier transform on signal amplitude\n    sound_data[birds[i]]['stft'] = librosa.stft(sound_data[birds[i]]['y'])\n    # convert to db\n    sound_data[birds[i]]['ydb'] = librosa.amplitude_to_db(abs(sound_data[birds[i]]['stft']))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"*  STFT(Short Time Fourier Transform) converts signal such that we can know the amplitude of given frequency at a given time. Using STFT we can determine the amplitude of various frequencies playing at a given time of an audio signal. ","execution_count":null},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"#show data\nsound_data[birds[1]]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(len(birds),1,figsize=(20,15))\nplt.tight_layout(3)\n\nfor i in range(len(birds)):\n    librosa.display.specshow(sound_data[birds[i]]['ydb'],\n                             sr = sound_data[birds[i]]['sr'],\n                             x_axis='time', y_axis='hz',\n                             ax = ax[i])\n    ax[i].set_title(birds[i].capitalize())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### ZERO CROSSING RATE\n* The zero crossing rate is the rate of sign-changes along a signal, i.e., the rate at which the signal changes from positive to negative or back. This feature has been used heavily in both speech recognition and music information retrieval.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in range(len(birds)):\n    sound_data[birds[i]]['zcr'] = librosa.zero_crossings(sound_data[birds[i]]['y'], pad=False).sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in range(len(birds)):\n    print(birds[i].capitalize(), 'Zero Crossing Rate: ', sound_data[birds[i]]['zcr'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### SPECTRAL CENTROID\n*  If the frequencies in music are same throughout then spectral centroid would be around a centre and if there are high frequencies at the end of sound then the centroid would be towards its end.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in range(len(birds)):\n    sound_data[birds[i]]['spec_c'] = librosa.feature.spectral_centroid(sound_data[birds[i]]['y'], sr= sound_data[birds[i]]['sr'])[0]\n    frames = range(len(sound_data[birds[i]]['spec_c']))\n    sound_data[birds[i]]['t_frame'] = librosa.frames_to_time(frames)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(len(birds),1,figsize=(14,15))\nplt.tight_layout(3)\n\nfor i in range(len(birds)):\n    librosa.display.waveplot(y = sound_data[birds[i]]['y'],\n                             sr = sound_data[birds[i]]['sr'],\n                             ax = ax[i], color = gen_color())\n    # Normalising the spectral centroid for visualisation\n    ax[i].plot(sound_data[birds[i]]['t_frame'], minmax_scale(sound_data[birds[i]]['spec_c'], axis=0), lw=1)\n    ax[i].set_title(birds[i].capitalize())\n    ax[i].legend(['Spectral Centroid', 'SoundWave'], loc ='upper left');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"### SPECTRAL ROLLOFF\n* Spectral rolloff is the frequency below which a specified percentage of the total spectral energy, e.g. 85%, lies.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in range(len(birds)):\n    sound_data[birds[i]]['spec_r'] = librosa.feature.spectral_rolloff(sound_data[birds[i]]['y'], sr= sound_data[birds[i]]['sr'])[0]\n    frames = range(len(sound_data[birds[i]]['spec_r']))\n    sound_data[birds[i]]['tr_frame'] = librosa.frames_to_time(frames)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(len(birds),1,figsize=(14,15))\nplt.tight_layout(3)\n\nfor i in range(len(birds)):\n    librosa.display.waveplot(y = sound_data[birds[i]]['y'],\n                             sr = sound_data[birds[i]]['sr'],\n                             ax = ax[i], color = gen_color())\n    # Normalising the spectral centroid for visualisation\n    ax[i].plot(sound_data[birds[i]]['tr_frame'], minmax_scale(sound_data[birds[i]]['spec_r'], axis=0), lw=1)\n    ax[i].set_title(birds[i].capitalize())\n    ax[i].legend(['Spectral Roll-off', 'SoundWave'], loc ='upper left');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### BPM","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in range(len(birds)):\n    sound_data[birds[i]]['bpm'] = librosa.beat.beat_track(sound_data[birds[i]]['y'], sr=sound_data[birds[i]]['sr'])[0]\n    print(birds[i],' BPM: ',sound_data[birds[i]]['bpm'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### HARMONICS AND PERCEPTUAL\n\n*  Harmonics - Partial tones that are whole multiples of the fundamental frequency.\n*  Perceptrual shock wave -  Represents the sound rhythm and emotion.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in range(len(birds)):\n    sound_data[birds[i]]['y_harm'], sound_data[birds[i]]['y_perc'] = librosa.effects.hpss(sound_data[birds[i]]['y'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(len(birds),1, figsize=(10,15))\nplt.tight_layout(3)\nfor i in range(len(birds)):\n    ax[i].set_title(birds[i].capitalize())\n    ax[i].plot(sound_data[birds[i]]['y_perc'], color= 'steelblue', lw=1);\n    ax[i].plot(sound_data[birds[i]]['y_harm'], color= 'salmon', lw=1);\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### CHROMA FREQUENCIES\n*  Chroma-based features are a powerful tool for analyzing music whose pitches can be meaningfully categorized (often into twelve categories) and whose tuning approximates to the equal-tempered scale. One main property of chroma features is that they capture harmonic and melodic characteristics of music, while being robust to changes in timbre and instrumentation.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in range(len(birds)):\n    base =  sound_data[birds[i]]\n    base['ch_fr'] = librosa.feature.chroma_stft(base['y'], sr = base['sr'])\n    print(birds[i], ' Chromogram Shape: ', base['ch_fr'].shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(len(birds), 1 , figsize=(10,15))\nplt.tight_layout(3)\n\nfor i in range(len(birds)):\n    ax[i].set_title(birds[i].capitalize())\n    librosa.display.specshow(sound_data[birds[i]]['ch_fr'], x_axis='time', y_axis='chroma', cmap='cividis', ax = ax[i])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### MFCC\n* The mel frequency cepstral coefficients (MFCCs) of a signal are a small set of features (usually about 10–20) which concisely describe the overall shape of a spectral envelope.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in range(len(birds)):\n    base =  sound_data[birds[i]]\n    base['mfcc'] = librosa.feature.mfcc(base['y'], sr = base['sr'])\n    print(birds[i], ' MFCC Shape: ', base['mfcc'].shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(len(birds), 1 , figsize=(10,15))\nplt.tight_layout(3)\n\nfor i in range(len(birds)):\n    ax[i].set_title(birds[i].capitalize())\n    librosa.display.specshow(sound_data[birds[i]]['mfcc'], x_axis='time', y_axis='log', cmap='viridis', ax = ax[i])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### REFERENCE\n* https://towardsdatascience.com/how-to-apply-machine-learning-and-deep-learning-methods-to-audio-analysis-615e286fcbbc\n* https://en.wikipedia.org","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}