{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Audio Processing Using Librosa for Beginners\n**Cornell Birdcall Identification : Build tools for bird population monitoring**\n\n-----------------------------------------------------------------------------------------------------------------------------------\n\n#### This Notebook is for absolute beginners who is starting out in audio processing. In this notebook, we will be using Librosa Library to load and analyze the audios.\n\n![](https://i.imgur.com/jsKJ4rF.jpg) \n \n## Table of Contents:\n* [**Introduction**](#intro)\n* [**Libraries used**](#lib)\n* [**Relevant features for Loading Audios**](#rel)\n* [**Selecting a Species for Analysis**](#sel)\n* [**Audio Loading**](#load)\n* [**Visualizing the Audio**](#viz)\n* [**Features**](#feat)\n    * [Mel-Frequency Cepstral Coefficients](#mcc)\n    * [Zero-crossing rate](#zcr)\n    * [Spectral Centroid](#cen)\n    * [Spectral Bandwidth](#band)\n    * [Spectral-roll off](#roll)\n    * [Chroma Vector feature](#chroma)\n    * [Pitch and Magnitude](#pit)\n* [**Conclusion**](#conc)\n \n\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"<a id=\"intro\"></a>\n# Introduction\n\nOver 10,000 bird species occur in the world, and they can be found in nearly every environment, from untouched rainforests to suburbs and even cities. Birds play an essential role in nature. They are high up in the food chain and integrate changes occurring at lower levels. As such, birds are excellent indicators of deteriorating habitat quality and environmental pollution. However, it is often easier to hear birds than see them. With proper sound detection and classification, researchers could automatically intuit factors about an area’s quality of life based on a changing bird population.\n\nIn this Kaggle competition, **The Cornell Lab of Ornithology’s Center for Conservation Bioacoustics**(CCB) provided a wide variety of bird vocalizations in soundscape recordings. Indepth and efficient analysis and inference on this can help the researchers to make better steps towards nature conservation.\n\nThis is a wide area and it requires strong technical knowledge to give out a useful result. The first and foremost task towards this goal is analyzing the audio given to us and deduce some information from those.\n\nWorking with audios are not a common task a typical Data science enthusiast goes through. **This notebook will help you with some basic ideas and concepts to start out with audio processing. So let's dive in...**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"<a id=\"lib\"></a>\n# Libraries Used\n\n## Librosa\n> LibROSA is a python package for music and audio analysis. It provides the building blocks necessary to create music information retrieval systems. The documentation for the library is available [here](https://librosa.org/librosa/). Go through the official tutorial of the same [here](https://librosa.org/librosa/tutorial.html). In case you are the person who always want to eat it from the research paper, [here](http://conference.scipy.org/proceedings/scipy2015/pdfs/brian_mcfee.pdf) it is.**\n\n## IPython.display.Audio\n> IPython.display.Audio will let you play audio directly in a jupyter notebook.**\n\nBefore getting into the technical complications and the code syntax, it is important to know how sound works. It is not mandatory but always a upperhand to know and understand the math behind sounds. A general understanding is cool too. This video, [What's Noise](https://www.youtube.com/watch?v=i_0DXxNeaQ0) will help you understand a bit about sound, if not all.\n\nAn audio signal is represented in the form of an audio signal having parameters such as frequency, bandwidth, decibel etc. A typical audio signal can be expressed as a function of Amplitude and Time. The audio signal is a three-dimensional signal in which three axes represent time, amplitude and frequency.\n\n![Imgur](https://i.imgur.com/5c82I1B.jpg)","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        os.path.join(dirname, filename);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**First let's read the train.csv file and understand the features inside.**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/birdsong-recognition/train.csv')\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"rel\"></a>\n## Relevant features in the train.csv data are:\n\n* ebird_code: a code for the bird species. You can review detailed information about the bird codes by appending the code to https://ebird.org/species/, such as https://ebird.org/species/amecro for the American Crow.\n* recodist: the user who provided the recording.\n* location: where the recording was taken. Some bird species may have local call 'dialects', so you may want to seek geographic diversity in your training data.\n* date: while some bird calls can be made year round, such as an alarm call, some are restricted to a specific season. You may want to seek temporal diversity in your training data.\n* filename: the name of the associated audio file.\n* species: the full name of the species.\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"<a id=\"sel\"></a>\n## Selecting a Species\n\n**From the dataset, we can focus on a certain bird species and analyze the sound/call of that particular bird. The ebird_code can be used to know more about that bird, corresponding filename can be used to deduce the audio from the input and the species name is available in the species column.**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head(1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> **Consider the species Alder Flycatcher. The ebird_code of the species is aldfly and the audio of the species is stored as XC134874.mp3 in aldfly directory inside train_audio. Using these details, we can display the extended information of the particular species from the [ebird](https://ebird.org/species/) site.**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"bird_code = train['ebird_code'].iloc[0]\nfilename = train['filename'].iloc[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from IPython.display import IFrame\n\nebird_code_path = 'https://ebird.org/species/'\nIFrame(ebird_code_path+bird_code, width=1200, height=600)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"load\"></a>\n## Loading the Audio\n\n**Using the librosa library we can now load the audio from the filename given in the data. We can also playback the audio using Ipython.display library.**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import librosa\naudio_data = '/kaggle/input/birdsong-recognition/train_audio/'\nx , sr = librosa.load(audio_data+bird_code+'/'+filename)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**.load loads an audio file and decodes it into a 1-dimensional array which is a time series x , and sr is a sampling rate of x . Default sris 22kHz. We can override the sr using the sr parameter in load().**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(type(x), type(sr))\nprint(x.shape, sr)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import IPython.display as ipd\nipd.Audio(audio_data+bird_code+'/'+filename)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"ext\"></a>\n## Feature Extraction\n\nThe data provided of audio cannot be understood by the models directly. So it is required to convert them into an understandable format. For this, feature extraction is used. It is a very important part in analysis and classification of the audio. We got to know feature extraction is inevitable in the case of audios. Now let's go through some of the techniques used for extracting features of music.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"<a id=\"viz\"></a>\n## Visualizing the Audio\n### Waveplot\n**Here, we plot a naive waveplot of the audio. Waveplots let us know the loudness of the audio at a given time.**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%matplotlib inline\nimport matplotlib.pyplot as plt\nimport librosa.display\nplt.figure(figsize=(14, 5))\nlibrosa.display.waveplot(x, sr=sr)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Spectogram\n\n**A spectrogram is a visual representation of the spectrum of frequencies of sound as they vary with time. It’s a representation of frequencies changing with respect to time for given music signals.'.stft' converts data into short term Fourier transform. STFT converts signal such that we can know the amplitude of given frequency at a given time. Using STFT we can determine the amplitude of various frequencies playing at a given time of an audio signal.**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"X = librosa.stft(x)\nXdb = librosa.amplitude_to_db(abs(X))\nplt.figure(figsize=(14, 5))\nlibrosa.display.specshow(Xdb, sr=sr, x_axis='time', y_axis='hz')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"feat\"></a>\n**For any ML experiment, it is required to transform the collected data to features, which can then be fed into an algorithm. Below are some of the most important features that may be needed to build a model for an audio classification task:**\n1. Mel-Frequency Cepstral Coefficients\n2. Zero-crossing rate\n3. Spectral Centroid\n4. Spectral Bandwidth\n5. Spectral-roll off\n6. Chroma Vector feature\n7. Pitch and Magnitude","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"<a id=\"mcc\"></a>\n## 1. Mel Frequency Cepstral Coefficients (MFCCs)\n**The mel frequency cepstral coefficients (MFCCs) of a signal are a small set of features (usually about 10-20) which concisely describe the overall shape of a spectral envelope. In MIR, it is often used to describe timbre.**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#mcc\nmfccs = librosa.feature.mfcc(y=x, sr=sr)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(mfccs.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In this case, mfcc computed 20 MFCCs over 1098 frames.\n\nThe very first MFCC, the 0th coefficient, does not convey information relevant to the overall shape of the spectrum. It only conveys a constant offset, i.e. adding a constant value to the entire spectrum. Therefore, many practitioners will discard the first MFCC when performing classification.\n\nDisplay the MFCCs:\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"librosa.display.specshow(mfccs, sr=sr, x_axis='time')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"zcr\"></a>\n## 2. Zero Crossing Rate\nThe zero crossing rate indicates the number of times that a signal crosses the horizontal axis.\nLet's Zoom in and make the wave simpler so that we can calculate Zero crossing rate manually and verify.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#zero crossing\nn0 = 7000\nn1 = 7025\nplt.figure(figsize=(14, 5))\nplt.plot(x[n0:n1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"zero_crossings = librosa.zero_crossings(x[n0:n1], pad=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"zero_crossings.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"zero_crossings.sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This gives the zero_crossings. We can verify this manually from the plot itself. There are clearly 7 zero crossings in the plot. Now we can use the whole audio to go through this and deduce the zero crossings of the whole data.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"zcrs = librosa.feature.zero_crossing_rate(x)\nprint(zcrs.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(14, 5))\nplt.plot(zcrs[0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<div id=\"cen\"></a>\n## 3. Spectral Centroid\nThe spectral centroid (Wikipedia) indicates at which frequency the energy of a spectrum is centered upon. This is like a weighted mean:\n> fc=∑kS(k)f(k)∑kS(k)\n\nwhere S(k) is the spectral magnitude at frequency bin k, f(k) is the frequency at bin k.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"spectral_centroids = librosa.feature.spectral_centroid(x, sr=sr)[0]\nspectral_centroids.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"frames = range(len(spectral_centroids))\nt = librosa.frames_to_time(frames)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import sklearn\ndef normalize(x, axis=0):\n    return sklearn.preprocessing.minmax_scale(x, axis=axis)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"librosa.display.waveplot(x, sr=sr, alpha=0.4)\nplt.plot(t, normalize(spectral_centroids), color='r')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"band\"></a>\n## 4.Spectral Bandwidth\n\nlibrosa.feature.spectral_bandwidth computes the order-p\n\nspectral bandwidth:\n> (∑kS(k)(f(k)−fc)p)1p\n\nwhere S(k)\nis the spectral magnitude at frequency bin k, f(k) is the frequency at bin k, and fc is the spectral centroid. When p=2, this is like a weighted standard deviation.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"spectral_bandwidth_2 = librosa.feature.spectral_bandwidth(x+0.01, sr=sr)[0]\nspectral_bandwidth_3 = librosa.feature.spectral_bandwidth(x+0.01, sr=sr, p=3)[0]\nspectral_bandwidth_4 = librosa.feature.spectral_bandwidth(x+0.01, sr=sr, p=4)[0]\nlibrosa.display.waveplot(x, sr=sr, alpha=0.4)\nplt.plot(t, normalize(spectral_bandwidth_2), color='r')\nplt.plot(t, normalize(spectral_bandwidth_3), color='g')\nplt.plot(t, normalize(spectral_bandwidth_4), color='y')\nplt.legend(('p = 2', 'p = 3', 'p = 4'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"roll\"></a>\n## 5. Spectral-roll off\nSpectral rolloff is the frequency below which a specified percentage of the total spectral energy lies.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#Spectral-roll off\nspectral_rolloff = librosa.feature.spectral_rolloff(x+0.01, sr=sr)[0]\nlibrosa.display.waveplot(x, sr=sr, alpha=0.4)\nplt.plot(t, normalize(spectral_rolloff), color='r')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"chroma\"></a>\n## 6. Chroma Feature\nA chroma vector (Wikipedia) (FMP, p. 123) is a typically a 12-element feature vector indicating how much energy of each pitch class, {C, C#, D, D#, E, ..., B}, is present in the signal.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#chroma features\nchromagram = librosa.feature.chroma_stft(x, sr=sr, hop_length=512)\nplt.figure(figsize=(15, 5))\nlibrosa.display.specshow(chromagram, x_axis='time', y_axis='chroma', hop_length=512, cmap='coolwarm')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"pit\"></a>\n## 7. Pitches and Magnitudes\nPitch is a perceptual property of sounds that allows their ordering on a frequency-related scale, or more commonly, pitch is the quality that makes it possible to judge sounds as \"higher\" and \"lower\" in the sense associated with musical melodies.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"pitches, magnitudes = librosa.piptrack(y=x, sr=sr)\nprint(pitches)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(magnitudes)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"conc\"></a>\n# Conclusion\n**These are some of the features that can be used for classification of audio data. This could help not just in this dataset. It is applicable in almost any audio processing. The notebook is specifically created in order to make it most general as possible so that we can use it for any audio related data. I used the official page of Librosa and the github repo to make this notebook. Kudos to all the heads behind this library. It really makes life easy for Data scientist who want to play around with music. I hope this help beginners start out in Audio processing and analysis field. The content of the notebook is kind of a starter kit. There's a whole lot to explore. So why waiting? Get Started...**","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}