{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Table of Contents\n\n* [Exploratory Data Analysis](#Header)\n    - [Metadata](#Metadata)\n        - [Missing Values](#MissingValues)\n        - [Species](#Species)\n        - [Date and Time](#DateTime)\n        - [Recordists](#Recordists)\n        - [Location](#Location)\n* [Audio Feature Extraction](#AudioFeatureExtraction)\n    - [Waveform](#Waveform)\n    - [Autocorrelation](#Autocorrelation)\n    - [Spectrogram](#Spectrogram)\n    - [Chromagram](#Chromagram)\n    - [Spectral](#Spectral)\n        - [Centroid](#Centroid)\n        - [Bandwidth](#Bandwidth)\n        - [Contrast](#Contrast)\n        - [Flatness](#Flatness)\n        - [Rolloff](#Rolloff)\n    - [MFCC](#MFCC)\n\n* [Afterword](#Thanks)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"<a id=\"Environment\"></a>\n## Environment","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport sys\nimport librosa\nimport librosa.display\nimport librosa.feature\nimport numpy as np\nimport pandas as pd\nimport plotly.express as xp\nimport plotly.graph_objects as go\nimport matplotlib.pyplot as plt\nimport IPython.display as ipd\nfrom sklearn.preprocessing import minmax_scale\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"Header\"></a>\n# Exploratory Data Analysis","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"<a id=\"Metadata\"></a>\n## Metadata\n\nThe train.csv file contains the metadata of the recording sample for that entry. From that metadata, we get these relevant features:\n\n- ebird_code\n- date/time\n- location\n- recordist\n- filename","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/birdsong-recognition/train.csv')\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"tags":[],"trusted":true},"cell_type":"code","source":"train.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"MissingValues\"></a>\n### Missing Values\n\nWe need to check to see if any of our relevant features are missing values and, if so, what to do about that.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"missing = train.isna().sum().sort_values(ascending=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"missing = missing[missing != 0]\nxp.bar(x=missing.index, y=missing, text=missing, title='Missing Values by Feature', labels={'x':'Features', 'y':'Quantity'})","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Luckily, none of our relevant data is missing","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"<a id=\"Species\"></a>\n### Species\nLet's explore the distribution of the bird species among samples. We'll use the ebird code instead of the full species name.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"counts = train['ebird_code'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"xp.bar(x=counts.index, y=counts, title='Species Distribution (by ebird code)', labels={'x':'Ebird Code', 'y':'Quantity'})","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Exactly 100 samples for about half of species in question\n- Redhead is minimum at 9 samples","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"<a id=\"DateTime\"></a>\n### Date and Time\nThe date and time of the recording could have an impact on which bird is making the call. Some birds may usually call only at certain times, and some birds are only in certain locations during certain times of the year.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# split datetime into separate dataframe\ndatetime = train[['date', 'time']]\ndatetime.date = pd.to_datetime(datetime.date, errors='coerce').dropna()\ndatetime['hour'] = pd.to_numeric(datetime.time.str.split(':', expand=True)[0], errors='coerce')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ax1 = datetime.date.value_counts().sort_values().plot(figsize=(10,6), title='Recordings by Date')\n\nax1.set_xlabel('Date')\nax1.set_ylabel('Quantity')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Majority of recordings taken in the past decade\n- Interesting spike around 2003\n- Cyclical spikes after 2013","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ax2 = datetime['hour'].value_counts().sort_index().plot(figsize=(10,6), title='Recordings by Time', kind='bar', figure=plt.figure())\n\nax2.set_xlabel('Hour')\nax2.set_ylabel('Quantity')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Most recordings taken between 6AM and 12PM\n- Gradual decrease as the day moves on from 8AM","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"<a id=\"Recordists\"></a>\n### Recordists\nWho recorded the samples? This could be important as certain recordists may have a particular interest in certain birds.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ax3 = train['recordist'].value_counts().sort_values(ascending=False).head(20).sort_values().plot(figsize=(10, 6), title='Recordings by Recordist', figure=plt.figure(), kind='barh', fontsize=9)\n\nax3.set_xlabel('Hour')\nax3.set_ylabel('Quantity')\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Majority of recordings made by only two people","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"<a id=\"Location\"></a>\n### Location\nCertain birds only inhabit certain areas. Therefore, we need to take location into account.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"counts = train['country'].value_counts().sort_values(ascending=False).head(10).sort_values()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"xp.bar(y=counts.index, x=counts, title='Number of Recordings by Country', labels={'y':'Country', 'x':'Quantity'}, orientation='h')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"coords = train.groupby(['latitude', 'longitude'], as_index=False)['ebird_code'].agg('count')\ncoords = coords[coords.latitude != 'Not specified']\ncoords = coords[coords.longitude != 'Not specified']\nxp.scatter_geo(lat=coords['latitude'], lon=coords['longitude'], title='Recording Locations')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Vast majority of data comes from North America, specifically from USA\n- Very little data from Africa and Asia","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"<a id=\"AudioFeatureExtraction\"></a>\n# Audio Feature Extraction\n\n<a id=\"Sample\"></a>\n## Sample\nFirst, let's take 5 audio samples from the first 5 birds. We'll look at the waveforms and listen to the songs.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"bird_codes = train.ebird_code.unique()[:5]\n\naudio = []\nfor bird in range(len(bird_codes)):\n    filename = train[train['ebird_code'] == bird_codes[bird]]['filename'].iloc[0]\n    path = os.path.join('../input/birdsong-recognition/train_audio/', bird_codes[bird], filename)\n    \n    # wave plot\n    plt.figure(figsize=(15,10))\n    plt.subplot(len(bird_codes), 1, bird+1)\n    data, srate = librosa.load(path)\n    librosa.display.waveplot(data, sr=srate)\n    plt.gca().set_title(bird_codes[bird])\n    plt.xticks([],[])\n    plt.xlabel('')\n    plt.show()\n    \n    # audio display\n    audio = ipd.Audio(path)\n    ipd.display(audio)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"Features\"></a>\n## Features\n\nAfter doing some research on audio signal classification, I have come up with the following features to extract from the audio files:\n\n- [Waveform](#Waveform)\n- [Autocorrelation](#Autocorrelation)\n- [Spectrogram](#Spectrogram)\n- [Chromagram](#Chromagram)\n- [Spectral](#Spectral)\n    - [Centroid](#Centroid)\n    - [Bandwidth](#Bandwidth)\n    - [Contrast](#Contrast)\n    - [Flatness](#Flatness)\n    - [Rolloff](#Rolloff)\n- [MFCC](#MFCC)\n\nWe'll do a sample feature extraction of bird code 'ameavo' as an example. (filename XC99571.mp3)\n\n<a id=\"Waveform\"></a>\n### Waveform","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"data, srate = librosa.load('../input/birdsong-recognition/train_audio/ameavo/XC99571.mp3')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# plot waveform as refresher\nplt.figure(figsize=(15,5))\nlibrosa.display.waveplot(data, sr=srate)\nplt.gca().set_title('ameavo')\nplt.xticks([],[])\nplt.xlabel('')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"Autocorrelation\"></a>\n### Autocorrelation\n\nAutocorrelation compares a signal with a lagged version of itself. It's main purpose is to find repeated patterns in a sample that might be hidden by noise.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"autocorrelation = librosa.autocorrelate(data, max_size=5000)","execution_count":null,"outputs":[]},{"metadata":{"tags":[],"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15,5))\nplt.plot(autocorrelation)\nplt.gca().set_title('Autocorrelation by Lag Time')\nplt.xlabel('Lag')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- autocorrelation very quickly falls off reaching almost 0 after a lag of about 500","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"<a id=\"Spectrogram\"></a>\n### Spectrogram\n\nThe spectrogram is a visual representation of a signal's spectrum of frequencies over time. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"spectrogram = librosa.stft(data)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(20,10))\nlibrosa.display.specshow(librosa.amplitude_to_db(abs(spectrogram)), sr=srate, x_axis='time', y_axis='hz')\nplt.xlabel('Time', fontsize=20)\nplt.ylabel('Frequency Band')\nplt.colorbar()\nplt.title('Spectrogram', fontsize=20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Pitch seems to hover around 2000 to 3500 Hz most of the time\n- Some spikes to 5500-7000 Hz","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"<a id=\"Chromagram\"></a>\n### Chromagram\n\nThe Chromagram is a visual representation of a signal's chroma feature. The chroma feature at any point in time is the intensity for each chroma value in the set {C, C♯, D, D♯, E , F, F♯, G, G♯, A, A♯, B}. These values are the rows of the chromagram.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"chroma = librosa.feature.chroma_stft(data, sr=srate)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(20,10))\nlibrosa.display.specshow(chroma, x_axis='time', y_axis='chroma')\nplt.xlabel('Time', fontsize=20)\nplt.ylabel('Chroma Value', fontsize=20)\nplt.colorbar()\nplt.clim(0,1)\nplt.title('Chromagram', fontsize=20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"Spectral\"></a>\n### Spectral Features\n\n<a id=\"Centroid\"></a>\n#### Spectral Centroid\n\nSpectral Centroid is a measurement of the \"center of gravity\" of the signal and is a common metric of timbre in a sound sample. It's essentially the dominant frequency at each point.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"centroid = librosa.feature.spectral_centroid(data)[0]","execution_count":null,"outputs":[]},{"metadata":{"tags":[],"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15,5))\nlibrosa.display.waveplot(data, sr=srate)\nplt.plot(librosa.frames_to_time(range(len(centroid))), minmax_scale(centroid), color='g')\nplt.gca().set_title('Spectral Centroid by Frame')\nplt.xlabel('Frame')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"Bandwidth\"></a>\n#### Spectral Bandwidth\n\nSpectral bandwidth represents the range between the lowest and highest frequency bands of the signal at a certain time.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"bandwidth = librosa.feature.spectral_bandwidth(data, sr=srate)[0]","execution_count":null,"outputs":[]},{"metadata":{"tags":[],"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15,5))\nlibrosa.display.waveplot(data, sr=srate)\nplt.plot(librosa.frames_to_time(range(len(bandwidth))), minmax_scale(bandwidth))\nplt.gca().set_title('Spectral Bandwidth by Time')\nplt.xlabel('Time')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- pure noise portions of sample are higher in bandwidth\n\n<a id=\"Contrast\"></a>\n#### Spectral Contrast\n\nSpectral contrast compares the max and min frequency values for each frequency band at a point in time. Thus, spectral contrast gives a robust measure of relative spectral characteristics.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"contrast = librosa.feature.spectral_contrast(data, sr=srate)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(20,10))\nlibrosa.display.specshow(contrast, x_axis='time')\nplt.xlabel('Time', fontsize=20)\nplt.colorbar()\nplt.title('Spectral Contrast', fontsize=20)\nplt.ylabel('Frequency Band', fontsize=20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- highest contrast occurs in edge frequency bands","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"<a id=\"Flatness\"></a>\n#### Spectral Flatness\n\nSpectral flatness compares the arithmetic and geometric means of the power spectrum. It is most often used to identify and separate tones versus noise.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"flatness = librosa.feature.spectral_flatness(data)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(20,10))\nlibrosa.display.specshow(flatness, x_axis='time')\nplt.xlabel('Time', fontsize=20)\nplt.colorbar()\nplt.clim(0,1)\nplt.title('Spectral Flatness', fontsize=20)\nplt.ylabel('Frequency Band', fontsize=20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Maximum value of .2 at points\n- Low noise in general\n\n<a id=\"Rolloff\"></a>\n#### Spectral Rolloff\n\nSpectral rolloff is the frequency under which a specified percentage of the energy lies","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"rolloff = librosa.feature.spectral_rolloff(data, sr=srate)[0]","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15,5))\nlibrosa.display.waveplot(data, sr=srate)\nplt.plot(librosa.frames_to_time(range(len(rolloff))), minmax_scale(rolloff))\nplt.gca().set_title('Spectral Bandwidth by Time')\nplt.xlabel('Time')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"MFCC\"></a>\n### MFCC\n\nMel-Frequency Cepstral Coefficients are a collection of coefficients that together give a representation of the overall spectral envelope of a signal. Probably the most common and important feature of audio signal processing in machine learning.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"mfcc = librosa.feature.mfcc(data, sr=srate, n_mfcc=30)","execution_count":null,"outputs":[]},{"metadata":{"tags":[],"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(20,10))\nlibrosa.display.specshow(minmax_scale(mfcc, axis=1), x_axis='time')\nplt.xlabel('Time', fontsize=20)\nplt.colorbar()\nplt.clim(0,1)\nplt.title('Mel-Frequency Cepstral Coefficients', fontsize=20)\nplt.show()\n\nprint()\nprint('MFCCs calculated: %d' % mfcc.shape[0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"Thanks\"></a>\n# Thank You for Reading!\n\nI am still very much new to data science, and I'm jumping in head-first. This is meant as a learning experience to help me learn some signal processing and audio classification techniques as well as a simple EDA and FE for those who aren't well-versed in audio processing. I invite any and all constructive feedback!\n\nThanks again! Hope this is helpful to you.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Resources\nHere are the major resources that I used while doing my research.\n\n- [Autocorrelation Wiki](https://en.wikipedia.org/wiki/Autocorrelation)\n- [Sanket Doshi - Music Feature Extraction in Python](https://towardsdatascience.com/extract-features-of-music-75a3f9bc265d)\n- [Spectral Features (IPython Notebook)](https://musicinformationretrieval.com/spectral_features.html#:~:text=Spectral%20contrast%20considers%20the%20spectral,difference%20in%20each%20frequency%20subband.)","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}