{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 🐦 About the competition\n\n ### 🥅 Goal\n \n **Indentify the species of the birds given raw audio in the wild.**\n \n ### 🧠 Understanding the given data\n \n **Training data:**\n * `train_short_audio`: Directory of short recordings of individual bird calls generously uploaded by users of [xenocanto.org](https://www.xeno-canto.org/). **This data will be used to train fine-grained audio classifier**. The audio recordings are downsampled to 32kHz and are in [ogg format](https://en.wikipedia.org/wiki/Ogg).\n * `train_soundscapes`: Directory of audio files that are comparable to the data that we will encounter in the test set. **This is the raw audio that's mentioned in the goal**. The recordings are are all roughly ten minutes long and in the ogg format. \n * `train_metadata.csv`: Wide range of metadata provided for the training data. \n * `train_soundscape_labels.csv`: This `csv` can be used to build the inference pipeline. \n \n**Testing data:**\n * `test_soundscapes`: Directory of recordings to be used for scoring. There are approximately 80 recordings **during submission** that will be will be roughly 10 minutes long and in ogg audio format.\n * `test.csv`: Same tabular information as `train_soundscape_labels.csv`.\n \n### Evaluation \n\nIn the `submission.csv`, for each `row_id/time window`, you need to provide a space delimited list of the set of unique birds that made a call beginning or ending in that time window. If there are no bird calls in a time window, use the code `nocall`.\n\nThe submissions will be evaluated based on their row-wise micro averaged [F1 score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html).","metadata":{}},{"cell_type":"markdown","source":"### About Weights and Biases\n\nThink of W&B like GitHub for machine learning models. With a few lines of code, save everything you need to debug, compare and reproduce your models — architecture, hyperparameters, model weights, GPU usage, and even datasets and predictions.\n\n* Create an account on https://wandb.ai.\n* Input your personal API token key to login (mine is added as [Kaggle Secrets](https://www.kaggle.com/product-feedback/114053))\n","metadata":{}},{"cell_type":"code","source":"%%capture \n# To get the latest version of W&B\n!pip install wandb --upgrade\n!pip install tensorflow-io","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import tensorflow as tf\nimport tensorflow_io as tfio\n\nimport os\nos.environ['WANDB_SILENT'] = \"true\"\nimport re\nimport gc\nimport glob\nimport wandb\nimport random\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport plotly.graph_objects as go\n%matplotlib inline\n\nimport IPython.display as ipd\n\n# Map libraries\nimport descartes\nimport geopandas as gpd\nfrom shapely.geometry import Point, Polygon\n\nfrom kaggle_secrets import UserSecretsClient\n\n# Audio specific imports\nimport librosa as lb\nimport librosa.display\n\n# W&B login\nuser_secrets = UserSecretsClient()\nwandb_api = user_secrets.get_secret(\"wandb_api\")\nwandb.login(key=wandb_api)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 📁 1. The CSV Files","metadata":{}},{"cell_type":"markdown","source":"## `train_metadata.csv`","metadata":{}},{"cell_type":"code","source":"METADATA_FILE_PATH = '../input/birdclef-2021/train_metadata.csv'\nmetadata_df = pd.read_csv(METADATA_FILE_PATH)\nmetadata_df.head(5)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load the file as W&B artifact. \nrun = wandb.init(project='birdclef', group='EDA')\nartifact = wandb.Artifact('train-metadata', type='dataset')\nartifact.add_file(METADATA_FILE_PATH)\nrun.log_artifact(artifact)\nrun.finish()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> 📌 `primary_label`: The primary bird sound that can be heard in the recording. <br>\n> 📌 `secondary_label`: Extra birds sounds present in the recording. <br>\n> 📌 `type`: The type of bird sound. <br>\n> 📌 `latitude` and `longitude`: Location where the recording was done. <br>\n> 📌 `scientific_name`: The scientific name of the bird species. <br>\n> 📌 `common_name`: The bird known in common language. <br>\n> 📌 `author`: Individual's name who recorded (possibly) and uploaded the audio. <br>\n> 📌 `date`: The date the audio was recorded (possibly) and uploaded. <br>\n> 📌 `filename`: The name of the audio file. <br>\n> 📌 `license`: The license associated with that recording. <br>\n> 📌 `rating`: The audio quality. <br>\n> 📌 `time`: The time of the day the recording was uploaded. <br>\n> 📌 `url`: The xenocanto.org url to d","metadata":{}},{"cell_type":"markdown","source":"#### What's the distribution of primary labels?","metadata":{}},{"cell_type":"code","source":"# Reference: https://www.kaggle.com/shahules/bird-watch-complete-eda-fe\n# Unique eBird codes\nspecies = metadata_df['primary_label'].value_counts()\n\n# Make bar chart\nfig = go.Figure(data=[go.Bar(y=species.values, x=species.index)],\n                layout=go.Layout(margin=go.layout.Margin(l=0, r=0, b=10, t=50)))\n\n# Show chart\nfig.update_layout(title='Number of traning samples per species')\nfig.show()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### What quality of audio recording available?","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(16, 6))\nax = sns.countplot(x = metadata_df['rating'], palette=\"hls\", order = metadata_df['rating'].value_counts().index)\n\nplt.title(\"Sound quality rating\", fontsize=16)\nplt.ylabel(\"Count\", fontsize=14)\nplt.yticks(fontsize=13)\nplt.xticks(rotation=45, fontsize=13)\nplt.xlabel(\"\");","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> 📌 Thankfully most of the audio is high quality. ","metadata":{}},{"cell_type":"markdown","source":"#### Where are the birds located?","metadata":{}},{"cell_type":"code","source":"# Reference: https://www.kaggle.com/andradaolteanu/birdcall-recognition-eda-and-audio-fe\n# SHP file\nworld_map = gpd.read_file(\"../input/world-shape-file/99bfd9e7-bb42-4728-87b5-07f8c8ac631c2020328-1-1vef4ev.lu5nk.shp\")\n\n# Coordinate reference system\ncrs = {\"init\" : \"epsg:4326\"}\n\n# Lat and Long need to be of type float, not object\ndata = metadata_df[metadata_df[\"latitude\"] != \"Not specified\"]\ndata[\"latitude\"] = data[\"latitude\"].astype(float)\ndata[\"longitude\"] = data[\"longitude\"].astype(float)\n\n# Create geometry\ngeometry = [Point(xy) for xy in zip(data[\"longitude\"], data[\"latitude\"])]\n\n# Geo Dataframe\ngeo_df = gpd.GeoDataFrame(data, crs=crs, geometry=geometry)\n\n# Create ID for species\nspecies_id = geo_df[\"primary_label\"].value_counts().reset_index()\nspecies_id.insert(0, 'ID', range(0, 0 + len(species_id)))\n\nspecies_id.columns = [\"ID\", \"primary_label\", \"count\"]\n\n# Add ID to geo_df\ngeo_df = pd.merge(geo_df, species_id, how=\"left\", on=\"primary_label\")\n\n# === PLOT ===\nfig, ax = plt.subplots(figsize = (16, 10))\nworld_map.plot(ax=ax, alpha=0.4, color=\"grey\")\n\npalette = iter(sns.hls_palette(len(species_id)))\n\nfor i in range(264):\n    geo_df[geo_df[\"ID\"] == i].plot(ax=ax, markersize=20, color=next(palette), marker=\"o\", label = \"test\");","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> 📌 Most of the audio recordings come from North and South America. A fair share of recordings are from Europe.","metadata":{}},{"cell_type":"markdown","source":"#### When was the audio uploaded?","metadata":{}},{"cell_type":"code","source":"date_uploaded = metadata_df['date'].apply(lambda x: x.split('-')[0])\n\nplt.figure(figsize=(16, 6))\nax = sns.countplot(x=date_uploaded.values, palette=\"hls\")\n\nplt.title(\"Audio Files Registration per Year Made\", fontsize=16)\nplt.xticks(rotation=90, fontsize=13)\nplt.yticks(fontsize=13)\nplt.ylabel(\"Frequency\", fontsize=14)\nplt.xlabel(\"\");","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> 📌 Most of the recordings were uploaded in the year between 2012 and 2020. <br>\n> 📌 Also note that some of the years are wrong (ex: 2104, 0199, etc).","metadata":{}},{"cell_type":"markdown","source":"## `train_soundscape_labels.csv`","metadata":{}},{"cell_type":"code","source":"TRAIN_SOUNDSCAPE = '../input/birdclef-2021/train_soundscape_labels.csv'\ntrain_soundscape_df = pd.read_csv(TRAIN_SOUNDSCAPE)\ntrain_soundscape_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Reference: https://www.kaggle.com/stefankahl/birdclef2021-exploring-the-data\n> 📌 `row_id`: Unique identifier of a 5-second segment of each soundscape file. <br>\n> 📌 `site`: Recording site of the soundscape data. In this competition, 4 different sites (COL = Colombia, COR = Costa Rica, SNE = Sierra Nevada, SSW = Sapsucker Woods) are included. <br>\n> 📌 `audio_id`: Identifier used to reference audio recordings. Filenames contain the file ID, recording site and recording date (yyyymmdd). <br>\n> 📌 `seconds`: End time of the 5-second segment for which this entry states the label. <br>\n> 📌 `birds`: primary label (i.e., eBird code) of the audible species of this segment. “nocall” references a segment without any bird vocalization. Segments can have more than one bird, in that case, eBird codes are separated by space. “nocall” can never appear together with other codes.\n","metadata":{}},{"cell_type":"markdown","source":"#### Most commom birds vocalization found?","metadata":{}},{"cell_type":"code","source":"train_soundscape_df['birds'].value_counts()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> 📌 Many segments are associated with `nocall`. `rucwar`, `bobfly1`, etc are commonly found. \n> 📌 There are segments with two or more birds volcalization found. **We thus need to build a multi-label classifier**.","metadata":{}},{"cell_type":"markdown","source":"## `test.csv`","metadata":{}},{"cell_type":"code","source":"TEST = '../input/birdclef-2021/test.csv'\ntest_df = pd.read_csv(TEST)\ntest_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 🔈 2.  The Audio Files","metadata":{}},{"cell_type":"markdown","source":"**TL;DR for `train_short_audio`**\n\n**Usage**: To train fine-grained multi-class (label) audio classifier. <br>\n**Num species**: 397 <br>\n**Num training samples**: 62874 <br>\n**Class-imbalance**: Yes","metadata":{}},{"cell_type":"markdown","source":"#### Number of bird species","metadata":{}},{"cell_type":"code","source":"SHORT_TRAIN = '../input/birdclef-2021/train_short_audio/'\nprint(f'Number of unique bird species: {len(os.listdir(SHORT_TRAIN))}')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Number of training data","metadata":{}},{"cell_type":"code","source":"recordings_per_label = {'species_id': [], \n                        'num_audio': []}\n\nfor label in os.listdir(SHORT_TRAIN):\n    num_recordings = len(os.listdir(SHORT_TRAIN+label))\n    recordings_per_label['species_id'].append(label)\n    recordings_per_label['num_audio'].append(num_recordings)\n        \nrecordings_per_label = pd.DataFrame.from_dict(recordings_per_label)\n\nrun = wandb.init(project='birdclef', group='EDA')\ndata = [[label, val] for (val, label) in sorted(zip(recordings_per_label.num_audio.values, recordings_per_label.species_id.values))[::-1]]\ntable = wandb.Table(data=data, columns = [\"species_id\", \"num_audio\"])\nwandb.log({\"recordings_per_label\" : wandb.plot.bar(table, \"species_id\", \"num_audio\",\n                               title=\"Number of recordings per label\")})\nrun.finish()\n\n# display W&B run page\nrun","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"(☝️ [W&B Run Page](https://wandb.ai/ayush-thakur/birdclef/runs/1grwh85p?workspace=user-ayush-thakur))\n> ⚠️ Disclaimer: Since there are too many labels, the `specied_id` (y-axis) looks clumsy. <br>\n> 📌 Pro-tip 1: Scroll over the bar chart to look at the number of recordings per `species_id`. <br>\n> 📌 Pro-tip 2: Click on the ✏️ (Edit panel) icon in the chart to visualize the expanded version of the bar chart. <br>\n> 📌 Pro-tip 3: In the `recordings_per_label_table` you can sort the columns in ascending or descending order by clicking on the column name. \n\nThere's significant class imbalance. Species `crfpar` and `stvhum2` got only 8 audio files. While 12 labels got 500 audio files each. Play with the W&B dashboard above to get more insight.","metadata":{}},{"cell_type":"markdown","source":"#### Number of recordings per label","metadata":{}},{"cell_type":"code","source":"num_data = np.sum(recordings_per_label['num_audio'])\nprint(f'Number of training data: {num_data}')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Listen to Audio\n\nWe will use Weights and Biases to visualize audio waveforms and listen to bird sounds.","metadata":{}},{"cell_type":"code","source":"NUM_AUDIO_PER_LABEL = 8 # \naudio_per_label = {}\n\nfor label in os.listdir(SHORT_TRAIN):\n    # add new key (label) to dict\n    audio_per_label[label] = []\n    # get NUM_AUDIO_PER_LABEL audio filenames per label\n    audio_filenames = random.sample(os.listdir(SHORT_TRAIN+label), NUM_AUDIO_PER_LABEL)\n    # append path to that file\n    for audio_filename in audio_filenames:\n        audio_per_label[label].append(SHORT_TRAIN+label+'/'+audio_filename)\n        \n# We will use Weights and Biases to visualize audio waveforms and listen to bird sounds.\nrun = wandb.init(project='birdclef', group='EDA')\nfor label, audio_paths in audio_per_label.items():\n    audio_arr = []\n    sr_arr = []\n    for audio_path in audio_paths:\n        audio, sr = lb.load(audio_path)\n        audio_arr.append(audio)\n        sr_arr.append(sr)\n        \n    # log audio data for each label per step. \n    wandb.log({'audio-samples': [wandb.Audio(audio, caption=f'{label}', sample_rate=sr) \n                                             for audio, sr in zip(audio_arr, sr_arr)]})\n\nrun.finish()\n\n# display W&B run page\nrun","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"(☝️ [W&B Run Page](https://wandb.ai/ayush-thakur/birdclef/runs/2713uqst?workspace=user-ayush-thakur))\n\n> 📌 Note: The dashboard above have 8 samples of audio for each label. <br>\n> 📌 Pro-tip 1: Click on the ⚙️ in the `audio-samples` chart above. There will be total of 397 steps, where each step represents an unique label. <br>\n> 📌 Pro-tip 2: The audio will appear for the selected step (label). Each audio player's caption is the label name. \n\nYou can visualize the waveform of the audio beside listening to the music with just 3 lines of code. **I highly recommend spending time listening to the sample audio.******\n\n👀 Some quick observations:\n* The audio duration varies from few seconds to few minutes.\n* The audio is noisy with the sound of rain, insects, humans talking, wind blowing, etc. \n* In some audio the sound of bird is coming from far off distance. ","metadata":{}},{"cell_type":"markdown","source":"# 🏆 3. Audio Features & Preprocessing","metadata":{}},{"cell_type":"markdown","source":"## Audio Normalization","metadata":{}},{"cell_type":"code","source":"# A sample audio. Change species_id to select different label\nsample_audio = audio_per_label['caltow'][3]\n# Load unnormalized audio\naudio, sr = lb.load(sample_audio)\n# Normalize audio\nnorm_audio = librosa.util.normalize(audio)\n\nprint(f'The shape of audio: {audio.shape}; sampling rate: {sr}; audio duration: {audio.shape[0]/sr} s')\n\nrun = wandb.init(project='birdclef', group='EDA')\nwandb.log({'audio_sample': [wandb.Audio(audio, caption=f'Audio Sample', sample_rate=sr)]})\nwandb.log({'normalized_audio-sample': [wandb.Audio(norm_audio, caption=f'Normalized Audio Sample', sample_rate=sr)]})\nrun.finish()\n\nrun","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> 📌 Note that this constrained each signal between 0 and 1. Not sure if it would be a great idea to normalize the audio like this. Here's a short [reddit post](https://www.reddit.com/r/MachineLearning/comments/4ea0m7/audio_normalizationpreprocessing_before/). ","metadata":{}},{"cell_type":"markdown","source":"## Trim the noise","metadata":{}},{"cell_type":"code","source":"position = tfio.experimental.audio.trim(audio, axis=0, epsilon=0.1)\ntrimmed_audio = audio[position[0]:position[1]]\n\nrun = wandb.init(project='birdclef', group='EDA')\nwandb.log({'Trimmed': [wandb.Audio(trimmed_audio, caption=f'Trimmed', sample_rate=sr)]})\nrun.finish()\n\nrun","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> 📌 Note how the initial and final silent waveform is removed. It might be useful preprocessing step after getting 5 second clips.","metadata":{}},{"cell_type":"markdown","source":"#### Let's see the effect of trimming on 5 second clips of the audio. ","metadata":{}},{"cell_type":"code","source":"audio_clips = []\naudio_time = len(audio)//sr\nprint(f'The duration of audio is: {audio_time}')\n\nstart_sample = 0\nend_sample = sr*5 # sampling rate is number of samples per second. \n\nfor i in range(audio_time//5):\n    audio_clips.append(audio[start_sample:end_sample])\n    start_sample = end_sample\n    end_sample+=sr*5\n    \nrun = wandb.init(project='birdclef', group='EDA')\nfor i, audio_clip in enumerate(audio_clips):\n    # Trim audio\n    position = tfio.experimental.audio.trim(audio_clip, axis=0, epsilon=0.1)\n    trimmed_audio = audio_clip[position[0]:position[1]]\n    \n    # Log clipped and trimmed audio\n    audio_arr = [audio_clip, trimmed_audio]\n    captions = ['Clipped', 'Trimmed']\n    \n    wandb.log({f'clipped_vs_trimmed': [wandb.Audio(aud, caption=f'{caption}', sample_rate=sr)\n                                              for aud, caption in zip(audio_arr, captions)]})\n\nrun.finish()\n\nrun","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> 📌 Click on the ⚙️ in the clipped_vs_trimmed chart above. Each step is a unique pair of audio clip and it's trimmed counterpart. \n> 📌 Note: By trimming we are losing on the background sound but there's no control over what is lost. Thus this might not be a good strategy to be applied on 5 second clips.","metadata":{}},{"cell_type":"markdown","source":"## Fade In and Fade Out","metadata":{}},{"cell_type":"code","source":"fade = tfio.experimental.audio.fade(trimmed_audio, fade_in=100000, fade_out=200000, mode=\"exponential\")\n\nfig, ax = plt.subplots(2, figsize = (20, 8), dpi=120)\nfig.suptitle('Original Vs Trimmed', fontsize=16)\nlb.display.waveplot(trimmed_audio, sr=sr, ax=ax[0])\nlb.display.waveplot(fade.numpy(), sr=sr, ax=ax[1])\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> 📌 [](http://)This Wikipedia page might be a good read. A recorded audio may be gradually reduced to silence at its end (fade-out), or may gradually increase from silence at the beginning (fade-in). Again I am not sure if it's going to be useful.","metadata":{}},{"cell_type":"markdown","source":"## Fast Fourier Transform","metadata":{}},{"cell_type":"code","source":"# compute fft\nfft = np.fft.fft(audio)\n# compute frequency \nfreq = np.fft.fftfreq(audio.size, 1/sr)\nfreq = freq[:len(freq)//2]\nprint(f'Max frequency in the audio: {freq[-1]}. Obviously it is going to be half of sampling rate')\n# get the magnitude\nmag_fft = abs(fft)\n# Remember 2nd half of the fft is repeated.\nmag_fft = mag_fft[:len(mag_fft)//2]\n\nfig, ax = plt.subplots(nrows=2, ncols=1, figsize=(20, 8), dpi=120);\nfig.suptitle('Fast Fourier Transform')\nax[0].plot(audio)\nax[0].set_xlabel('Time', fontsize=16);\nax[0].set_ylabel('Amplitude', fontsize=16);\n\nax[1].plot(freq, mag_fft)\nax[1].set_xlabel('Freq(Hz)', fontsize=16);\nax[1].set_ylabel('Power', fontsize=16);","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> 📌 Note that while computing FFT the relevant information is in the first half of the array. The other half is simply repeated in inverse indices order. <br>\n> 📌 There are some major regions of frequency concentration that too in high frequency region. <br>\n> 📌 Note that there is numerical difference while computing FFT using TensorFlow and Numpy. For training a neura network it might not be an issue.","metadata":{}},{"cell_type":"markdown","source":"## Spectrogram\n\nA spectrogram is a visual representation of the spectrum of frequencies of a signal as it varies with time. ","metadata":{}},{"cell_type":"markdown","source":"### Linear Power Spectrogram","metadata":{}},{"cell_type":"code","source":"# Parameters\nn_fft = 2048\nhop_length = 512","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Short-time Fourier transform (STFT)\nS = np.abs(lb.stft(audio, n_fft=n_fft, hop_length=hop_length))\n# Convert an amplitude spectrogram to Decibels-scaled spectrogram.\nS_to_DB = librosa.amplitude_to_db(S, ref=np.max)\n\n# Plot spectrogram\nplt.figure(figsize=(16,4))\nlb.display.specshow(S_to_DB, y_axis='linear', sr=sr, hop_length=hop_length,\n                   x_axis='time') # Note linear y axis\nplt.title('Linear-Frequency Power Spectrogram');\nplt.colorbar();","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> 📌 `n_fft=2048` samples, corresponds to a physical duration of 93 milliseconds at a sample rate of 22050 Hz.","metadata":{}},{"cell_type":"markdown","source":"### Log Power Spectrogram","metadata":{}},{"cell_type":"code","source":"# Short-time Fourier transform (STFT)\nS = np.abs(lb.stft(audio, n_fft=n_fft, hop_length=hop_length))\n# Convert an amplitude spectrogram to Decibels-scaled spectrogram.\nS_to_DB = librosa.amplitude_to_db(S, ref=np.max)\n\n# Plot spectrogram \nplt.figure(figsize=(16,4))\nlb.display.specshow(S_to_DB, y_axis='log', sr=sr, hop_length=hop_length,\n                   x_axis='time') # Note logarithmic yaxis\nplt.title('Log-Frequency Power Spectrogram');\nplt.colorbar();","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Constant Q (Note) Power Spectrogram","metadata":{}},{"cell_type":"code","source":"# Short-time Fourier transform (STFT)\nCQT_note = np.abs(lb.cqt(audio, sr=sr, hop_length=hop_length))\n# Convert an amplitude spectrogram to Decibels-scaled spectrogram.\nCQT_note = librosa.amplitude_to_db(CQT_note, ref=np.max)\n\n# Plot spectrogram\nplt.figure(figsize=(16,4))\nlb.display.specshow(CQT_note, y_axis='cqt_note', sr=sr, hop_length=hop_length,\n                   x_axis='time') # Note cqt_note y axis\nplt.title('Constant-Q (Note) Power Spectrogram');\nplt.colorbar();","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Constant Q (Hz) Power Spectrogram","metadata":{}},{"cell_type":"code","source":"# Short-time Fourier transform (STFT)\nCQT_note = np.abs(lb.cqt(audio, sr=sr, hop_length=hop_length))\n# Convert an amplitude spectrogram to Decibels-scaled spectrogram.\nCQT_note = librosa.amplitude_to_db(CQT_note, ref=np.max)\n\n# Plot spectrogram \nplt.figure(figsize=(16,4))\nlb.display.specshow(CQT_note, y_axis='cqt_hz', sr=sr, hop_length=hop_length,\n                   x_axis='time') # Note cqt_hz y axis\nplt.title('Constant-Q (Hz) Power Spectrogram');\nplt.colorbar();","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Tempogram with BPM markers","metadata":{}},{"cell_type":"code","source":"Tgram = lb.feature.tempogram(y=audio, sr=sr)\n\n# Plot spectrogram\nplt.figure(figsize=(16,4))\nlb.display.specshow(Tgram, y_axis='tempo', sr=sr, hop_length=hop_length,\n                   x_axis='time') # Note cqt_hz y axis\nplt.title('Tempogram with BPM Markers');\nplt.colorbar();","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Chromagram with Pitch Classes","metadata":{}},{"cell_type":"code","source":"C = librosa.feature.chroma_cqt(y=audio, sr=sr)\n\n# Plot spectrogram\nplt.figure(figsize=(16,4))\nlb.display.specshow(C, y_axis='chroma', sr=sr, hop_length=hop_length,\n                   x_axis='time') # Note cqt_hz y axis\nplt.title('Chromagram with Pitch Class');\nplt.colorbar();","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Let's look at the spectrogram for every class using W&B.\n\n**We will visualize linear-frequency power spectrogram for audio with rating 4 and more.**","metadata":{}},{"cell_type":"code","source":"def get_path(label, filename):\n    return '../input/birdclef-2021/train_short_audio/'+f'{label}/{filename}'\n\n# Dataframe audio filenames with rating 4 or more.\nmetadata_tmp_df = metadata_df.loc[metadata_df['rating'] > 4]\nmetadata_tmp_df.loc[:, 'kaggle_path'] = metadata_tmp_df.apply(lambda row: get_path(row['primary_label'],\n                                                                            row['filename']), axis=1)\n\n# Get random samples for each label.\naudio_per_label = {}\n\nfor label in os.listdir(SHORT_TRAIN):\n    tmp = metadata_tmp_df.loc[metadata_tmp_df['primary_label'] == label]\n    tmp = tmp.sample(n=1, axis=0, replace=True)\n    audio_per_label[label] = tmp.kaggle_path.values[0]","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# We will use Weights and Biases to visualize different spectrograms.\nrun = wandb.init(project='birdclef', group='EDA')\nc = 0\nfor label, audio_path in audio_per_label.items():\n    # Image name\n    img_name = audio_path.split('/')[-1].split('.')[0]\n\n    # Load audio\n    audio, sr = lb.load(audio_path)\n\n    # Compute spectrogram\n    # Short-time Fourier transform (STFT)\n    S = np.abs(lb.stft(audio, n_fft=n_fft, hop_length=hop_length))\n    # Convert an amplitude spectrogram to Decibels-scaled spectrogram.\n    S_to_DB = librosa.amplitude_to_db(S, ref=np.max)\n\n    # Save as spectrogram\n    fig = plt.figure(figsize=(16,4))\n    lb.display.specshow(S_to_DB, y_axis='linear', sr=sr, hop_length=hop_length,\n                       x_axis='time') # Note linear y axis\n    plt.xticks([]); plt.yticks([]); plt.xlabel(''); plt.ylabel('');\n    plt.savefig(f'{img_name}.png');\n    plt.close(fig)\n\n    # log audio data for each label per step. \n    wandb.log({\"linear-power-spectrogram\": [wandb.Image(f'{img_name}.png', caption=f'{label}')]})\n    \n    c+=1\n    if c==20:\n        break\n\nrun.finish()\n\n# display W&B run page\nrun","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":">  📌 Click on the ⚙️ in the linear-power-spectrogram chart above to visualize the spectrogram per label. Note that I have only logged spectrogram for 20 labels.","metadata":{}},{"cell_type":"markdown","source":"#### Spectral Centroid\n\nThe spectral centroid indicates at which frequency the energy of a spectrum is centered upon or in other words It indicates where the ” center of mass” for a sound is located.","metadata":{}},{"cell_type":"code","source":"sample_audio = audio_per_label['caltow']\naudio, sr = lb.load(sample_audio)\n\nS = np.abs(lb.stft(audio, n_fft=n_fft, hop_length=hop_length))\nS_to_DB = librosa.amplitude_to_db(S, ref=np.max)\n\ncent = lb.feature.spectral_centroid(S=S)\ntimes = lb.times_like(cent)\n\nfig, ax = plt.subplots(figsize=(16,4))\nlibrosa.display.specshow(S_to_DB, y_axis='linear', x_axis='time', ax=ax)\nax.plot(times, cent.T, label='Spectral centroid', color='w')\nax.legend(loc='upper right');\nax.set(title='Log Power spectrogram');","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# WORK IN PROGRESS :D","metadata":{}}]}