","metadata":{}},{"cell_type":"markdown","source":"**Weights & Biases** is the machine learning platform for developers to build better models faster.\n\nYou can use W&B's lightweight, interoperable tools to\n\n- quickly track experiments,\n- version and iterate on datasets,\n- evaluate model performance,\n- reproduce models,\n- visualize results and spot regressions,\n- and share findings with colleagues.\n \nSet up W&B in 5 minutes, then quickly iterate on your machine learning pipeline with the confidence that your datasets and models are tracked and versioned in a reliable system of record.\n\nIn this notebook I will use Weights and Biases's amazing features to perform wonderful visualizations and logging seamlessly.","metadata":{}},{"cell_type":"code","source":"# Initialise the Run\n\nrun = wandb.init(project=config['competition'], job_type='Visualization', name='BirdCLEF 2023 Audio')","metadata":{"execution":{"iopub.status.busy":"2023-03-07T21:42:49.838228Z","iopub.execute_input":"2023-03-07T21:42:49.839481Z","iopub.status.idle":"2023-03-07T21:43:33.580385Z","shell.execute_reply.started":"2023-03-07T21:42:49.839434Z","shell.execute_reply":"2023-03-07T21:43:33.576203Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Here is a minimal example on how you can log the audio to your wandb dashboard \n\n# Initialise a table with the column names\ntable = wandb.Table(columns=['Audio Sample', 'Primary Label'])\n\n# For simplicity I have selected 100 rows only, you can have the entire dataframe\nminimal_df = df.sample(100).reset_index(drop=True)\n\n# Log the data to table\nfor i in tqdm(range(len(minimal_df))):\n row = minimal_df.loc[i]\n audio = wandb.Audio(row.full_path, sample_rate=config['sample_rate'])\n table.add_data(audio, row.primary_label)\n\nwandb.log({'BirdCLEF 2023 Audio': table})\n\n# Finish the run\nrun.finish()","metadata":{"execution":{"iopub.status.busy":"2023-03-07T21:47:14.385751Z","iopub.execute_input":"2023-03-07T21:47:14.386164Z","iopub.status.idle":"2023-03-07T21:47:27.924746Z","shell.execute_reply.started":"2023-03-07T21:47:14.386129Z","shell.execute_reply":"2023-03-07T21:47:27.923451Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### [Interactive W&B Table for Audio $\\rightarrow$](https://wandb.ai/ishandutta/BirdCLEF2023/runs/4rvhdvlv?workspace=user-ishandutta)\n\n
\n
Basic Analysis
","metadata":{}},{"cell_type":"markdown","source":"## **
Get Audio Samples** ","metadata":{}},{"cell_type":"code","source":"df.primary_label.nunique()","metadata":{"execution":{"iopub.status.busy":"2023-03-10T18:52:41.830671Z","iopub.execute_input":"2023-03-10T18:52:41.831028Z","iopub.status.idle":"2023-03-10T18:52:41.846288Z","shell.execute_reply.started":"2023-03-10T18:52:41.830996Z","shell.execute_reply":"2023-03-10T18:52:41.844637Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# df.primary_label.unique()","metadata":{"execution":{"iopub.status.busy":"2023-03-10T18:52:43.043262Z","iopub.execute_input":"2023-03-10T18:52:43.043630Z","iopub.status.idle":"2023-03-10T18:52:43.050595Z","shell.execute_reply.started":"2023-03-10T18:52:43.043596Z","shell.execute_reply":"2023-03-10T18:52:43.048721Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_audio_sample(df: pd.DataFrame, bird_label: str):\n \"\"\"\n Function to get the audio sample for a bird using it's label\n \n Args:\n df (pandas.DataFrame): DataFrame with the metadata\n birl_label (str): Label of the bird for which audio sample is required\n \"\"\"\n AUDIO_DIR_PATH = \"/kaggle/input/birdclef-2023/train_audio\"\n\n df['full_path'] = AUDIO_DIR_PATH + '/' + df['filename']\n \n return df[df['primary_label'] == bird_label].sample(1, random_state = 42)['full_path'].values[0]","metadata":{"execution":{"iopub.status.busy":"2023-03-10T18:52:43.382398Z","iopub.execute_input":"2023-03-10T18:52:43.382779Z","iopub.status.idle":"2023-03-10T18:52:43.390488Z","shell.execute_reply.started":"2023-03-10T18:52:43.382741Z","shell.execute_reply":"2023-03-10T18:52:43.389058Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"abethr1_file = get_audio_sample(df, 'abethr1')\nblcapa2_file = get_audio_sample(df, 'blcapa2')\nchibat1_file = get_audio_sample(df, 'chibat1')\ndotbar1_file = get_audio_sample(df, 'dotbar1')","metadata":{"execution":{"iopub.status.busy":"2023-03-10T18:52:46.332228Z","iopub.execute_input":"2023-03-10T18:52:46.332606Z","iopub.status.idle":"2023-03-10T18:52:46.363415Z","shell.execute_reply.started":"2023-03-10T18:52:46.332572Z","shell.execute_reply":"2023-03-10T18:52:46.361852Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ipd.Audio(abethr1_file)","metadata":{"execution":{"iopub.status.busy":"2023-03-10T18:52:46.618405Z","iopub.execute_input":"2023-03-10T18:52:46.618759Z","iopub.status.idle":"2023-03-10T18:52:46.643272Z","shell.execute_reply.started":"2023-03-10T18:52:46.618728Z","shell.execute_reply":"2023-03-10T18:52:46.642266Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ipd.Audio(blcapa2_file)","metadata":{"execution":{"iopub.status.busy":"2023-03-10T18:52:46.960457Z","iopub.execute_input":"2023-03-10T18:52:46.960841Z","iopub.status.idle":"2023-03-10T18:52:46.976059Z","shell.execute_reply.started":"2023-03-10T18:52:46.960806Z","shell.execute_reply":"2023-03-10T18:52:46.974595Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ipd.Audio(chibat1_file)","metadata":{"execution":{"iopub.status.busy":"2023-03-10T18:52:47.289019Z","iopub.execute_input":"2023-03-10T18:52:47.289632Z","iopub.status.idle":"2023-03-10T18:52:47.333768Z","shell.execute_reply.started":"2023-03-10T18:52:47.289595Z","shell.execute_reply":"2023-03-10T18:52:47.332758Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ipd.Audio(dotbar1_file)","metadata":{"execution":{"iopub.status.busy":"2023-03-10T18:52:47.609495Z","iopub.execute_input":"2023-03-10T18:52:47.610456Z","iopub.status.idle":"2023-03-10T18:52:47.649599Z","shell.execute_reply.started":"2023-03-10T18:52:47.610410Z","shell.execute_reply":"2023-03-10T18:52:47.648609Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"> ### **
This notebook is inspired by the [Audio Signal Processing For Machine Learning Series](https://www.youtube.com/playlist?list=PL-wATfeyAMNqIee7cH3q1bh4QJFAaeNv0) by Valerio Velardo. ** ","metadata":{}},{"cell_type":"markdown","source":"## **
Load Audio Signal** ","metadata":{}},{"cell_type":"code","source":"# Using librosa we will load these audio files\n# These files are downsampled at 32Khz, which we pass\n# to the sr argument of librosa.load or we can pass None\n\nabethr1, sr = librosa.load(abethr1_file, sr=None)\nblcapa2, sr = librosa.load(blcapa2_file, sr=None)\nchibat1, sr = librosa.load(chibat1_file, sr=None)\ndotbar1, sr = librosa.load(dotbar1_file, sr=None)","metadata":{"execution":{"iopub.status.busy":"2023-03-10T18:52:52.432033Z","iopub.execute_input":"2023-03-10T18:52:52.432513Z","iopub.status.idle":"2023-03-10T18:53:01.482181Z","shell.execute_reply.started":"2023-03-10T18:52:52.432471Z","shell.execute_reply":"2023-03-10T18:53:01.480520Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"abethr1_file","metadata":{"execution":{"iopub.status.busy":"2023-03-10T18:53:01.484890Z","iopub.execute_input":"2023-03-10T18:53:01.485702Z","iopub.status.idle":"2023-03-10T18:53:01.492441Z","shell.execute_reply.started":"2023-03-10T18:53:01.485664Z","shell.execute_reply":"2023-03-10T18:53:01.491619Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"abethr1.size","metadata":{"execution":{"iopub.status.busy":"2023-03-10T18:53:01.493782Z","iopub.execute_input":"2023-03-10T18:53:01.494327Z","iopub.status.idle":"2023-03-10T18:53:01.508797Z","shell.execute_reply.started":"2023-03-10T18:53:01.494284Z","shell.execute_reply":"2023-03-10T18:53:01.507038Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Tip: If you want to load 5 seconds of a file \n# which starts from 10 seconds use offset and duration parameters\n\nabethr1_5, sr = librosa.load(abethr1_file, sr=None, offset=15.0, duration=5.0)\nabethr1_5.size","metadata":{"execution":{"iopub.status.busy":"2023-03-10T18:53:01.511266Z","iopub.execute_input":"2023-03-10T18:53:01.511793Z","iopub.status.idle":"2023-03-10T18:53:01.537596Z","shell.execute_reply.started":"2023-03-10T18:53:01.511741Z","shell.execute_reply":"2023-03-10T18:53:01.535897Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# For now let us focus on abethr1\nprint(abethr1)","metadata":{"execution":{"iopub.status.busy":"2023-03-10T18:53:01.538987Z","iopub.execute_input":"2023-03-10T18:53:01.539470Z","iopub.status.idle":"2023-03-10T18:53:01.547260Z","shell.execute_reply.started":"2023-03-10T18:53:01.539424Z","shell.execute_reply":"2023-03-10T18:53:01.545426Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Every element of this array is actually a sample of the audio\n# The value corresponding to that audio sample is the amplitude w.r.t that sample\n\nprint(f\"abethr1 has a total of {len(abethr1)} samples\")","metadata":{"execution":{"iopub.status.busy":"2023-03-10T18:53:01.549253Z","iopub.execute_input":"2023-03-10T18:53:01.549852Z","iopub.status.idle":"2023-03-10T18:53:01.560803Z","shell.execute_reply.started":"2023-03-10T18:53:01.549811Z","shell.execute_reply":"2023-03-10T18:53:01.559115Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **
Calculate Audio Duration** ","metadata":{}},{"cell_type":"code","source":"# To find the duration of 1 sample, we take inverse of sampling rate\nsample_duration = 1/sr\nprint(f\"Duration of 1 sample of audio: {sample_duration:.6f} seconds\")","metadata":{"execution":{"iopub.status.busy":"2023-03-10T18:53:01.562551Z","iopub.execute_input":"2023-03-10T18:53:01.562949Z","iopub.status.idle":"2023-03-10T18:53:01.573292Z","shell.execute_reply.started":"2023-03-10T18:53:01.562905Z","shell.execute_reply":"2023-03-10T18:53:01.571654Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Now via ipd.Audio we already saw that the duration of abethr1 is 19 secs\n# We can also calculate it as\n\nduration = sample_duration * len(abethr1)\nprint(f\"Duration of Audio Signal: {duration} seconds\")","metadata":{"execution":{"iopub.status.busy":"2023-03-10T18:53:01.575110Z","iopub.execute_input":"2023-03-10T18:53:01.575515Z","iopub.status.idle":"2023-03-10T18:53:01.587226Z","shell.execute_reply.started":"2023-03-10T18:53:01.575478Z","shell.execute_reply":"2023-03-10T18:53:01.585442Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **
Waveforms** ","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(17, 15))\n\nbirds = {\n 'abethr1': abethr1, \n 'blcapa2': blcapa2, \n 'chibat1': chibat1, \n 'dotbar1': dotbar1}\n\nfor i, (bird_name, bird_arr) in enumerate(birds.items()):\n plt.subplot(2, 2, i+1)\n # Librosa has an inbuilt function to display the plot directly\n librosa.display.waveshow(bird_arr)\n plt.title(str(bird_name))\n plt.ylim((-1,1))","metadata":{"execution":{"iopub.status.busy":"2023-03-10T18:53:01.589044Z","iopub.execute_input":"2023-03-10T18:53:01.589462Z","iopub.status.idle":"2023-03-10T18:53:03.653482Z","shell.execute_reply.started":"2023-03-10T18:53:01.589432Z","shell.execute_reply":"2023-03-10T18:53:03.652333Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As an exercise, listen to the audio signals again and focus on the portions where the waveforms show spikes. \nAre you able to listen the spikes in the audio?\n \nAnother interesting thing to note is that the `blcapa2` audio is a lot more soothing and has observable patterns than the `dotbar1`. Can we say that the audio for `blcapa2` is a song and that for `dotbar1` is not?\n\nLet us find out!","metadata":{}},{"cell_type":"code","source":"def get_audio_type(df: pd.DataFrame, full_audio_path: str):\n \"\"\"\n Function to get the audio type from the audio path\n \n Args:\n df (pd.DataFrame): Metadata for the audio signals\n full_audio_path (str): File path for the audio signal\n \n Returns:\n type of audio signal\n \"\"\"\n \n return df[df['full_path'] == full_audio_path]['type'].values[0]","metadata":{"execution":{"iopub.status.busy":"2023-03-10T18:53:03.656651Z","iopub.execute_input":"2023-03-10T18:53:03.657490Z","iopub.status.idle":"2023-03-10T18:53:03.663452Z","shell.execute_reply.started":"2023-03-10T18:53:03.657445Z","shell.execute_reply":"2023-03-10T18:53:03.661803Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"Type of Audio for blcapa2: {get_audio_type(df, blcapa2_file)}\")\nprint(f\"Type of Audio for dotbar1: {get_audio_type(df, dotbar1_file)}\")","metadata":{"execution":{"iopub.status.busy":"2023-03-10T18:53:03.664867Z","iopub.execute_input":"2023-03-10T18:53:03.665246Z","iopub.status.idle":"2023-03-10T18:53:03.686233Z","shell.execute_reply.started":"2023-03-10T18:53:03.665212Z","shell.execute_reply":"2023-03-10T18:53:03.684321Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Thus our thought process was correct!","metadata":{}},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"
\n
\n
Time Domain Features
","metadata":{}},{"cell_type":"markdown","source":"On a high level this includes three main features:\n1. Amplitude Envelope (AE)\n2. Root Mean Square Energy (RMS)\n3. Zero Crossing Rate (ZCR)","metadata":{}},{"cell_type":"markdown","source":"## **
Amplitude Envelope (AE)** ","metadata":{}},{"cell_type":"markdown","source":"Amplitude Envelope gives us the envelope (boundaries) of the sound. What is the boundary for a single frame? It is it's maximum amplitude.\n \nHence, the aim of amplitude envelope is to get the maximum amplitude for each frame. But why is this helpful?\n \nIntuitively, the amplitude of an audio signal is indicative of how loud the audio is or simply what is it's volume. \n \nTo obtain the AE, we split the audio signal into multiple windows, each having the same size. Then for each of the windows we find the maximum amplitude amongst the constituent frames in it. \n\nAn interesting application of AE is for onset detection, or the detection of the beginning of a sound. ","metadata":{}},{"cell_type":"code","source":"def get_amplitude_envelope(audio_signal: np.array, window_size: int = 1024, hop_size: int = 512):\n \"\"\"\n Function to obtain the amplitude envelope of an audui signal\n \n Args:\n audio_signal (np.array): The original audio signal\n window_size (int): The size of the window w.r.t which we find the max amplitude\n hop_size (int): Hop Lentgth for the windows\n \n Returns:\n amplitude_envelope (np.array): Amplitude Envelope Array\n \"\"\"\n \n amplitude_envelope = []\n\n for window in range(0, len(audio_signal), hop_size):\n local_window = audio_signal[window: window + window_size]\n max_amplitude = max(local_window)\n amplitude_envelope.append(max_amplitude)\n \n return np.array(amplitude_envelope)","metadata":{"execution":{"iopub.status.busy":"2023-03-10T19:14:57.672542Z","iopub.execute_input":"2023-03-10T19:14:57.673501Z","iopub.status.idle":"2023-03-10T19:14:57.680287Z","shell.execute_reply.started":"2023-03-10T19:14:57.673459Z","shell.execute_reply":"2023-03-10T19:14:57.678534Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ae_abethr1 = get_amplitude_envelope(audio_signal=abethr1, window_size=1024, hop_size=512)","metadata":{"execution":{"iopub.status.busy":"2023-03-10T19:14:58.635169Z","iopub.execute_input":"2023-03-10T19:14:58.635524Z","iopub.status.idle":"2023-03-10T19:14:58.704630Z","shell.execute_reply.started":"2023-03-10T19:14:58.635492Z","shell.execute_reply":"2023-03-10T19:14:58.703401Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **
Root Mean Square Energy (RMS)** ","metadata":{}},{"cell_type":"markdown","source":"One key problem with the Amplitude Envelope is that it is very sensitive to outliers. To tackle this we have another time domain feature called the Root Mean Squared Energy. It is simply the Root Mean Squared of all the samples in a frame. \n \nThis is an indicator of loudness as well but is much less sensitive to outliers as compared to Amplitude Envelope.","metadata":{}},{"cell_type":"markdown","source":"## **
Zero Crossing Rate (ZCR)** ","metadata":{}},{"cell_type":"markdown","source":"The horizontal axis in the audio signal indicates zero amplitude. Hence Zero Crossing Rate means how many times is the signal crossing the horizontal line. This is generally a good comparison between percussive sounds and pitched sounds. Percussive sounds have higher ZCR as compared to pitched sounds which are relatively more stable. ","metadata":{}},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"
More Plots coming soon!
\n \n
![](\"https://static.wixstatic.com/media/5f8fae_7581e21a24a1483085024f88b0949a9d~mv2.jpg/v1/fill/w_934,h_379,al_c,q_90/5f8fae_7581e21a24a1483085024f88b0949a9d~mv2.jpg\")
","metadata":{}},{"cell_type":"markdown","source":"
\n
If you find this notebook useful, do give me an upvote, it helps to keep up my motivation. This notebook will be updated frequently so keep checking for furthur developments.
","metadata":{}}]}![](\"https://i.natgeofe.com/n/a189dd67-bc78-4716-aa29-cdbaceb5e4d0/photo-ark-parrots-endangered-bird-world-intelligence-3_3x2.jpg\")
![](\"https://i.imgur.com/1sm6x8P.png\")
![](\"https://i.ibb.co/PzCngK6/Screenshot-2.png\")