![](\"https://upload.wikimedia.org/wikipedia/commons/thumb/2/20/Circle-icons-arrow-up.svg/1200px-Circle-icons-arrow-up.svg.png\"){kind=icon}
Back to the table of contents![](\"https://academy.allaboutbirds.org/wp-content/uploads/ARTICLE-SONG-1440X8003.png\")
","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Overview","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"\nIn this notebook, I've tried to share what I learned so far in audio analysis after spending a few weeks with bird audio data.\n\nNotebook highlights:\nBack to the table of contents","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![](\"https://i.ytimg.com/vi/Z7YM-HAz-IY/hqdefault.jpg?sqp=-oaymwEXCNACELwBSFryq4qpAwkIARUAAIhCGAE=&rs=AOn4CLCKV3O45uiA3h90AD0SbZ3UHVbpMQ\")
\n Great tutorials by Seth Adams that covers Audio classification with live coding using python and freesound-audio-tagging Dataset\n
\n![](\"https://i.ytimg.com/vi/fMqL5vckiU0/hqdefault.jpg?sqp=-oaymwEXCNACELwBSFryq4qpAwkIARUAAIhCGAE=&rs=AOn4CLC-PAj2nDdxSsUw7_OJHuig4H3CVg\")
\n In this series, I explore theory and implementation of deep learning in the Python programming language. The course focuses on applications of deep learning for audio and music, but discusses general algorithms and principles applicable to any problem. I use TensorFlow.\n
\n![](\"https://haythamfayek.com/assets/posts/post1/hamming_window.jpg\")
\n Speech processing plays an important role in any speech system whether its Automatic Speech Recognition (ASR) or speaker recognition or something else. Mel-Frequency Cepstral Coefficients (MFCCs) were very popular features for a long time; but more recently, filter banks are becoming increasingly popular. In this post, I will discuss filter banks and MFCCs and why are filter banks becoming increasingly popular\n
\nBack to the table of contents","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"\n* [Bird Song Classification using an EfficientNet](https://www.kaggle.com/frlemarchand/bird-song-classification-using-an-efficientnet)\n\n* [NLP? Audio](https://www.kaggle.com/eladwar/digitize-audio) \n\nFor some of the code \n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Imports \nBack to the table of contents","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"!cd /kaggle/input/audio2numpy\n!mkdir audio\n!cp -r /kaggle/input/audio2numpy/* ./audio/\nfrom audio.audio2numpy import open_audio\n!pip install python_speech_features 2>/dev/null 1>/dev/null","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":false},"cell_type":"code","source":"import os\nimport gc\nfrom tqdm import tqdm_notebook\nimport warnings\nwarnings.filterwarnings(\"ignore\")\nimport random\n\n\nimport numpy as np\nimport pandas as pd\n\nfrom PIL import Image\nimport wave\nfrom scipy.io import wavfile\nimport librosa\nfrom librosa.feature import melspectrogram\n\n\nimport tensorflow.keras.layers as L\nfrom tensorflow.keras.models import Model\nimport tensorflow as tf \nfrom sklearn.metrics import accuracy_score\nfrom joblib import Parallel, delayed\nfrom sklearn.preprocessing import LabelEncoder\n\nfrom sklearn.utils import shuffle\nfrom sklearn.utils import class_weight\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nsns.set()\n\nfrom python_speech_features import mfcc,logfbank","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data preparation \nBack to the table of contents","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"in this section we gonna try to take the birds with the most clear audio sample ","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":" \ntrain = pd.read_csv(\"/kaggle/input/birdsong-recognition/train.csv\")\ntrain = train.query(\"rating>=4\")\n\nbirds_count = {}\nfor bird_species, count in zip(train.ebird_code.unique(), train.groupby(\"ebird_code\")[\"ebird_code\"].count().values):\n birds_count[bird_species] = count\nmost_represented_birds = [key for key,value in birds_count.items() if value == 100]\n\ntrain_nocall = train.loc[~train.ebird_code.isin(most_represented_birds)] \n\ntrain = train.loc[train.ebird_code.isin(most_represented_birds)]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Take 1000 random samples from the rest of the birds ","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"train[\"bird_name\"] = train.ebird_code\ntrain_nocall[\"bird_name\"] = train_nocall.ebird_code\ntrain_nocall = train_nocall.sample(1000)\ntrain_nocall.ebird_code = \"nocall\"\ntrain = pd.concat([train,train_nocall])\ntrain.reset_index(inplace=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Calculating and ploting FFT, FBANK, and MFCC... \nBack to the table of contents","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"All you need to know about Speech Processing FFT FBANK MFCC ... \n1. [Mel Frequency Cepstral Coefficient (MFCC) tutorial](http://practicalcryptography.com/miscellaneous/machine-learning/guide-mel-frequency-cepstral-coefficients-mfccs/)\n2. [Speech Processing for Machine Learning: Filter banks, Mel-Frequency Cepstral Coefficients (MFCCs) and What's In-Between](https://haythamfayek.com/2016/04/21/speech-processing-for-machine-learning.html)","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"def c_fft(y,r):\n n=len(y)\n f=np.fft.rfftfreq(n,1/r)\n f2 = abs(np.fft.rfft(y)/n)\n return (f2,f)\n\n\ndef envelope(y,rate,th):\n mask = []\n \n y = pd.Series(y).apply(np.abs)\n \n y_mean = y.rolling(window = int(rate/10),min_periods = 1, center = True).mean()\n for m in y_mean:\n \n if m > th :\n mask.append(True)\n else :\n mask.append(False)\n \n return mask\nsig = {}\nfft = {}\nfbank = {}\nmfcc_s = {}\n\nfor i in most_represented_birds:\n w_fn = train.loc[train.ebird_code == i,\"filename\"].values\n s,r = librosa.load(\"/kaggle/input/birdsong-recognition/train_audio/\"+i+\"/\"+w_fn[0],sr=44100)\n mask=envelope(s,r,0.005)\n s = s[mask]\n sig[i] = s\n fft[i] = c_fft(s,r)\n fbank[i] = logfbank(s[:r],r,nfilt = 26,nfft=1103).T\n mfcc_s[i] = mfcc(s[:r],r,numcep=13,nfilt=26,nfft=1103).T\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def plot_sig(sig):\n fig,axes = plt.subplots(nrows=10,ncols=5,sharex=False,sharey=True,figsize=(20,18))\n i=0\n for x in range(10):\n for y in range(5):\n if x == 9 and y == 4:\n pass\n else :\n axes[x,y].set_title(list(sig.keys())[i])\n axes[x,y].plot(list(sig.values())[i])\n axes[x,y].get_xaxis().set_visible(False)\n axes[x,y].get_yaxis().set_visible(False)\n i += 1 \n \nplot_sig(sig)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# FFT plot \nBack to the table of contents","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"def plot_fft(fft):\n fig,axes = plt.subplots(nrows=10,ncols=5,sharex=False,sharey=True,figsize=(20,18))\n i=0\n for x in range(10):\n for y in range(5):\n if x == 9 and y == 4:\n pass\n else :\n data = list(fft.values())[i]\n Y ,f = data[0], data[1]\n axes[x,y].set_title(list(fft.keys())[i])\n axes[x,y].plot(f,Y)\n axes[x,y].get_xaxis().set_visible(False)\n axes[x,y].get_yaxis().set_visible(False)\n i += 1 \n\nplot_fft(fft)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Fbank Plot \nBack to the table of contents","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"def plot_fbank(fbank):\n fig,axes = plt.subplots(nrows=10,ncols=5,sharex=False,sharey=True,figsize=(20,18))\n i=0\n for x in range(10):\n for y in range(5):\n if x == 9 and y == 4:\n pass\n else :\n axes[x,y].set_title(list(fbank.keys())[i])\n axes[x,y].imshow(list(fbank.values())[i],cmap=\"hot\",interpolation=\"nearest\")\n axes[x,y].get_xaxis().set_visible(False)\n axes[x,y].get_yaxis().set_visible(False)\n i += 1 \n\nplot_fbank(fbank)\nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# MFCC Plot \nBack to the table of contents","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"def plot_mfcc_s(mfcc):\n fig,axes = plt.subplots(nrows=10,ncols=5,sharex=False,sharey=True,figsize=(20,18))\n i=0\n for x in range(10):\n for y in range(5):\n if x == 9 and y == 4:\n pass\n else :\n axes[x,y].set_title(list(fbank.keys())[i])\n axes[x,y].imshow(list(fbank.values())[i],cmap=\"hot\",interpolation=\"nearest\")\n axes[x,y].get_xaxis().set_visible(False)\n axes[x,y].get_yaxis().set_visible(False)\n i += 1 \n\nplot_mfcc_s(mfcc_s)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Modeling \nBack to the table of contents","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"we gonna treat the audio like text Data by converting audio signals to numpy arrays ","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"NUM_BINS = 1000\nMAX_LEN = 10000","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"code","source":"def audio_read(path):\n recording, sr = open_audio(path)\n \n if recording.shape[0] != recording.size:\n return recording.mean(axis=1) \n else:\n return recording","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def tokenize(path, NUM_BINS = NUM_BINS,MAX_LEN=MAX_LEN):\n signal = np.resize(audio_read(path), (MAX_LEN,))\n signal_bins = np.linspace(signal.min(), signal.max(), NUM_BINS + 1)\n signal = np.digitize(signal, bins=signal_bins) - 1 \n signal = np.minimum(signal, NUM_BINS - 1)\n return signal","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"train_audio = []\ndirectory = \"/kaggle/input/birdsong-recognition/train_audio/\"\nfor idx, row in train.iterrows():\n train_audio.append(directory+row.bird_name+'/'+row.filename)\n \ntrain_audio = np.array(train_audio)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Converting the audio to Numpy array to make the process easier","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"%%time\nX = np.array([Parallel(n_jobs=4)(delayed(tokenize)(filename) for filename in tqdm_notebook(train_audio))])[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def build_model(MAX_LEN = MAX_LEN, NUM_BINS = NUM_BINS):\n \n ids = L.Input((MAX_LEN,))\n x = L.Embedding(NUM_BINS,132,input_length=MAX_LEN)(ids)\n x = L.Bidirectional(L.LSTM(132, return_sequences=True))(x)\n x = L.Flatten()(x)\n x = L.Dense(len(most_represented_birds)+1,activation='softmax')(x)\n\n model = Model(inputs=[ids], outputs=x)\n optimizer = tf.keras.optimizers.Adam(learning_rate=1e-4)\n model.compile(loss='categorical_crossentropy', optimizer=optimizer,metrics = 'accuracy')\n\n return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"model = build_model()\nprint(model.summary())","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"le = LabelEncoder()\nseed = 43\nle.fit(train['ebird_code'])\nY = pd.get_dummies(train['ebird_code']).values\nle.inverse_transform([1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"X.shape,Y.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"class_weights = class_weight.compute_class_weight('balanced',np.unique(train.ebird_code),train.ebird_code)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"with tf.device('/gpu:0'):\n model.fit(X,Y,epochs=5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"test = pd.read_csv('/kaggle/input/birdsong-recognition/test.csv')\nTEST_FOLDER = '../input/birdsong-recognition/test_audio/'\ntry:\n preds = []\n for index, row in test.iterrows():\n # Get test row information\n site = row['site']\n start_time = row['seconds'] - 5\n row_id = row['row_id']\n audio_id = row['audio_id']\n\n if site == 'site_1' or site == 'site_2':\n x = np.array([tokenize(TEST_FOLDER + audio_id + '.mp3',start_time)])\n else:\n x = np.array([tokenize(TEST_FOLDER + audio_id + '.mp3')])\n \n pred = le.inverse_transform(np.argmax(model.predict(x),axis=1))[0]\n preds.append([row_id, pred])\nexcept:\n preds = pd.read_csv('../input/birdsong-recognition/sample_submission.csv')\n \npreds = pd.DataFrame(preds, columns=['row_id', 'birds'])\npreds.to_csv('submission.csv', index = False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"