{"cells":[{"metadata":{"trusted":true},"cell_type":"code","source":"!nvidia-smi\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# feature extractoring and preprocessing data\nimport librosa\nimport librosa.display\nimport pandas as pd\nimport numpy as np\nimport scipy.signal\nimport matplotlib.pyplot as plt\n%matplotlib inline\nfrom PIL import Image\nfrom pathlib import Path\nfrom pylab import rcParams\nrcParams['figure.figsize'] = 14, 6\n\nimport csv\n# Preprocessing\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import LabelEncoder, StandardScaler\n#Reports\nfrom sklearn.metrics import classification_report, confusion_matrix\n\nimport warnings\nwarnings.filterwarnings('ignore')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Reading some audio samples"},{"metadata":{},"cell_type":"markdown","source":"*Applying Pre-Processing Techniques from Example File*"},{"metadata":{"trusted":true},"cell_type":"code","source":"sr = 16000\nbr_cl1 = '../input/birdsong-recognition/example_test_audio/BLKFR-10-CPL_20190611_093000.pt540.mp3'\nbr_cl2 = '../input/birdsong-recognition/example_test_audio/ORANGE-7-CAP_20190606_093000.pt623.mp3'\n\n# 10 seconds of each file\ny1,sr = librosa.load(br_cl1, mono=True, sr=sr, offset=0, duration=10)\ny2,sr = librosa.load(br_cl2, mono=True, sr=sr, offset=0, duration=10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from IPython.display import Audio, IFrame, display\n\ndisplay(Audio(y1,rate=sr))\ndisplay(Audio(y2,rate=sr))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As you can hear, the first audio presents a high level background noise, and birds seems far from the mic. In the second audio, bird sounds are much more distinguished from the other noises. We can say that the second audio presents a better SNR (signal-noise ratio)."},{"metadata":{"trusted":true},"cell_type":"code","source":"librosa.display.waveplot(y1,sr=sr, x_axis='time')\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"librosa.display.waveplot(y2,sr=sr, x_axis='time');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"If you look to both signal waves, you can see that the first sound presents a lower difference between the background level and some sound events, while this difference is much higher in the second sound. If you do it in sync with the audio, we can see that some bird calls does not appear from the average sound level."},{"metadata":{},"cell_type":"markdown","source":"## Logmel-spectogram"},{"metadata":{},"cell_type":"markdown","source":"A very common preprocessing technique in audio detection applications is to transform audios to its log mel-spectogram representation.\nSome concepts here: https://en.wikipedia.org/wiki/Mel-frequency_cepstrum"},{"metadata":{"trusted":true},"cell_type":"code","source":"S1 = librosa.feature.melspectrogram(y=y1, sr=sr, n_mels=64)\nD1 = librosa.power_to_db(S1, ref=np.max)\nlibrosa.display.specshow(D1, x_axis='time', y_axis='mel');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"S2 = librosa.feature.melspectrogram(y=y2, sr=sr, n_mels=64)\nD2 = librosa.power_to_db(S2, ref=np.max)\nlibrosa.display.specshow(D2, x_axis='time', y_axis='mel');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The differences become very clear in the log mel spectogram. In the fist case, you can see a lot of artefacts on low frequencies (not birds), and the birds are in levels below the background noises. Besides, background noises are higher in frequencies below 2 kHz."},{"metadata":{},"cell_type":"markdown","source":"## Filtering low-frequencies"},{"metadata":{"trusted":true},"cell_type":"code","source":"from scipy import signal\nimport random\n\n\ndef f_high(y,sr):\n    b,a = signal.butter(10, 2000/(sr/2), btype='highpass')\n    yf = signal.lfilter(b,a,y)\n    return yf\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"yf1 = f_high(y1, sr)\nyf2 = f_high(y2, sr)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"librosa.display.waveplot(y1,sr=sr, x_axis='time');\nlibrosa.display.waveplot(yf1,sr=sr, x_axis='time');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"librosa.display.waveplot(y2,sr=sr, x_axis='time');\nlibrosa.display.waveplot(yf2,sr=sr, x_axis='time');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Sf1 = librosa.feature.melspectrogram(y=yf1, sr=sr, n_mels=64)\nDf1 = librosa.power_to_db(Sf1, ref=np.max)\nlibrosa.display.specshow(Df1, x_axis='time', y_axis='mel');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Sf2 = librosa.feature.melspectrogram(y=yf2, sr=sr, n_mels=64)\nDf2 = librosa.power_to_db(Sf2, ref=np.max)\nlibrosa.display.specshow(Df2, x_axis='time', y_axis='mel');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"display(Audio(yf1,rate=sr))\ndisplay(Audio(yf2,rate=sr))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## PCEN"},{"metadata":{},"cell_type":"markdown","source":"PCEN has become a very useful strategy for acoustic event detection, and it has shown to perform better in such tasks as a frontend. Its idea is to perform non-linear compression on time-frequency channels.\n\nI am using the example shown here: https://librosa.org/doc/latest/generated/librosa.pcen.html?highlight=pcen#librosa.pcen"},{"metadata":{"trusted":true},"cell_type":"code","source":"Dp1 = librosa.pcen(S1 * (2**31), sr=sr, gain=1.1, hop_length=512, bias=2, power=0.5, time_constant=0.8, eps=1e-06, max_size=2)\nDp2 = librosa.pcen(S2 * (2**31), sr=sr, gain=1.1, hop_length=512, bias=2, power=0.5, time_constant=0.8, eps=1e-06, max_size=2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"librosa.display.specshow(Dp1, x_axis='time', y_axis='mel');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"librosa.display.specshow(Dp2, x_axis='time', y_axis='mel');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"yp1 = librosa.feature.inverse.mel_to_audio(Dp1)\nyp2 = librosa.feature.inverse.mel_to_audio(Dp2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"librosa.display.waveplot(yp1,sr=sr, x_axis='time');\nlibrosa.display.waveplot(y1,sr=sr, x_axis='time');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"librosa.display.waveplot(yp2,sr=sr, x_axis='time');\nlibrosa.display.waveplot(y2,sr=sr, x_axis='time');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"display(Audio(yp1,rate=sr))\ndisplay(Audio(yp2,rate=sr))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Spectral Gating"},{"metadata":{"trusted":true},"cell_type":"code","source":"import time\nfrom datetime import timedelta as td\n\n\ndef _stft(y, n_fft, hop_length, win_length):\n    return librosa.stft(y=y, n_fft=n_fft, hop_length=hop_length, win_length=win_length)\n\n\ndef _istft(y, hop_length, win_length):\n    return librosa.istft(y, hop_length, win_length)\n\n\ndef _amp_to_db(x):\n    return librosa.core.amplitude_to_db(x, ref=1.0, amin=1e-20, top_db=80.0)\n\n\ndef _db_to_amp(x,):\n    return librosa.core.db_to_amplitude(x, ref=1.0)\n\n\ndef plot_spectrogram(signal, title):\n    fig, ax = plt.subplots(figsize=(20, 4))\n    cax = ax.matshow(\n        signal,\n        origin=\"lower\",\n        aspect=\"auto\",\n        cmap=plt.cm.seismic,\n        vmin=-1 * np.max(np.abs(signal)),\n        vmax=np.max(np.abs(signal)),\n    )\n    fig.colorbar(cax)\n    ax.set_title(title)\n    plt.tight_layout()\n    plt.show()\n\n\ndef plot_statistics_and_filter(\n    mean_freq_noise, std_freq_noise, noise_thresh, smoothing_filter\n):\n    fig, ax = plt.subplots(ncols=2, figsize=(20, 4))\n    plt_mean, = ax[0].plot(mean_freq_noise, label=\"Mean power of noise\")\n    plt_std, = ax[0].plot(std_freq_noise, label=\"Std. power of noise\")\n    plt_std, = ax[0].plot(noise_thresh, label=\"Noise threshold (by frequency)\")\n    ax[0].set_title(\"Threshold for mask\")\n    ax[0].legend()\n    cax = ax[1].matshow(smoothing_filter, origin=\"lower\")\n    fig.colorbar(cax)\n    ax[1].set_title(\"Filter for smoothing Mask\")\n    plt.show()\n\ndef removeNoise(\n    audio_clip,\n    noise_clip,\n    n_grad_freq=2,\n    n_grad_time=4,\n    n_fft=2048,\n    win_length=2048,\n    hop_length=512,\n    n_std_thresh=1.5,\n    prop_decrease=1.0,\n    verbose=False,\n    visual=False,\n):\n    \"\"\"Remove noise from audio based upon a clip containing only noise\n\n    Args:\n        audio_clip (array): The first parameter.\n        noise_clip (array): The second parameter.\n        n_grad_freq (int): how many frequency channels to smooth over with the mask.\n        n_grad_time (int): how many time channels to smooth over with the mask.\n        n_fft (int): number audio of frames between STFT columns.\n        win_length (int): Each frame of audio is windowed by `window()`. The window will be of length `win_length` and then padded with zeros to match `n_fft`..\n        hop_length (int):number audio of frames between STFT columns.\n        n_std_thresh (int): how many standard deviations louder than the mean dB of the noise (at each frequency level) to be considered signal\n        prop_decrease (float): To what extent should you decrease noise (1 = all, 0 = none)\n        visual (bool): Whether to plot the steps of the algorithm\n\n    Returns:\n        array: The recovered signal with noise subtracted\n\n    \"\"\"\n    if verbose:\n        start = time.time()\n    # STFT over noise\n    noise_stft = _stft(noise_clip, n_fft, hop_length, win_length)\n    noise_stft_db = _amp_to_db(np.abs(noise_stft))  # convert to dB\n    # Calculate statistics over noise\n    mean_freq_noise = np.mean(noise_stft_db, axis=1)\n    std_freq_noise = np.std(noise_stft_db, axis=1)\n    noise_thresh = mean_freq_noise + std_freq_noise * n_std_thresh\n    if verbose:\n        print(\"STFT on noise:\", td(seconds=time.time() - start))\n        start = time.time()\n    # STFT over signal\n    if verbose:\n        start = time.time()\n    sig_stft = _stft(audio_clip, n_fft, hop_length, win_length)\n    sig_stft_db = _amp_to_db(np.abs(sig_stft))\n    if verbose:\n        print(\"STFT on signal:\", td(seconds=time.time() - start))\n        start = time.time()\n    # Calculate value to mask dB to\n    mask_gain_dB = np.min(_amp_to_db(np.abs(sig_stft)))\n    #print(noise_thresh, mask_gain_dB)\n    # Create a smoothing filter for the mask in time and frequency\n    smoothing_filter = np.outer(\n        np.concatenate(\n            [\n                np.linspace(0, 1, n_grad_freq + 1, endpoint=False),\n                np.linspace(1, 0, n_grad_freq + 2),\n            ]\n        )[1:-1],\n        np.concatenate(\n            [\n                np.linspace(0, 1, n_grad_time + 1, endpoint=False),\n                np.linspace(1, 0, n_grad_time + 2),\n            ]\n        )[1:-1],\n    )\n    smoothing_filter = smoothing_filter / np.sum(smoothing_filter)\n    # calculate the threshold for each frequency/time bin\n    db_thresh = np.repeat(\n        np.reshape(noise_thresh, [1, len(mean_freq_noise)]),\n        np.shape(sig_stft_db)[1],\n        axis=0,\n    ).T\n    # mask if the signal is above the threshold\n    sig_mask = sig_stft_db < db_thresh\n    if verbose:\n        print(\"Masking:\", td(seconds=time.time() - start))\n        start = time.time()\n    # convolve the mask with a smoothing filter\n    sig_mask = scipy.signal.fftconvolve(sig_mask, smoothing_filter, mode=\"same\")\n    sig_mask = sig_mask * prop_decrease\n    if verbose:\n        print(\"Mask convolution:\", td(seconds=time.time() - start))\n        start = time.time()\n    # mask the signal\n    sig_stft_db_masked = (\n        sig_stft_db * (1 - sig_mask)\n        + np.ones(np.shape(mask_gain_dB)) * mask_gain_dB * sig_mask\n    )  # mask real\n    sig_imag_masked = np.imag(sig_stft) * (1 - sig_mask)\n    sig_stft_amp = (_db_to_amp(sig_stft_db_masked) * np.sign(sig_stft)) + (\n        1j * sig_imag_masked\n    )\n    if verbose:\n        print(\"Mask application:\", td(seconds=time.time() - start))\n        start = time.time()\n    # recover the signal\n    recovered_signal = _istft(sig_stft_amp, hop_length, win_length)\n    recovered_spec = _amp_to_db(\n        np.abs(_stft(recovered_signal, n_fft, hop_length, win_length))\n    )\n    if verbose:\n        print(\"Signal recovery:\", td(seconds=time.time() - start))\n    if visual:\n        plot_spectrogram(noise_stft_db, title=\"Noise\")\n    if visual:\n        plot_statistics_and_filter(\n            mean_freq_noise, std_freq_noise, noise_thresh, smoothing_filter\n        )\n    if visual:\n        plot_spectrogram(sig_stft_db, title=\"Signal\")\n    if visual:\n        plot_spectrogram(sig_mask, title=\"Mask applied\")\n    if visual:\n        plot_spectrogram(sig_stft_db_masked, title=\"Masked signal\")\n    if visual:\n        plot_spectrogram(recovered_spec, title=\"Recovered spectrogram\")\n    return recovered_signal","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"noise1 = y1[5*sr:6*sr]\nyg1 = removeNoise(audio_clip=y1, noise_clip=noise1,\n    n_grad_freq=2,\n    n_grad_time=4,\n    n_fft=2048,\n    win_length=2048,\n    hop_length=512,\n    n_std_thresh=1.5,\n    prop_decrease=1.0,\n    verbose=False,\n    visual=False)\nnoise2 = y2[0:1*sr]\nyg2 = removeNoise(audio_clip=y2, noise_clip=noise2,\n    n_grad_freq=2,\n    n_grad_time=4,\n    n_fft=2048,\n    win_length=2048,\n    hop_length=512,\n    n_std_thresh=2.5,\n    prop_decrease=1.0,\n    verbose=False,\n    visual=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"librosa.display.waveplot(y1,sr=sr, x_axis='time');\nlibrosa.display.waveplot(yg1,sr=sr, x_axis='time');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"librosa.display.waveplot(y2,sr=sr, x_axis='time');\nlibrosa.display.waveplot(yg2,sr=sr, x_axis='time');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Sg1 = librosa.feature.melspectrogram(y=yg1, sr=sr, n_mels=64)\nDg1 = librosa.power_to_db(Sg1, ref=np.max)\nlibrosa.display.specshow(Dg1, x_axis='time', y_axis='mel');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Sg2 = librosa.feature.melspectrogram(y=yg2, sr=sr, n_mels=64)\nDg2 = librosa.power_to_db(Sg2, ref=np.max)\nlibrosa.display.specshow(Dg2, x_axis='time', y_axis='mel');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"display(Audio(yg1,rate=sr))\ndisplay(Audio(yg2,rate=sr))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# loading train data\ntrain_df = pd.read_csv('../input/birdsong-recognition/train.csv')\ntest_df = pd.read_csv('../input/birdsong-recognition/test.csv')\ntrain_df.head(3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# checking the train data set\ntrain_df.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import seaborn as sns\nsns.set_style('darkgrid')\nfig = plt.figure(figsize=(15,5))\nax1 = fig.add_subplot(121)\nsns.distplot(train_df.species.value_counts(),ax=ax1)\nax1.set_title(\"Distribution of number of bird species\")\nax1.set_xlabel(\"Count of bird species\")\n\ncounts, bin_edges = np.histogram(train_df.species.value_counts(), bins=10, density = True)\n\n# plotting cdf \n# https://stackoverflow.com/questions/10640759/how-to-get-the-cumulative-distribution-function-with-numpy\npdf = counts/(sum(counts))\ncdf = np.cumsum(pdf)\nax2 = fig.add_subplot(122)\nax2.plot(bin_edges[1:], cdf)\nax2.set_title(\"Commulative distribution of count of the species\")\nax2.set_xlabel(\"Count\")\nax2.set_ylabel(\"percentage\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12,6))\nax = sns.countplot(x='rating',data=train_df)\nplt.title(\"Distribution of Ratings of the Audio\", fontsize=16)\nplt.xlabel(\"ratings\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"****Predictive Modelling****"},{"metadata":{},"cell_type":"markdown","source":"Importing Libraries"},{"metadata":{"trusted":true},"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\")\nimport pandas as pd\nimport numpy as np\nimport zipfile\nfrom IPython.display import Audio\nfrom pydub import AudioSegment\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport seaborn as sns\nimport os\nimport pickle\nimport random\nfrom tqdm import tqdm\nimport datetime\n#from audiomentations import *\nimport multiprocessing\nimport cv2\n# # library used to process the audio files\nimport librosa\nimport librosa.display\nimport sklearn.preprocessing\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import f1_score\nimport tensorflow as tf\nimport tensorflow_addons as tfa\nfrom tensorflow.keras.utils import to_categorical\nfrom tensorflow.keras.utils import plot_model\nfrom tensorflow.keras.layers import Input, Dense, Conv2D, Softmax, LSTM, Embedding, concatenate\nfrom tensorflow.keras.layers import Dropout, BatchNormalization,GlobalAveragePooling2D\nfrom tensorflow.keras.applications import ResNet50, InceptionV3, VGG16\nfrom tensorflow.keras.models import Model,Sequential\nfrom tensorflow.keras.callbacks import ModelCheckpoint, TensorBoard, ReduceLROnPlateau, EarlyStopping\nfrom IPython.display import Image","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Data Loading"},{"metadata":{"trusted":true},"cell_type":"code","source":"# loading train data\ntrain_df = pd.read_csv('../input/birdsong-recognition/train.csv')\ntest_df = pd.read_csv('../input/birdsong-recognition/example_test_audio_summary.csv')\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"null_columns=train_df.columns[train_df.isnull().any()]\ntrain_df[null_columns].isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data4=list()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\n\ntrain_audio_path = '../input/birdsong-recognition/train_audio'\nprint(\"train_audio exists: \", os.path.exists(train_audio_path))\n\nfor dirname, _, filenames in os.walk(train_audio_path):\n    _.sort()\n    for filename in sorted(filenames):\n        print(os.path.join(dirname, filename))\n        data4.append((dirname,filename))\n        \ndf2= pd.DataFrame(data4, columns=['dirnames','filename'])\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df2.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df2.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df2['directory'] = df2['dirnames'].str.cat(df2['filename'], sep =\"/\") \nprint(df2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df2.tail()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df2.isnull().sum","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# concatenating the DataFrames \ndf3 = pd.concat([train_df, df2], join = 'outer', axis = 1) \n  \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df3.isnull().sum","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df3.tail()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df=df3","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Analyzing the Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"# checking the train data set\ntrain_df.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Checking the unique types of bird species in the data\nprint(\"Number of bird species:\",len(train_df['species'].unique()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_labels = train_df['ebird_code'].unique()\nprint(\"Bird species in abbrebviated: \",y_labels[:5])\nprint(\"BIrd Speccies: \",train_df['species'].unique()[:5])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Note: The birdnames have been abbreviated, and there are 263 unique classes."},{"metadata":{},"cell_type":"markdown","source":"Encoding & Preparing the Dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"# dataframe containg the path to the audio files and the class labels\ndf_audio = train_df[['filename','directory','duration','sampling_rate','ebird_code']]\ndf_audio.to_csv('df_audio.csv',index=False)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ndf_audio = pd.read_csv('df_audio.csv')\ndf_audio.head(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_audio = pd.read_csv('df_audio.csv')\ndf_audio.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''\n#split the data into train and validation and save in X_train, X_cv, y_train, y_cv\n#use stratify sampling,random state of 45 and validaton size of 20%\nX_train, X_test, y_train, y_test = train_test_split(df_audio,df_audio.ebird_code,test_size=0.2, \n                                                    random_state=45,stratify=df_audio.ebird_code)\n\nprint(\"Training data: \",X_train.shape,y_train.shape)\nprint(\"Validataion data: \",X_test.shape,y_test.shape)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nbird_species_mapping = dict(zip(train_df['ebird_code'],train_df['species']))\n# Integer coding the class labels\nebird_code_labeled = dict()\nebird_code_classes = dict()\n\nebird_code_labeled['nocall'] = 0\nebird_code_classes[0] = 'nocall'\nfor i, bird in enumerate(df_audio['ebird_code'].unique()):\n    ebird_code_labeled[bird] = i+1\n    ebird_code_classes[i+1] = bird\n    \n# pickle.dump((bird_species_mapping,ebird_code_labeled,ebird_code_classes), open('model_logs/bird_species_code.pkl','wb'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Generating samples of 5 seconds from the audio files. (Many audio files don't have multiple of 20 second split)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# sampling audio to 5 seconds length\ndef raw_data(filename,bird_code,raw_df):\n    sample_rate = 44100\n    try:\n        samples, sample_rate = librosa.load(filename,sr=44100)\n    except:\n        sound = AudioSegment.from_file(filename)\n        samples = sound.get_array_of_samples()\n        new_sound = sound._spawn(samples)\n        samples = np.array(samples).astype(np.float32)\n\n    samples, _ = librosa.effects.trim(samples)\n    data_point_per_second = 10\n    \n    #Take 10 data points every second\n    samples = samples[0::int(sample_rate/data_point_per_second)]\n    #We normalize each sample before extracting 5s samples from it\n    normalized_sample = sklearn.preprocessing.minmax_scale(samples, axis=0)\n    \n    #only take 5s samples and add them to the dataframe\n    song_sample = []\n    sample_length = 5*data_point_per_second\n    for idx in range(0,len(normalized_sample),sample_length): \n        song_sample = normalized_sample[idx:idx+sample_length]\n        if len(song_sample)>=sample_length:\n            raw_df = raw_df.append({\"song_samples\":np.asarray(song_sample).astype(np.float32),\n                                            \"bird_code\":ebird_code_labeled[bird_code]}, ignore_index=True)\n    return raw_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''\nraw_df = pd.DataFrame(columns=[\"song_samples\",\"bird_code\"])\nfor i, row in tqdm(df_audio[['directory','ebird_code']][12327:].iterrows()):\n    raw_df = raw_data(row.directory, row.ebird_code, raw_df)\n    \nraw_df.to_csv('raw_df.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# stacking the image to 3 channels for CNN models\ndef mono_to_color(X: np.ndarray, mean=None, std=None, norm_max=None, norm_min=None, eps=1e-6):\n    # Stack X as [X,X,X]\n    X = np.stack([X, X, X], axis=-1)\n\n    # Standardize\n    mean = mean or X.mean()\n    X = X - mean\n    std = std or X.std()\n    Xstd = X / (std + eps)\n    _min, _max = Xstd.min(), Xstd.max()\n    norm_max = norm_max or _max\n    norm_min = norm_min or _min\n    if (_max - _min) > eps:\n        # Normalize to [0, 255]\n        V = Xstd\n        V[V < norm_min] = norm_min\n        V[V > norm_max] = norm_max\n        V = 255 * (V - norm_min) / (norm_max - norm_min)\n        V = V.astype(np.uint8)\n    else:\n        # Just zero\n        V = np.zeros_like(Xstd, dtype=np.uint8)\n    return V","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nos.makedirs('./processed_data/ResNet/tr/images_data')\nos.makedirs('./processed_data/Inception/tr/images_data')\nos.makedirs('./processed_data/spec_data/tr/images_data')\n\n#test dir\nos.makedirs('./processed_data/ResNet/val/images_data')\nos.makedirs('./processed_data/Inception/val/images_data')\nos.makedirs('./processed_data/spec_data/val/images_data')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls ..","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# sampling audio to 5 seconds length\ndef spectrogram_images(directory,bird_code):\n    sample_rate = 44100\n    try:\n        samples, sample_rate = librosa.load(directory,sr=44100)\n    except:\n        sound = AudioSegment.from_file(directory)\n        samples = sound.get_array_of_samples()\n        new_sound = sound._spawn(samples)\n        samples = np.array(samples).astype(np.float32)\n\n    samples, _ = librosa.effects.trim(samples)\n    \n    #only take 5s samples and save them to the np array\n    sample_length = 5*sample_rate\n    X = np.abs(librosa.stft(samples[:44100*5], n_fft=1024, hop_length=512))\n    spectrogram = librosa.amplitude_to_db(X)\n    image = mono_to_color(spectrogram)\n\n    res = cv2.resize(image, (224, 224))\n    res = (res / 255.0).astype(np.float32)\n    res = np.asarray(res)\n    \n    incep = cv2.resize(image, (299, 299))\n    incep = (incep / 255.0).astype(np.float32)\n    incep = np.asarray(incep)\n    \n    label = ebird_code_labeled[bird_code]\n    \n    return spectrogram,res,incep,label\n\n\ndef func(start,end,idx):\n    labels = []\n    img_res = []\n    img_incep = []\n    spec = []\n\n    for i, row in X_train[['directory','ebird_code']][start*idx:end*(idx+1)].iterrows():\n        mel_spec,res,incep,label = spectrogram_images(row.directory, row.ebird_code)\n        spec.extend(mel_spec)\n        img_res.append(res)\n        img_incep.append(incep)\n        labels.append(label)\n    \n    # stroing the data as npz file for easy access\n#     np.savez_compressed('processed_data/VGG/STFT/train/images_data'+str(idx),a=spec,b=labels)\n    np.savez_compressed('processed_data/ResNet/tr/images_data'+str(idx),a=img_res,b=labels)\n    np.savez_compressed('processed_data/Inception/tr/images_data'+str(idx),a=img_incep,b=labels)\n    np.savez_compressed('processed_data/spec_data/tr/images_data'+str(idx),a=spec,b=labels)\n    \ndef func1(start,end,idx):\n    labels = []\n    img_res = []\n    img_incep = []\n    spec = []\n\n    for i, row in X_cv[['directory','ebird_code']][start*idx:end*(idx+1)].iterrows():\n        mel_spec,res,incep,label = spectrogram_images(row.directory, row.ebird_code)\n        spec.extend(mel_spec)\n        img_res.append(res)\n        img_incep.append(incep)\n        labels.append(label)\n        \n#     np.savez_compressed('processed_data/VGG/STFT/validation/images_data'+str(idx),a=spec,b=labels)    \n    np.savez_compressed('processed_data/ResNet/val/images_data'+str(idx),a=img_res,b=labels)\n    np.savez_compressed('processed_data/Inception/val/images_data'+str(idx),a=img_incep,b=labels)\n    np.savez_compressed('processed_data/spec_data/val/images_data'+str(idx),a=spec,b=labels)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nfor i in tqdm(range(0,535,8)):\n    if i != 528:\n        p1 = multiprocessing.Process(target=func, args=(32,32,i))\n        p2 = multiprocessing.Process(target=func, args=(32,32,i+1))\n        p3 = multiprocessing.Process(target=func, args=(32,32,i+2))\n        p4 = multiprocessing.Process(target=func, args=(32,32,i+3))\n        p5 = multiprocessing.Process(target=func, args=(32,32,i+4))\n        p6 = multiprocessing.Process(target=func, args=(32,32,i+5))\n        p7 = multiprocessing.Process(target=func, args=(32,32,i+6))\n        p8 = multiprocessing.Process(target=func, args=(32,32,i+7))\n\n        p1.start()\n        p2.start()\n        p3.start()\n        p4.start()\n        p5.start()\n        p6.start()\n        p7.start()\n        p8.start()\n\n        p1.join()\n        p2.join()\n        p3.join()\n        p4.join()\n        p5.join()\n        p6.join()\n        p7.join()\n        p8.join()\n    else:\n        p1 = multiprocessing.Process(target=func, args=(32,32,i))\n        p2 = multiprocessing.Process(target=func, args=(32,32,i+1))\n        p3 = multiprocessing.Process(target=func, args=(32,32,i+2))\n        p4 = multiprocessing.Process(target=func, args=(32,32,i+3))\n        p5 = multiprocessing.Process(target=func, args=(32,32,i+4))\n        p6 = multiprocessing.Process(target=func, args=(32,32,i+5))\n        p7 = multiprocessing.Process(target=func, args=(32,32,i+6))\n\n        p1.start()\n        p2.start()\n        p3.start()\n        p4.start()\n        p5.start()\n        p6.start()\n        p7.start()\n\n        p1.join()\n        p2.join()\n        p3.join()\n        p4.join()\n        p5.join()\n        p6.join()\n        p7.join()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('ac')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\n# train set\ntrain_dir_path = \"./processed_data/Inception/tr\"\ntrain_files_path = os.listdir(train_dir_path)\ntrain_gen_res = DataGenerator(train_dir_path, train_files_path)\n\n# test set\ntest_dir_path = \"./processed_data/Inception/val\"\ntest_files_path = os.listdir(test_dir_path)\ntest_gen_res = DataGenerator(train_dir_path, test_files_path)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true},"cell_type":"code","source":"os.makedirs('/kaggle/tmp')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from distutils.dir_util import copy_tree\n\nfromDir='./processed_data'\ntoDir='/kaggle/tmp'\n\ncopy_tree(fromDir,toDir)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import shutil\nshutil.make_archive('/kaggle/tmp', 'zip', '/kaggle/tmp')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from IPython.display import FileLink\nFileLink('/kaggle/tmp.zip')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}