Hi everyone, this is one of my first notebooks here on Kaggle so i hope that you could enjoy. So here we go, my engineer degree makes me think in this project like a signal processing, nothing special on this kind of project, in general terms a sound is a signal over time taking by a microphone. That’s the reason why other goodfellas out there face this problem in a similar way. This project demands us to use fast Fourier transform and MFCC to extract feature from the wave sound, these techniques allow us to see inside the wave sound and understand the behavior over time for the different frequencies that compounds that sound.
\n\n
Maybe you ask, why did i use those techniques? what’s so special? I tell you what, you already have a similar algorithm inside your brain that allows you to differentiate between bass and sharp sounds, also you could perceive the difference between a sound at 100Hz and 400Hz, but is very difficult perceive a difference between a sound at 10Khz and 12Khz, that’s because we didn’t perceive the sound in a linear way, that’s why we use MFCC which is a combination of algorithms among them MEL, which compress frequencies representation from linear to logarithmic. About this topic I going to write bellow on his corresponding cell, so check it out.
\n\n
Finally, I going to face this problem using ConvNN-BI.LSMT-LSTM-ML to create a model to classify every clip of 5 s length, then add noise to those clip to augment our data, all this using a generator for TensorFlow. Hence, for the TEST prediction I’ll turn into little audio clips every audio test and them make the inference and merge all of them and create my submission table. If you have any other ideas or opinions, please let me know in the comments bellow.
The submission file only contains information about species detected in the audio file, so we don’t need time localization. This is a problem well known like weak annotation for Sound Event Detection (SED) . We going to analyze the submission next.
\n\n
\n \n
"},{"metadata":{},"cell_type":"markdown","source":"# 1. DataSet General file overview\n\n* train_tp.csv - training data of true positive species labels, with corresponding time localization\n* train_fp.csv - training data of false positives species labels, with corresponding time localization\n* sample_submission.csv - a sample submission file in the correct format; note each species column has an s prefix.\n* train/ - the training audio files\n* test/ - the test audio files; the task is to predict the species found in each audio file\n* tfrecords/{train,test} - competition data in the TFRecord format, which includes recording_id, audio_wav (encoded in 16-bit PCM format), and label_info (for train only), which provides a , -delimited string of the columns below (minus recording_id), where multiple labels for a recording_id are ; -delimited."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"import numpy as np \nimport pandas as pd \n\n# Input data files are available in the read-only \"../input/\" directory\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# paths\nsample_submission = \"../input/rfcx-species-audio-detection/sample_submission.csv\"\ntfrecords = \"../input/rfcx-species-audio-detection/tfrecords\"\ntest_folder = \"../input/rfcx-species-audio-detection/test\"\ntrain_folder = \"../input/rfcx-species-audio-detection/train\"\nfile_csv_tp = \"../input/rfcx-species-audio-detection/train_tp.csv\"\nfile_csv_fp = \"../input/rfcx-species-audio-detection/train_fp.csv\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import gc\nimport os\nimport random\nimport numpy as np\nimport pandas as pd\nfrom tqdm import tqdm\n\nimport seaborn as sns\nimport IPython.display as ipD\nimport matplotlib.pyplot as plt\nimport matplotlib.patches as ptc\nimport soundfile as sf\n\nimport plotly.figure_factory as ff\nimport plotly.graph_objects as go\nimport plotly.express as px\nfrom plotly.offline import iplot\n\nfrom sklearn.preprocessing import StandardScaler, MinMaxScaler\n\nmm = MinMaxScaler()\nss = StandardScaler()\n\nimport librosa\nimport librosa.display as LD\n\nimport pylab\nimport tensorflow as tf\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Dense, Dropout, Activation, Flatten, BatchNormalization,LSTM,Reshape,Input, Lambda,Bidirectional\nfrom tensorflow.keras.layers import Conv2D, MaxPooling2D\nfrom tensorflow.keras.utils import to_categorical\nfrom tensorflow.keras import optimizers\nfrom tensorflow.keras.callbacks import ModelCheckpoint, LearningRateScheduler, TensorBoard,EarlyStopping\nimport tensorflow.keras as keras\nimport time\n\n\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2. EDA"},{"metadata":{},"cell_type":"markdown","source":"## 2.1 EDA CSV files\n\nAccording to the metadata from Kaggle the columns have the following explanations.\n\n* **recording_id** - unique identifier for recording\n* **species_id** - unique identifier for species\n* **songtype_id** - unique identifier for songtype\n* **t_min** - start second of annotated signal\n* **f_min** - lower frequency of annotated signal\n* **t_max** - end second of annotated signal\n* **f_max**- upper frequency of annotated signal\n* **is_tp**- [tfrecords only] an indicator of whether the label is from the train_tp (1) or train_fp (0) file.\n"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"table_tp = pd.read_csv(file_csv_tp)\nipD.display(table_tp.nunique())\nipD.display(table_tp.info())\ntable_tp.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"table_fp = pd.read_csv(file_csv_fp)\nipD.display(table_fp.nunique())\nipD.display(table_fp.info())\ntable_fp.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"table_submission = pd.read_csv(sample_submission)\nipD.display(table_submission.nunique())\nipD.display(table_submission.info())\ntable_submission.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.2. Specie ID representation"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"temp_df = table_tp[\"species_id\"].value_counts().reset_index() \ntemp_df.columns = [\"specie\", \"count\"]\ntemp_df.sort_values(by = \"specie\", inplace= True )\ntemp_df[\"specie\"] = temp_df[\"specie\"].astype(str)\n\ntemp_df_fp = table_fp[\"species_id\"].value_counts().reset_index() \ntemp_df_fp.columns = [\"specie\", \"count\"]\ntemp_df_fp[\"specie\"] = temp_df_fp[\"specie\"].astype(str)\ntemp_df_fp.sort_values(by = \"specie\", inplace= True )\n\ntrace1 = go.Bar(\n x = temp_df['specie'],\n y = temp_df['count'], name =\"TP\",\n marker = dict(color = 'rgb(1,200,15)',\n line=dict(color='rgb(0,0,0)',width=1.5)),\n text=temp_df['count'], textposition='outside')\n\ntrace2 = go.Bar(\n x = temp_df_fp['specie'],\n y = -temp_df_fp['count'], name =\"FP\",\n marker = dict(color = 'rgb(250,13,92)',\n line=dict(color='rgb(0,0,0)',width=1.5)),\n text=temp_df_fp['count'], textposition='outside')\n\n\n\nlayout = go.Layout(template= \"plotly_dark\",title = 'SPECIES DISTRIBUTION TP vs FP' , xaxis = dict(title = 'Specie ID'), yaxis = dict(title = 'Count'), height=650)\nfig = go.Figure(data = [trace1,trace2], layout = layout)\nfig.update_yaxes(showticklabels=False)\nfig.update_layout(barmode='overlay')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2.3. Song type char representation\n\nFor a naked eye this seems to be irrelevant for the model, because exist only on two species, maybe analysis those species, could be relevant, but not for the whole model. So, song type going to be ignore this feature for now.\n\nIn the comments I’ve found that exist because some birds have two different ways to sing, that´s a cool interesting topic, so we could analyze those birds like a distinct species and by the end merge each other for the submission. *[I´ll not cover that on this notebook sorry]*"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"trace = go.Pie(labels=table_tp[\"songtype_id\"].value_counts().index.sort_values(), \n values=table_tp[\"songtype_id\"].value_counts().values, \n hoverinfo='percent+value+label', \n textinfo='percent',\n textposition='inside',\n showlegend=True,\n marker=dict(colors=plt.cm.viridis_r(np.linspace(0, 1, 28)),\n line=dict(color='#000000',\n width=2),\n )\n )\n\ntrace2 = go.Pie(labels=table_fp[\"songtype_id\"].value_counts().index.sort_values(), \n values=table_fp[\"songtype_id\"].value_counts().values, \n hoverinfo='percent+value+label', \n textinfo='percent',\n textposition='inside',\n showlegend=True,\n marker=dict(colors=plt.cm.viridis_r(np.linspace(0, 1, 28)),\n line=dict(color='#000000',\n width=2),\n )\n )\n\nlayout = go.Layout(title=\"Songtype ID in TP\",template= \"plotly_dark\")\nfig=go.Figure(data=[trace], layout=layout)\nfig.show()\n\nlayout = go.Layout(title=\"Songtype ID in FP\",template= \"plotly_dark\")\nfig=go.Figure(data=[trace2], layout=layout)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.figure(figsize=(18, 6))\nsns.countplot(x=\"species_id\", hue=\"songtype_id\", data=table_tp, palette = sns.color_palette(\"Dark2\", 3))\nplt.title(\"Species ID distribution for TRUE POSITIVES grouped by song_type\")\nplt.show()\n\nplt.figure(figsize=(18, 6))\nsns.countplot(x=\"species_id\", hue=\"songtype_id\", data=table_fp)\nplt.title(\"Species ID distribution for False POSITIVES grouped by song_type\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2.4. Frequencies behavior"},{"metadata":{"trusted":true},"cell_type":"code","source":"fmean_table = table_tp[[\"recording_id\",\"species_id\",\"f_min\", \"f_max\"]].groupby(\"species_id\").mean().reset_index()\nfmean_table.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fmean_table[\"species_id\"] = fmean_table[\"species_id\"].astype(str)\n\ntrace1 = go.Bar(\n x = fmean_table['f_max'].astype(int).apply(lambda x: np.log10(x)),\n y = fmean_table['species_id'], name =\"F_max\", orientation='h',\n marker = dict(color = 'rgb(1,200,0)',\n line=dict(color='rgb(0,0,0)',width=3.0)),\n text=fmean_table['f_max'].astype(int), textposition='outside')\n\ntrace2 = go.Bar(\n x = fmean_table['f_min'].astype(int).apply(lambda x: np.log10(x)),\n y = fmean_table['species_id'], name =\"F_min\", orientation='h',\n marker = dict(color = 'rgb(250,13,92)',\n line=dict(color='rgb(0,0,0)',width=3.0)),\n text=fmean_table['f_min'].astype(int), textposition='outside')\n\nlayout = go.Layout(template= \"plotly_dark\",title = 'Mean - Frequencies MIN MAX for every Specie' , xaxis = dict(title = 'frequency Hz [LOG]'),\n yaxis = dict(title = 'Specie ID'), \n height=650\n )\nfig = go.Figure(data = [trace2, trace1], layout = layout,)\nfig.update_xaxes(showticklabels=False)\nfig.update_layout(barmode='stack')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2.5 Mean - length of every audio for every specie"},{"metadata":{"trusted":true},"cell_type":"code","source":"time_table = table_tp[[\"recording_id\",\"species_id\",\"t_min\", \"t_max\"]]\ntime_table[\"dt\"] = time_table[\"t_max\"] - time_table[\"t_min\"]\ntime_table = time_table[[\"species_id\", \"dt\"]].groupby(\"species_id\").mean()\n\ntemp_df = time_table.reset_index() \ntemp_df.columns = [\"specie\", \"dt\"]\ntemp_df.sort_values(by = \"specie\", inplace= True )\ntemp_df[\"specie\"] = temp_df[\"specie\"].astype(str)\n\n\ntrace1 = go.Bar(\n x = temp_df['specie'],\n y = temp_df['dt'], name =\"TP\",\n marker = dict(color = 'rgb(1,200,15)',\n line=dict(color='rgb(0,0,0)',width=1.5)),\n text=temp_df['dt'].apply(lambda x: round(x,2)), textposition='outside')\n\n\n\nlayout = go.Layout(template= \"plotly_dark\",title = 'Mean - length of every audio for every specie' , xaxis = dict(title = 'Specie ID'), yaxis = dict(title = 'Seconds (s)'), height=650)\nfig = go.Figure(data = [trace1], layout = layout)\nfig.update_yaxes(showticklabels=True)\nfig.update_layout(barmode='overlay')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2.6. Audio exploration\n\nThe next that i going to do is, pick a random sample audio, from the folder Test and train, them open those using librosa libreri to get the audio in vector format, this is the beggining of audio analisis."},{"metadata":{"trusted":true},"cell_type":"code","source":"# choose a sample from train and test\ntr = os.path.join(train_folder, os.listdir(train_folder)[np.random.randint(0, len(os.listdir(train_folder)))])\nts = os.path.join(test_folder, os.listdir(test_folder)[np.random.randint(0, len(os.listdir(test_folder)))])\n\n# load the np array and the samping rate\ntrx, trsr = librosa.load(tr)\ntsx, tssr = librosa.load(ts)\nrecording_id_train_sample = (tr.split(\"/\")[-1]).split(\".\")[0]\nrecording_id_test_sample = (ts.split(\"/\")[-1]).split(\".\")[0]\n\n# Print audio conincidences inside the dataset\nprint(\"=\"*10, \"Training Sample\", \"=\"*10)\nipD.display(table_fp.append(table_tp, ignore_index=True)[table_fp.append(table_tp, ignore_index=True)[\"recording_id\"]==recording_id_train_sample] )\nprint(\"=\"*10, \"Test Sample\", \"=\"*10)\nprint(\"Test Data: \", recording_id_test_sample)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print the shape and the sampling rate\nprint(\"=====Train sample=======\")\nprint(trx.shape, trsr)\nprint(\"=====Test sample=======\")\nprint(tsx.shape, tssr)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2.6.1. Audio signal Plotting\n"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def graphWaveform(signal, sr): \n trace1 = {\n \"mode\": \"lines\", \n \"name\": \"Signal with last point\", \n \"type\": \"scatter\", \n \"x\":[1/sr*pos for pos in range(len(signal))],\n \"y\":signal,\n }\n layout = {\n \"title\": \"Waveform Audio Signal\", \n \"xaxis\": {\n \"title\": \"Time (s)\", \n \"titlefont\": {\n \"size\": 14, \n \"color\": \"black\", \n \"family\": \"Courier New, monospace\"\n }\n }, \n \"yaxis\": {\n \"title\": \"Amp\", \n \"titlefont\": {\n \"size\": 14, \n \"color\": \"black\", \n \"family\": \"Courier New, monospace\"\n }\n }, \n \"legend\": {\n \"font\": {\n \"size\": 12, \n \"color\": \"#000\", \n \"family\": \"sans-serif\"\n }, \n \"bgcolor\": \"#FFFFFF\", \n \"traceorder\": \"normal\", \n \"bordercolor\": \"#FFFFFF\", \n \"borderwidth\": 2\n }\n }\n fig = go.Figure(data = [trace1], layout = layout,)\n fig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#graphWaveform(trx, sr= trsr) # too much memory consumption, under your risk","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(20, 5))\nLD.waveplot(trx, sr=trsr)\nplt.xlabel(\"Time (s)\", fontsize=14)\nplt.ylabel(\"Amp []\", fontsize=14)\nplt.title(\"Audio Signal WaveForm - TRAIN SAMPLE\", fontsize=16)\nplt.show()\nipD.Audio(tr)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(20, 5))\nLD.waveplot(tsx, sr=tssr)\nplt.xlabel(\"Time (s)\", fontsize=14)\nplt.ylabel(\"Amp []\", fontsize=14)\nplt.title(\"Audio Signal WaveForm - TEST SAMPLE\", fontsize=16)\nplt.show()\nipD.Audio(ts)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2.6.2. Audio - Feature extraction"},{"metadata":{},"cell_type":"markdown","source":"# 2.6.2.1. Short Time Fourier Transform (STFT)\n\nSTFT is the same Fourier transform widely known, but using a window time to analyses that portion of time, then the process repeats to the whole signal in time, using a step time of course. So, in every little window time analysis we get a spectrum decomposition. This is a good approach to analyses signals, but what happen if the signal is not visible o poorly visible for our selected window time ... we change it right? ... yhea. But what happen if this change made us lost others signals? This is the limits of STFT in this kind of situation we need to use another kind of algorithm decomposition like wavelet. Don’t worry about it, it just a comment, but I hope that you get the idea or remember it.\n\nThis stft is best visualized by spectrogram plot. But what is a spectrogram? It's a spectrum of frequency of a signal, in more simples terms a one channel image.\n\nIf you want to know more about this topic check this video.\n\n\n\n\nOr in this one from 3Blue1Brown\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"def graph_specsSTFT(signal, sr, title,x_axis, y_axis, **paramSTFT):\n \"\"\"\n signal: vector representation of the sound or signal, samples over time\n sr: sample rate\n title: title of the spectrum plot\n x_axis : axis tipe for the espectrum representation\n y_axis: axis Y type for the spectrum, could be \"log\" or \"hz\"\n **paramSTFT: are the parameter takin for the stft from librosa library \n \"\"\"\n TRX = librosa.stft(signal, **paramSTFT)\n print(\"Shape of the stft: \", TRX.shape)\n # convert into db\n TRXdb = librosa.amplitude_to_db(abs(TRX))\n plt.figure(figsize=(14, 5))\n librosa.display.specshow(TRXdb, sr=sr, x_axis=x_axis, y_axis=y_axis)\n plt.title(title)\n plt.colorbar()\n plt.show()\n ","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"help(librosa.stft)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Parameters for the STFT\nn_fft= 1024 # n_fft/2+1 the numbers of bands to decompouse the frequencie spectrum\nwin_length=1024 # Windowing to analyse the signal\nhop_length=int(win_length/2) # step between sequential windows over time\nstft_param = {\"n_fft\":n_fft, \"win_length\" :win_length, \"hop_length\":hop_length }","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Spectrum for Train Sample"},{"metadata":{"trusted":true},"cell_type":"code","source":"graph_specsSTFT(trx,trsr,\"In Linear scale\", 'time', 'hz',**stft_param )\ngraph_specsSTFT(trx,trsr,\"In Log scale\", 'time', 'log',**stft_param )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Spectrum for Test Sample"},{"metadata":{"trusted":true},"cell_type":"code","source":"graph_specsSTFT(tsx,tssr,\"In Linear scale\", 'time', 'hz',**stft_param )\ngraph_specsSTFT(tsx,tssr,\"In Log scale\", 'time', 'log',**stft_param )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2.6.2.2. Mel-Frequency Cepstral Coefficients(MFCCs)\n\nMFCCs has been a hot topic in this kind of task, audio analysis, but widely used for human sound recognition (MEL), because this algorithm tries to imitate how the humans ear works. But what happen if the sound is so complex to be ear like a human does? in that case we have a problem, maybe we only need play with the STFT.\nIn any case, let’s see what is MFCC and how to implement.\n\nIf you want to see the original post so [check it out ](https://medium.com/prathena/the-dummys-guide-to-mfcc-aceab2450fd)\n\nLet’s begin by expanding the acronym MFCC — Mel Frequency Cepstral Co-efficients.\n\nEver heard the word cepstral before? Probably not. It’s spectral with the spec reversed! Why though? For a very basic understanding, cepstrum is the information of rate of change in spectral bands. In the conventional analysis of time signals, any periodic component (for eg, echoes) shows up as sharp peaks in the corresponding frequency spectrum (ie, Fourier spectrum. This is obtained by applying a Fourier transform on the time signal). This can be seen in the following image.\n\n\nOn taking the log of the magnitude of this Fourier spectrum, and then again taking the spectrum of this log by a cosine transformation (I know it sounds complicated, but bear with me please!), we observe a peak wherever there is a periodic element in the original time signal. Since we apply a transform on the frequency spectrum itself, the resulting spectrum is neither in the frequency domain nor in the time domain and hence Bogert et al. decided to call it the quefrency domain. And this spectrum of the log of the spectrum of the time signal was named cepstrum (ta-da!).\n\nThe following image is a summary of the above explained steps.\n\n\n> Cepstrum was first introduced to characterize the seismic echoes resulting due to earthquakes.\n\nPitch is one of the characteristics of a speech signal and is measured as the frequency of the signal. Mel scale is a scale that relates the perceived frequency of a tone to the actual measured frequency. It scales the frequency in order to match more closely what the human ear can hear (humans are better at identifying small changes in speech at lower frequencies). This scale has been derived from sets of experiments on human subjects. Let me give you an intuitive explanation of what the mel scale captures.\n\nThe range of human hearing is 20Hz to 20kHz. Imagine a tune at 300 Hz. This would sound something like the standard dialer tone of a land-line phone. Now imagine a tune at 400 Hz (a little higher pitched dialer tone). Now compare the distance between these two howsoever this may be perceived by your brain. Now imagine a 900 Hz signal (similar to a microphone feedback sound) and a 1kHz sound. The perceived distance between these two sounds may seem greater than the first two although the actual difference is the same (100Hz). The mel scale tries to capture such differences. A frequency measured in Hertz (f) can be converted to the Mel scale using the following formula :\n\n\nAny sound generated by humans is determined by the shape of their vocal tract (including tongue, teeth, etc). If this shape can be determined correctly, any sound produced can be accurately represented. The envelope of the time power spectrum of the speech signal is representative of the vocal tract and MFCC (which is nothing but the coefficients that make up the Mel-frequency cepstrum) accurately represents this envelope. The following block diagram is a step-wise summary of how we arrived at MFCCs:\n\n\n\n> Here, Filter Bank refers to the mel filters (coverting to mel scale) and Cepstral Coefficients are nothing but MFCCs."},{"metadata":{"_kg_hide-input":false,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"help(librosa.feature.mfcc)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Train Sample MFCCs"},{"metadata":{"trusted":true},"cell_type":"code","source":"mfccs = librosa.feature.mfcc(trx, sr=22000,n_mfcc=64)\nprint(mfccs.shape)\n#Displaying the MFCCs:\nplt.figure(figsize=(15, 7))\nLD.specshow(mfccs, sr=trsr, x_axis='time')\nplt.title(\"MFCCS vs Time - Train Sample\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Test Sample MFCCs"},{"metadata":{"trusted":true},"cell_type":"code","source":"mfccs = librosa.feature.mfcc(tsx, sr=tssr)\nprint(mfccs.shape)\n#Displaying the MFCCs:\nplt.figure(figsize=(15, 7))\nLD.specshow(mfccs, sr=tssr, x_axis='time')\nplt.title(\"MFCCS vs Time - Tet Sample\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Mel Spectogram"},{"metadata":{"_kg_hide-output":true,"trusted":true,"collapsed":true},"cell_type":"code","source":"help(librosa.feature.melspectrogram)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"melspec = librosa.feature.melspectrogram(trx, sr=trsr,n_fft =550, win_length = 550, hop_length=128)\nprint(melspec.shape)\nplt.figure(figsize=(10, 4))\nlibrosa.display.specshow(librosa.power_to_db(melspec, ref=np.max),\n y_axis='mel',\n x_axis='time')\nplt.colorbar(format='%+2.0f dB')\nplt.title('Mel spectrogram')\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3.1. Preprocessing DATA\n\nI decided to make a simple multi classifier deep learning model, which going to classify a clip audio file with length 5 seconds. For the submission every audio will turn into little chunks of the same length then they will be classify and finally merge the results for the whole audio sample."},{"metadata":{},"cell_type":"markdown","source":"# 3.1. Stratify Records\n\nThis is a classification problem, so we have to stratify the species to get better metrics."},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\ndf_train, df_test, _, _ = train_test_split(table_tp,\n table_tp[\"species_id\"], \n test_size=0.20,\n random_state =50,\n stratify= table_tp[\"species_id\"])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3.2. Augmented data\n\nWe have to create a class to achieve this task, TensorFlow has a simple way to create this class, let's seen below."},{"metadata":{"trusted":true},"cell_type":"code","source":"class DataAudioGenerator(tf.keras.utils.Sequence):\n def __init__(self, \n df_table,\n train_folder,\n batch_size,\n n_fft, # stft params\n win_length, # stft params\n hop_length, # stft params\n w_lenght = 5, # duration time for every clip audio\n stride = 3, # Only for test pourpouse, to cut audio into pises of w_lenght with stride stride jeje\n samplerate=44000, \n childrens=2, # number of synthetic audios for every original\n shuffle=False,\n num_classes=None, \n noise_var=0.0025,\n random_state=None,\n train_clip_folder = None,\n mode = \"TRAIN\"):\n \n assert batch_size%(childrens+1) == 0 , 'batch_size debe ser multiplo de (childrens+1)'\n self.batch_sizes = batch_size//(childrens+1)\n self.childrens = childrens\n self.stride = stride\n self.mode = mode\n self.train_folder = train_folder\n self.train_clip_folder = train_clip_folder\n self.w_lenght = w_lenght\n self.samplerate = samplerate\n self.n_fft = n_fft\n self.win_length = win_length \n self.hop_length = hop_length\n self.stft_param = {\"n_fft\":n_fft, \"win_length\" :win_length, \"hop_length\":hop_length }\n self.df_table = df_table\n self.random_state = random_state\n self.indices = df_table.index.tolist()\n self.labels = df_table[\"species_id\"] if mode == \"TRAIN\" else []\n self.num_classes = num_classes\n self.noise_var = noise_var\n self.shuffle = shuffle\n self.noise_v = [np.stack([random.uniform(-self.noise_var,self.noise_var) for _ in range(self.samplerate*self.w_lenght)]),\n np.stack([random.gauss(0,self.noise_var) for _ in range(self.samplerate*self.w_lenght)]),\n ]\n self.on_epoch_end()\n self.__info()\n \n def __info(self):\n print(\"Audio generator=====>\")\n print(\"We have {:4d} original audios - {}\".format(len(self.indices), self.mode))\n if self.mode == \"TRAIN\":\n print(\"We'll generate and add {:4d} synthetic audios - {}\".format(len(self.indices)*self.childrens, self.mode))\n \n def __len__(self):\n #batches per epochs\n return len(self.indices) // self.batch_sizes\n\n def __getitem__(self, index):\n self.batch_sizes = 1 if self.mode == \"TEST\" else self.batch_sizes\n index = self.index[index * self.batch_sizes:(index + 1) * self.batch_sizes]\n batch = [self.indices[k] for k in index]\n X, y = self.__get_data(batch)\n return X, y\n\n def on_epoch_end(self):\n self.index = np.arange(len(self.indices))\n if self.shuffle == True:\n np.random.shuffle(self.index)\n \n def __get_data(self, batch):\n X = []\n y = []\n #print(batch)\n if self.mode == \"TRAIN\":\n for i, id in enumerate(batch):\n y_yi = self.labels[id]\n hotencode_yi = keras.utils.to_categorical(y_yi, num_classes=self.num_classes, dtype='float32')\n signal = self.extract_audio_clip(id)\n spec_db = self.spec_transp_scal(signal)\n\n X.append(spec_db) # original X\n y.append(hotencode_yi) # original Y\n for i_aum in range(1,self.childrens+1):\n random_base = self.random_state + int(time.time())\n random.seed(random_base)\n signal = self.noise_signal(signal)\n spec_db = self.spec_transp_scal(signal)\n X.append(spec_db) # original X\n y.append(hotencode_yi) # original Y\n return np.reshape(np.stack(X), (int(self.batch_sizes*(self.childrens+1)), spec_db.shape[0], -1,1) ), np.stack(y) #\n elif self.mode == \"TEST\":\n for i, id in enumerate(batch):\n signal = self.extract_audio_clip(id)\n spec_db = np.stack([self.spec_transp_scal(signal_i) for signal_i in signal])\n return np.expand_dims(spec_db, axis = 3) , []\n \n \n \n def extract_audio_clip(self, index):\n if self.mode == \"TRAIN\":\n time_start = self.df_table.loc[index]['t_min']\n time_end = self.df_table.loc[index]['t_max']\n duration = time_end-time_start\n\n record_name = self.df_table.loc[index][\"recording_id\"] # train_clip_folder\n record_path = os.path.join(self.train_folder, record_name+\".flac\")\n #record_path_clip = os.path.join(self.train_clip_folder, record_name+ \"_\" + str(int(time_start*10))+\".flac\")\n signal, srate = librosa.load(record_path, \n sr = self.samplerate, \n mono = True,offset=time_start,\n duration = duration if duration > self.w_lenght else self.w_lenght)\n\n # try :\n # signal, srate = sf.read(record_path_clip)\n # self.samplerate = srate\n # return signal\n # except:\n # signal, srate = sf.read(record_path)\n # self.samplerate = srate\n\n #signal, srate = sf.read(record_path)\n self.samplerate = srate\n \n elif self.mode == \"TEST\":\n signal, srate = librosa.load(record_path, \n sr = self.samplerate, \n mono = True,)\n effective_length = self.samplerate * self.w_lenght\n stride = self.stride * self.samplerate\n signal = [signal[i:i+effective_length].astype(np.float32) for i in range(0, 60*self.samplerate+stride-effective_length, stride)]\n signal[-1] = signal[-1] if len(signal[-1]) == effective_length else np.pad(signal[-1], (0,effective_length-len(signal[-1])), constant_values = (0,0))\n return np.stack(signal)\n \n len_signal = len(signal)\n effective_length = self.samplerate * self.w_lenght\n time_start = 0 # time_start * self.samplerate\n time_end = duration * self.samplerate\n \n if len_signal > effective_length:\n # Positioning sound slice\n center = np.round((time_start + time_end) / 2)\n beginning = center - effective_length / 2\n if beginning < 0:\n beginning = 0\n beginning = np.random.randint(beginning, center) # random position between center and the biggining to introduce data augmentation\n ending = beginning + effective_length\n if ending > len_signal:\n ending = len_signal\n beginning = ending - effective_length\n signal = signal[beginning:ending].astype(np.float32)\n else:\n signal = signal.astype(np.float32)\n signal = signal if len(signal) == effective_length else np.pad(signal, (0,effective_length-len(signal)), constant_values = (0,0)) #padding or cut\n \n #sf.write(record_path_clip, signal, srate) \n \n return signal\n \n def spec_transp_scal(self, signal):\n signal = librosa.feature.melspectrogram(signal, sr=self.samplerate, **self.stft_param)# librosa.amplitude_to_db(abs(librosa.stft(signal, **self.stft_param)))\n signal = np.transpose(signal) # transpose because we going to reshape inside the model and use a LSTM to extract time features \n return (signal - np.min(signal))/(np.max(signal)-np.min(signal) ) # min max scale\n \n def noise_signal(self, signal):\n max_leng = len(signal)\n noise = random.randint(0,2)\n \n if noise == 0:\n signal = np.roll(signal,int(self.samplerate/random.randint(2,4)))+ self.noise_v[0]\n elif noise == 1:\n signal = np.roll(signal,int(self.samplerate/random.randint(5,10))) + self.noise_v[1]\n else: \n signal = np.roll(signal,int(self.samplerate/5))\n \n return signal\n ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def model_CRNN(input_shape_nn,n_clases=24, saved_file = None):\n initializer = tf.random_normal_initializer(0,0.02)\n reshape_time_len = int(input_shape_nn[0]/8)\n reshape_feature_len = int(input_shape_nn[1]/8)\n \n model = Sequential()\n model.add(Input(input_shape_nn ))\n\n model.add(Conv2D(32, (3, 3),padding='same', kernel_initializer = initializer ) )\n model.add(Activation('relu'))\n model.add(MaxPooling2D(pool_size=(2, 2)))\n\n model.add(Conv2D(64, (3, 3),padding='same', kernel_initializer = initializer,use_bias = False))\n model.add(BatchNormalization())\n model.add(Activation('relu'))\n model.add(MaxPooling2D(pool_size=(2, 2)))\n \n model.add(Conv2D(128, (3, 3),padding='same', kernel_initializer = initializer,use_bias = False ))\n model.add(BatchNormalization())\n model.add(Activation('relu'))\n model.add(MaxPooling2D(pool_size=(2, 2)))\n\n model.add(Reshape((reshape_time_len, 128*reshape_feature_len)))\n #model.add(Permute((2, 1)))\n model.add(Bidirectional(LSTM(128,return_sequences=False,dropout=0.2)))\n #model.add(LSTM(128,return_sequences=False,dropout=0.2 ))\n \n #model.add(Flatten())\n \n model.add(Dense(128, kernel_initializer = initializer))#input_shape=features.shape[1:]\n model.add(Activation('relu'))\n model.add(BatchNormalization())\n #model.add(Dropout(0.25))\n \n model.add(Dense(64, kernel_initializer = initializer))#input_shape=features.shape[1:]\n model.add(Activation('relu'))\n model.add(BatchNormalization())\n #model.add(Dropout(0.25))\n\n model.add(Dense(n_clases,kernel_initializer = initializer))\n model.add(Activation('softmax'))\n sgd = optimizers.SGD(lr=0.001, decay=1e-6, momentum=1e-5, nesterov=True)\n model.compile(loss='categorical_crossentropy',\n optimizer=sgd, #optimizers.Adam(lr=1e-4),\n metrics=['accuracy',tf.keras.metrics.PrecisionAtRecall(recall=0.8)])\n #tf.keras.utils.plot_model(model, to_file='NN_model.jpg', show_shapes=True)\n if (saved_file):\n try:\n #model.load_model(saved_file)\n model.load_weights(saved_file)\n print(\"Pesos cargados\")\n except:\n print(\"No se puede cargar los pesos\")\n model.summary()\n return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"param_generator_train = {\n \"df_table\": df_train,\n \"train_folder\":train_folder,\n \"train_clip_folder\": None,\n \"batch_size\":3,\n \"n_fft\":1100, # stft params\n \"win_length\":1100, # stft params\n \"hop_length\":512, # stft params\n \"samplerate\": 44000,\n \"w_lenght\" : 2, # Signal window length to analyze and then feed to our model\n \"childrens\":2, # Number of augmented audios for every original audio\n \"shuffle\":True,\n \"num_classes\":24, \n \"noise_var\":0.0025,\n \"random_state\":10\n}\nparam_generator_val = {\n \"df_table\": df_test,\n \"train_folder\":train_folder,\n \"train_clip_folder\": None,\n \"batch_size\":4,\n \"n_fft\":1100, # stft params\n \"win_length\":1100, # stft params\n \"hop_length\":512, # stft params\n \"samplerate\": 44000,\n \"w_lenght\" : 2, # Signal window length to analyze and then feed to our model\n \"childrens\":1, # Number of augmented audios for every original audio\n \"shuffle\":True,\n \"num_classes\":24, \n \"noise_var\":0.0025,\n \"random_state\":10\n}\nparam_test = {\n \"df_table\": table_submission,\n \"train_folder\":test_folder,\n \"batch_size\":4,\n \"n_fft\":1100, # stft params\n \"win_length\":1100, # stft params\n \"hop_length\":512, # stft params\n \"samplerate\": 44000,\n \"w_lenght\" : 2, # Signal window length to analyze and then feed to our model\n \"stride\" : 1,\n \"childrens\":1, # Number of augmented audios for every original audio\n \"shuffle\":True,\n \"num_classes\":24, \n \"noise_var\":0.0025,\n \"random_state\":10,\n \"mode\": \"TEST\",\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_datagen = DataAudioGenerator(**param_generator_train)\nval_datagen = DataAudioGenerator(**param_generator_val)\ntest_datagen = DataAudioGenerator(**param_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"input_shape_nn = train_datagen.__getitem__(1)[0].shape[1:]\nprint(\"Model Input size: {}\".format(input_shape_nn))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model Time\n\nIs time to train our simple model and see what we have\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"model = model_CRNN(input_shape_nn)\nn_epochs = 0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# %% \n# Training model using generators\nlogdir=\"logs\" \nepoch_add = 100\ntboard_callback = TensorBoard(log_dir=logdir)\nmodel_checkpoint = ModelCheckpoint('soundClassCRNNGEN{val_loss:.4f}acc_val{val_accuracy:.4f}_train{accuracy:.4f}.hdf5', monitor='val_loss',verbose=1, save_best_only=True) \nearlyStopping = EarlyStopping(monitor='val_loss',patience=100,min_delta=0)\n \nhistory = model.fit(train_datagen,\n #steps_per_epoch = 8, #\n #batch_size=16, #\n epochs=n_epochs+epoch_add,\n initial_epoch = n_epochs,\n callbacks=[tboard_callback,model_checkpoint,earlyStopping],\n validation_data = val_datagen,\n workers=-1)\nn_epochs=n_epochs+epoch_add ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
![](\"https://d13z1xw8270sfc.cloudfront.net/origin/320545/dsc_0081.jpg\")
\n![](\"http://d33wubrfki0l68.cloudfront.net/508a62f305652e6d9af853c65ab33ae9900ff38e/17a88/images/tasks/challenge2016/task3_overview.png\")
\n