{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport warnings                        # To ignore any warnings\nwarnings.filterwarnings(\"ignore\")\n%matplotlib inline\n%pylab inline\nimport os\nimport pandas as pd\nimport librosa\nimport librosa.display\nimport IPython.display as ipd\nimport glob \nimport matplotlib.pyplot as plt\n%config InlineBackend.figure_format = 'retina'\n\nimport numpy as np\nimport pandas as pd\nimport wave\nfrom scipy.io import wavfile\nimport os\nimport librosa\nimport warnings\nfrom sklearn.utils import shuffle\nimport sklearn\nfrom tqdm import tqdm\n\nimport tensorflow as tf\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras import layers\nfrom tensorflow.keras import Input\nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras.callbacks import ModelCheckpoint, ReduceLROnPlateau, EarlyStopping\nfrom tensorflow.keras.utils import to_categorical\nfrom tensorflow.keras.layers import Dense, Flatten, Dropout, Activation, LSTM, SimpleRNN\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nsns.set()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Background & References\n\n**Problem Statement**\nThe basic problem statement boils down to using Audio Clips of bird calls to identify the bird speceis. What we will try, in this notebook is try to implement sequential learning in the form of LSTM for this classification task.\n\nThis is my first attempt at using LSTM for sequential learning from audio recordings, please feel free to post any suggestions or questions\n\nPS This is a work in progress, keep checking on this nb for future updates\n\n\n**Kernels Used for Reference**\n\nThe beauty of Kaggle is that a lot of work is already done, a lot of the work presented here borrows from the below Kernel:\n\n[My Chen's Kernel on Using LSTM for Heart Sound Analysis](https://www.kaggle.com/mychen76/heart-sounds-analysis-and-classification-with-lstm)\n\n[Francois Lemarchand's Notebook for this competition](https://www.kaggle.com/frlemarchand/bird-song-classification-using-an-lstm)\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## EDA\n\nTraining and Test Dataset summary:\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv('../input/birdsong-recognition/train.csv')\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Whoa, there are so many columns, it would be fun to explore, but before that, let us have a look at the test dataset to see if all these columns are actually available","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test = pd.read_csv('../input/birdsong-recognition/test.csv',)\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Unlike the test data, where we have multiple additional columns available, test column has only the audio clip, hence this is purely a Ornithological Language Processing Task ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Y variable - Ebird Code \n\nprint('Number of Unique Birds in the the Dataset is: ' + str(train_df['ebird_code'].nunique()))\n\n# Distribution of the labels\n\ntrain_df['ebird_code'].value_counts().plot.bar()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Great, an extremely legible label. We can see that about 50% of the birds have hundred recordings, the rest trailing off, with the minimum being a 100 recordings\n\nSince the test set is purely based on audio recordings, we will be focussing on audio features rather than additional EDA on the test columns","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Audio Processing\n\n**From Trigger Words Notebook (Sequence Model -Coursera)**\n\nWhat really is an audio recording? \n* A microphone records little variations in air pressure over time, and it is these little variations in air pressure that your ear also perceives as sound. \n* You can think of an audio recording is a long list of numbers measuring the little air pressure changes detected by the microphone. \n* We will use audio sampled at 44100 Hz (or 44100 Hertz). \n    * This means the microphone gives us 44,100 numbers per second. \n    * Thus, a 10 second audio clip is represented by 441,000 numbers (= $10 \\times 44,100$). \n\n#### Spectrogram\n* It is quite difficult to figure out from this \"raw\" representation of audio whether the word \"activate\" was said. \n* In  order to help your sequence model more easily learn to detect trigger words, we will compute a *spectrogram* of the audio. \n* The spectrogram tells us how much different frequencies are present in an audio clip at any moment in time. \n* If you've ever taken an advanced class on signal processing or on Fourier transforms:\n    * A spectrogram is computed by sliding a window over the raw audio signal, and calculating the most active frequencies in each window using a Fourier transform. \n    * If you don't understand the previous sentence, don't worry about it.\n\nLet's look at an example. \n   ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Play the firt clip for an aldfly\naldfly = '../input/birdsong-recognition/train_audio/aldfly/XC134874.mp3'\ny,sr = librosa.load(aldfly, sr=None)\nipd.Audio(aldfly) \n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Visualising the Audio**\n\nAudio Clips can be visualised as a sequence of waves, with Amplitudes, crests and troughs using librosa\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(14, 5))\nlibrosa.display.waveplot(y,sr = sr)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Spectogram**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"Y = librosa.stft(y)\nXdb = librosa.amplitude_to_db(abs(Y))\nplt.figure(figsize=(14, 5))\nlibrosa.display.specshow(Xdb, sr=sr, x_axis='time', y_axis='hz')\nplt.colorbar()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The color in the spectrogram shows the degree to which different frequencies are present (loud) in the audio at different points in time.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# LSTM","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Before we delve into the task of using LSTM for the task, we need to understand why LSTM works here\n\nAn easier analogy would be understanding how Echo uses LSTM to understand your voice command, we can later apply the same philosophy here.\n\nWhen you use a regular ANN, every input is in iteself an isolated data point (following the definition of IID). Say you are building a Churn risk model, two customers would be independently evaluated on their Churn Risk, outcome of one doesn't affect the other at all. Or in other words, they have no memory of the previous outcome while evaluating the new outcome.\n\nBut by using LSTMs, we can harness the sequence and order into data and use it while making classifications. \n\n**Will Add more later**\n\n[Andrew Ng explaining LSTM](https://www.youtube.com/watch?v=5wh4HWWfZIY)\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#Using Francois's code to extract the data/ run model\n\n\ndef get_sample(filename, bird, samples_df):\n    wave_data, wave_rate = librosa.load(filename)\n    data_point_per_second = 10\n    \n    #Take 10 data points every second\n    prepared_sample = wave_data[0::int(wave_rate/data_point_per_second)]\n    #We normalize each sample before extracting 5s samples from it\n    normalized_sample = sklearn.preprocessing.minmax_scale(prepared_sample, axis=0)\n    \n    #only take 5s samples and add them to the dataframe\n    song_sample = []\n    sample_length = 5*data_point_per_second\n    for idx in range(0,len(normalized_sample),sample_length): \n        song_sample = normalized_sample[idx:idx+sample_length]\n        if len(song_sample)>=sample_length:\n            samples_df = samples_df.append({\"song_sample\":np.asarray(song_sample).astype(np.float32),\n                                            \"bird\":ebird_to_id[bird]}, \n                                           ignore_index=True)\n    return samples_df\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"birds_selected = shuffle(train_df[\"ebird_code\"].unique())\ntrain_df = train_df.query(\"ebird_code in @birds_selected\")\n\nebird_to_id = {}\nid_to_ebird = {}\nebird_to_id[\"nocall\"] = 0\nid_to_ebird[0] = \"nocall\"\nfor idx, unique_ebird_code in enumerate(train_df.ebird_code.unique()):\n    ebird_to_id[unique_ebird_code] = str(idx+1)\n    id_to_ebird[idx+1] = str(unique_ebird_code)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nwarnings.filterwarnings(\"ignore\")\nsamples_df = pd.DataFrame(columns=[\"song_sample\",\"bird\"])\n\n#We limit the number of audio files being sampled to 5000 in this notebook to save time\n#However, we have already limited the number of bird species\nsample_limit = 5000\nwith tqdm(total=sample_limit) as pbar:\n    for idx, row in train_df[:sample_limit].iterrows():\n        pbar.update(1)\n        audio_file_path = \"/kaggle/input/birdsong-recognition/train_audio/\"\n        audio_file_path += row.ebird_code\n        samples_df = get_sample('{}/{}'.format(audio_file_path, row.filename), row.ebird_code, samples_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"samples_df = shuffle(samples_df)\nsamples_df[:10]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sequence_length = 50\ntraining_percentage = 0.9\ntraining_item_count = int(len(samples_df)*training_percentage)\nvalidation_item_count = len(samples_df)-int(len(samples_df)*training_percentage)\ntraining_df = samples_df[:training_item_count]\nvalidation_df = samples_df[training_item_count:]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Define the Neural Network Model\n\nIf you are new to Keras, please refer to this [Keras Tutorial](https://machinelearningmastery.com/tutorial-first-neural-network-python-keras/)\n\nThis is a multiclass classification problem, therefore we will be using the following architecture:\n\nOutput Layer: Softmax with 264 outputs (This will output 264 Probabilities, corresponding to each of the 264 birds\n\nOptimiser: [Adam](https://machinelearningmastery.com/adam-optimization-algorithm-for-deep-learning/)\n\nLoss: [Categorical Cross Entropy](https://peltarion.com/knowledge-center/documentation/modeling-view/build-an-ai-model/loss-functions/categorical-crossentropy)\n\nMetrics: Accuracy ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#Base Model, lots of room for improvement\n\n\nmodel = Sequential()\nmodel.add(LSTM(32, return_sequences=True, recurrent_dropout=0.2,input_shape=(None, sequence_length)))\nmodel.add(LSTM(32,recurrent_dropout=0.2))\nmodel.add(Dense(128,activation = 'relu'))\nmodel.add(Dropout(0.3))\nmodel.add(Dense(128,activation = 'relu'))\nmodel.add(Dropout(0.3))\nmodel.add(Dense(len(ebird_to_id.keys()), activation=\"softmax\"))\n\nmodel.summary()\n\ncallbacks = [ReduceLROnPlateau(monitor='val_loss', patience=2, verbose=1, factor=0.7),\n             EarlyStopping(monitor='val_loss', patience=10),\n             ModelCheckpoint(filepath='best_model.h5', monitor='val_loss', save_best_only=True)]\nmodel.compile(loss=\"categorical_crossentropy\", optimizer='adam')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train = np.asarray(np.reshape(np.asarray([np.asarray(x) for x in training_df[\"song_sample\"]]),(training_item_count,1,sequence_length))).astype(np.float32)\ngroundtruth = np.asarray([np.asarray(x) for x in training_df[\"bird\"]]).astype(np.float32)\nY_train = to_categorical(\n                groundtruth, num_classes=len(ebird_to_id.keys()), dtype='float32'\n            )\n\n\nX_validation = np.asarray(np.reshape(np.asarray([np.asarray(x) for x in validation_df[\"song_sample\"]]),(validation_item_count,1,sequence_length))).astype(np.float32)\nvalidation_groundtruth = np.asarray([np.asarray(x) for x in validation_df[\"bird\"]]).astype(np.float32)\nY_validation = to_categorical(\n                validation_groundtruth, num_classes=len(ebird_to_id.keys()), dtype='float32'\n            )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history = model.fit(X_train, Y_train, \n          epochs = 100, \n          batch_size = 32, \n          validation_data=(X_validation, Y_validation), \n          callbacks=callbacks)\n\nplt.plot(history.history['loss'])\nplt.plot(history.history['val_loss'])\nplt.title('Loss over epochs')\nplt.ylabel('Loss')\nplt.xlabel('Epoch')\nplt.legend(['Train', 'Validation'], loc='best')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Making Predictions","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"model.load_weights(\"best_model.h5\")\n\ndef predict_submission(df, audio_file_path):\n        \n    loaded_audio_sample = []\n    previous_filename = \"\"\n    data_point_per_second = 10\n    sample_length = 5*data_point_per_second\n    wave_data = []\n    wave_rate = None\n    \n    for idx,row in df.iterrows():\n        if previous_filename == \"\" or previous_filename!=row.filename:\n            filename = '{}/{}.mp3'.format(audio_file_path, row.filename)\n            wave_data, wave_rate = librosa.load(filename)\n            sample = wave_data[0::int(wave_rate/data_point_per_second)]\n        previous_filename = row.filename\n        \n        #basically allows to check if we are running the examples or the test set.\n        if \"site\" in df.columns:\n            if row.site==\"site_1\" or row.site==\"site_2\":\n                song_sample = np.array(sample[int(row.seconds-5)*data_point_per_second:int(row.seconds)*data_point_per_second])\n            elif row.site==\"site_3\":\n                #for now, I only take the first 5s of the samples from site_3 as they are groundtruthed at file level\n                song_sample = np.array(sample[0:sample_length])\n        else:\n            #same as the first condition but I isolated it for later and it is for the example file\n            song_sample = np.array(sample[int(row.seconds-5)*data_point_per_second:int(row.seconds)*data_point_per_second])\n\n        input_data = np.reshape(np.asarray([song_sample]),(1,sequence_length)).astype(np.float32)\n        prediction = model.predict(np.array([input_data]))\n        predicted_bird = id_to_ebird[np.argmax(prediction)]\n\n        df.at[idx,\"birds\"] = predicted_bird\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"audio_file_path = \"/kaggle/input/birdsong-recognition/example_test_audio\"\nexample_df = pd.read_csv(\"/kaggle/input/birdsong-recognition/example_test_audio_summary.csv\")\nexample_df[\"filename\"] = [ \"BLKFR-10-CPL_20190611_093000.pt540\" if filename==\"BLKFR-10-CPL\" else \"ORANGE-7-CAP_20190606_093000.pt623\" for filename in example_df[\"filename\"]]\n\n\nif os.path.exists(audio_file_path):\n    example_df = predict_submission(example_df, audio_file_path)\nexample_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"audio_file_path = \"/kaggle/input/birdsong-recognition/test_audio/\"\ntest_df = pd.read_csv(\"/kaggle/input/birdsong-recognition/test.csv\")\nsubmission_df = pd.read_csv(\"/kaggle/input/birdsong-recognition/sample_submission.csv\")\n\nif os.path.exists(audio_file_path):\n    submission_df = predict_submission(test_df, audio_file_path)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_df[[\"row_id\",\"birds\"]].to_csv('submission.csv', index=False)\nsubmission_df.head()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}