{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Rainforest Connection Data Exploration 🦜\nHey, welcome to my notebook :)\n\nI'll be trying to extract useful information from the dataset and describing my thought process along the way.\nLet's get into it!\n\n*Note:* A lot of this information is new to me, and I'm always open to suggestions! I'll leave space at the top here to include where and who I've gathered some of my knowledge from, feel free to expand the section below if interested! Remember that open-source is a beautiful thing, so long as people share their knowledge AND cite their inspiration :)"},{"metadata":{},"cell_type":"markdown","source":"### Resources/Citations:\n* Valerio Velardo has an amazing series on YouTube regarding deep learning and audio analysis. Much of my inspiration comes from his tutorial videos, and he deserves the attention way more than me! Please check out his videos is you're interested in learning more: https://www.youtube.com/playlist?list=PL-wATfeyAMNrtbkCNsLcpoAyBBRJZVlnf"},{"metadata":{},"cell_type":"markdown","source":"## Importing Data and Loading Modules"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport librosa as lb # audio proccessing\nimport librosa.display # cool audio visuals\nimport matplotlib.pyplot as plt # to support librosa display\nimport IPython.display as ipd # for playing audio\n\nimport os\n\n# View files if interested\n# for dirname, _, filenames in os.walk('/kaggle/input'):\n#     for filename in filenames:\n#         print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"## Displaying a Waveform"},{"metadata":{"trusted":true},"cell_type":"code","source":"file = '/kaggle/input/rfcx-species-audio-detection/train/003b04435.flac'\nsignal, sr = lb.load(file, sr=22050) # load file into librosa with sample rate\nlb.display.waveplot(signal, sr=sr)\nplt.xlabel('Time')\nplt.ylabel('Amplitude')\nplt.show()\n\nipd.Audio(file) # to listen to the audio","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Isn't that neat?! This is the waveform of the audio in the time-domain. In it's current state it shows us the amplitude of the wave as it progresses over time, but not much more. In order to get more information on this audio, we have to apply some transformations.\n\nTo switch from the time-domain to frequency-domain, we can apply a Fast Fourier Transformation (FFT). For that, we'll use numpy."},{"metadata":{},"cell_type":"markdown","source":"## Fast Fourier Transform (FFT)"},{"metadata":{"trusted":true},"cell_type":"code","source":"fft = np.fft.fft(signal)\n\nmagnitude = np.abs(fft)\nfrequency = np.linspace(0, sr, len(magnitude))\nplt.plot(frequency, magnitude)\nplt.xlabel('Frequency')\nplt.ylabel('Magnitude')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So cool :)\n\nThis gives us the magnitude of all the frequencies in our audio, over the duration of the whole file. \n\nIf you have a keen eye, you might've noticed that this graph is symmetrical! This is due to a property of FFTs that I won't get into, but we can reduce this graph to only the needed information, as the second half of the graph is duplicated."},{"metadata":{"trusted":true},"cell_type":"code","source":"left_frequency = frequency[:int(len(frequency)/2)]\nleft_magnitude = magnitude[:int(len(frequency)/2)]\nplt.plot(left_frequency, left_magnitude)\nplt.xlabel('Frequency')\nplt.ylabel('Magnitude')\nplt.show()  ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There we go! You can see that the loudest sounds in this file are at ~2000Hz and ~4200Hz. Maybe this can help give us a better idea of what species are present? 🤔"},{"metadata":{},"cell_type":"markdown","source":"## Short Time Fourier Transforms (STFT)\nRemember that this is the magnitude of the frequencies throughout the **whole duration of the audio**. A more useful graph would show us what frequencies are present on a time axis. One idea to implement this would be to make a bunch of these frequency-domain graphs for short durations in the audio, and then combine them together to form a time axis. This is what Short Time Fourier Transforms (STFT) do, and the visuals we can produce with this are know as **spectrograms**."},{"metadata":{"trusted":true},"cell_type":"code","source":"n_fft = 2048 # number of samples per FFT (the duration of each slice)\nhop_length = 512 # shift\n\nstft = lb.core.stft(signal, hop_length=hop_length, n_fft=n_fft)\n\nspectrogram = np.abs(stft)\n\nlb.display.specshow(spectrogram, sr=sr, hop_length=hop_length)\nplt.xlabel('Time')\nplt.ylabel('Frequency')\nclb = plt.colorbar()\nclb.set_label('Amplitude')\nplt.show()  ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we see that frequency is mapped over time, with the color representing the amplitude of the frequency at that time. That's sorta cool... but I can't see much!\n\nLet's apply a logarithm to change our amplitude to decibels, hopefully we can see better afterwards..."},{"metadata":{"trusted":true},"cell_type":"code","source":"log_spectrogram = lb.amplitude_to_db(spectrogram)\n\nlb.display.specshow(log_spectrogram, sr=sr, hop_length=hop_length)\nplt.xlabel('Time')\nplt.ylabel('Frequency')\nclb = plt.colorbar()\nclb.set_label('Amplitude')\nplt.show()  ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Much better! As deduced by our frequency-domain graph earlier, we can see again where the loudest frequencies are present. But now we can see how that fluctuates over time, and how different frequencies may pop in or out. "},{"metadata":{},"cell_type":"markdown","source":"## Mel Frequency Cepstal Coefficients (MFCCs)\nThe last feature that is useful to extract is the Mel Frequency Cepstral Coefficients (MFCCs). This can give you information about the timbral/textural aspects of the audio, and approximate how the human auditory system interprets sound. This is especially useful in speech recognition, but could prove very important for this competition as well! "},{"metadata":{"trusted":true},"cell_type":"code","source":"MFFCs = lb.feature.mfcc(signal, n_fft=n_fft, hop_length=hop_length, n_mfcc=13) # 13 coefficients\n\nlb.display.specshow(MFFCs, sr=sr, hop_length=hop_length)\nplt.xlabel('Time')\nplt.ylabel('MFCC')\nclb = plt.colorbar()\nclb.set_label('Volume')\nplt.show()  ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Groovy!"},{"metadata":{},"cell_type":"markdown","source":"## Conclusion \nFor now, I won't get into how this transformation is performed or how it can be implemented with machine learning. If enough people are intersted, I'll continue this series, let me know!\n\nSee what you can find, and share your new-found knowledge with the community! Happy coding :)"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}