{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# ๐Ÿฆ Audio 101. 1- Audio manipulation & musical notes\n\n## [BirdCLEF 2022](https://www.kaggle.com/c/birdclef-2022)\n### Identify bird calls in soundscapes\n![](https://storage.googleapis.com/kaggle-competitions/kaggle/33246/logos/header.png)\n\n\n## Hi and welcome! This is the first kernel of the series `Audio 101`, the documentation of my learning process in the amazing world of audio processing.\n\n**In this short kernel we will go over some very basics abilities for working with audio. We will load the `.ogg` files with `torchaudio`, play them with `IPython.display.Audio` and get familiar manipulating waveforms. We will build a simple musical note generator and play a musical scale!.**\n\nThis series aims to get a good understanding of the specific topic from zero.\n\nThe ideal reader is a Data Scientist noob with some general knowledge about Deep Learning, but no technical expertise in Audio Processing. \n\n---\n\nThe full series consists of the following notebooks:\n1. _[๐Ÿฆ Audio 101. 1-Audio manipulation & musical notes](https://www.kaggle.com/julian3833/audio-101-1-audio-manipulation-musical-notes/) (This notebook)_\n2. [๐Ÿฆ Audio 101. 2- Detailed EDA](https://www.kaggle.com/julian3833/audio-101-2-detailed-eda/) \n\n\n\nThis is an ongoing project, so expect more notebooks to be added to the series soon. Actually, we are currently working on the following ones:\n* **Plot Fourier Transforms and Spectrograms**\n* **Build a simple CNN classifier model over image features**\n* **Study the previous competition [BirdCLEF 2021 - Birdcall Identification](https://www.kaggle.com/c/birdclef-2021) and migrate some good models**\n\n---\n\n# Intro\n\n\nI have started learning speech recognition some days ago.. Lucky me this competition started! An ongoing competition is always very engaging; and engagement helps with the learning process.\n\n\nThis is not an EDA of the competition itself but a small exploration of very simple concepts about handling audio data. We load, explore and analyze a file and, after discovering that an audio is just a 1D np.array we build some nice musical note generator function using `np.sin`. Finally, we will create and play a musical scale. \n\nIn case you are just learning audio processing with DL as I am, you can check the notebooks of my repo [speech-101](https://github.com/dataista0/speech-101/nbs). The first one has a good collection of resources to start diving into the \"Speech\" world (with previous Kaggle competitions on Audio data). The second one builds a neat speech recognizer that triggers commands on demand using `speechrecognizer` following a youtube tutorial:\n* [Day 1 - NB1 - Googling and finding relevant root resources.ipynb](https://github.com/dataista0/speech-101/blob/main/nbs/Day%201%20-%20NB1%20-%20Googling%20and%20finding%20relevant%20root%20resources.ipynb)\n* [Day 1 - NB2 - Simple speech recognizer with speechrecognition.ipynb](https://github.com/dataista0/speech-101/blob/main/nbs/Day%201%20-%20NB2%20-%20Simple%20speech%20recognizer%20with%20speechrecognition.ipynb)\n\nYou might find these resources that I gathered on Speech Recognition valuable as well:\n\n\n## Theoretical introductions: \n* Introduction to Deep Learning for Audio and Speech Applications - YouTube\n* Stanford Seminar - Deep Learning in Speech Recognition - YouTube\n* Speech Recognition - Wikipedia\n* Speech Processing - Wikipedia\n\n\n## Kaggle speech competitions\n\n* TensorFlow Speech Recognition Challenge | Kaggle\n* Rainforest Connection Species Audio Detection | Kaggle\n* BirdCLEF 2021 - Birdcall Identification | Kaggle\n\n\n## Other Kaggle resources\n* Speech representation and data exploration | Kaggle\n* End-to-end baseline TF Estimator LB 0.72 | Kaggle\n* Wav2vec+wandb- Learning audio representation ๐Ÿ”ฅ๐Ÿค— | Kaggle\n\n\n## Datasets, models and papers\n* Google AI Blog: Launching the Speech Commands Dataset\n* anton-l/wav2vec2-random-tiny-classifier ยท Hugging Face\n* Wav2Vec2\n* [2006.11477] wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations\n\n\n## More resources\n\n* 13. Speech Recognition with Convolutional Neural Networks in Keras/TensorFlow\n* Speech Recognition in Python\n* Sound: Crash Course Physics #18\n* Speech Recognition Using Python | How Speech Recognition Works In Python | Simplilearn\n* But what is the Fourier Transform? A visual introduction. 3blue 1 brown YT\n\n\n---\n\n\n\n# Please _DO_ upvote if you found this useful or interesting!\n\n\nEnough chitchat, let's code!","metadata":{}},{"cell_type":"markdown","source":"# Load a file with `torchaudio`","metadata":{}},{"cell_type":"code","source":"import torchaudio\nimport pandas as pd\n\na_file = \"../input/birdclef-2022/train_audio/akekee/XC174953.ogg\"","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-02-16T23:06:43.044485Z","iopub.execute_input":"2022-02-16T23:06:43.045438Z","iopub.status.idle":"2022-02-16T23:06:43.049223Z","shell.execute_reply.started":"2022-02-16T23:06:43.045392Z","shell.execute_reply":"2022-02-16T23:06:43.048666Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# It seems it's a common practice in the audio world to return this tuple of (waveform, sample_rate)\nwaveform, sample_rate = torchaudio.load(a_file)\n\n# It returns a tensor. Let's go to numpy\nwaveform = waveform.numpy()\n\n# Samples per second\nprint(sample_rate)","metadata":{"execution":{"iopub.status.busy":"2022-02-16T23:08:57.72196Z","iopub.execute_input":"2022-02-16T23:08:57.722716Z","iopub.status.idle":"2022-02-16T23:08:57.734882Z","shell.execute_reply.started":"2022-02-16T23:08:57.722677Z","shell.execute_reply":"2022-02-16T23:08:57.733635Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# See:\n* This awesome EDA: [๐Ÿฆ BirdCLEF 2022: EDA ๐Ÿฆ](https://www.kaggle.com/prokaggler/birdclef-2022-eda)\n* The documentation of [torchaudio](https://pytorch.org/audio/stable/torchaudio.html)","metadata":{}},{"cell_type":"code","source":"# Total samples (channels x samples) actually\nprint(waveform.shape)","metadata":{"execution":{"iopub.status.busy":"2022-02-16T23:08:25.669113Z","iopub.execute_input":"2022-02-16T23:08:25.669605Z","iopub.status.idle":"2022-02-16T23:08:25.673909Z","shell.execute_reply.started":"2022-02-16T23:08:25.669568Z","shell.execute_reply":"2022-02-16T23:08:25.673219Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Let's drop the first dimension and get into pandas world\nwaveform = pd.Series(waveform[0])\nwaveform.shape","metadata":{"execution":{"iopub.status.busy":"2022-02-16T23:09:00.57356Z","iopub.execute_input":"2022-02-16T23:09:00.574112Z","iopub.status.idle":"2022-02-16T23:09:00.581686Z","shell.execute_reply.started":"2022-02-16T23:09:00.574072Z","shell.execute_reply":"2022-02-16T23:09:00.580605Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Since we have a sample rate of 32000 and 76904 samples, this means that the audio lasts for 2.40325 seconds\nseconds = waveform.shape[0] / sample_rate\nseconds","metadata":{"execution":{"iopub.status.busy":"2022-02-16T23:10:46.739057Z","iopub.execute_input":"2022-02-16T23:10:46.73963Z","iopub.status.idle":"2022-02-16T23:10:46.745141Z","shell.execute_reply.started":"2022-02-16T23:10:46.739588Z","shell.execute_reply":"2022-02-16T23:10:46.744515Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# A sound is just a one-dimensional array!|\nwaveform.plot(figsize=(20, 5), alpha=0.5, color='red', title=\"First example\");","metadata":{"execution":{"iopub.status.busy":"2022-02-16T23:11:17.663746Z","iopub.execute_input":"2022-02-16T23:11:17.664891Z","iopub.status.idle":"2022-02-16T23:11:18.10127Z","shell.execute_reply.started":"2022-02-16T23:11:17.664846Z","shell.execute_reply":"2022-02-16T23:11:18.100155Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# It looks \"dense\" because it oscilates a big deal and it's very compresed, but it's just a simple line\nwaveform[10000:10200].plot(figsize=(20, 5), alpha=0.5, color='red', title=\"First example - Zoom-in in 100 samples\");","metadata":{"execution":{"iopub.status.busy":"2022-02-16T23:18:28.087907Z","iopub.execute_input":"2022-02-16T23:18:28.088443Z","iopub.status.idle":"2022-02-16T23:18:28.310886Z","shell.execute_reply.started":"2022-02-16T23:18:28.088387Z","shell.execute_reply":"2022-02-16T23:18:28.309986Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Since the sample rate is 32K, 400 samples is a sound that lasts 6.25 milliseconds\n200 / sample_rate","metadata":{"execution":{"iopub.status.busy":"2022-02-16T23:18:39.953155Z","iopub.execute_input":"2022-02-16T23:18:39.954175Z","iopub.status.idle":"2022-02-16T23:18:39.959694Z","shell.execute_reply.started":"2022-02-16T23:18:39.954124Z","shell.execute_reply":"2022-02-16T23:18:39.958816Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Using `torchaudio.info`","metadata":{}},{"cell_type":"code","source":"info = torchaudio.info(a_file)\n[attr for attr in dir(info) if not attr.startswith(\"_\")]","metadata":{"execution":{"iopub.status.busy":"2022-02-16T23:14:03.052146Z","iopub.execute_input":"2022-02-16T23:14:03.05324Z","iopub.status.idle":"2022-02-16T23:14:03.066778Z","shell.execute_reply.started":"2022-02-16T23:14:03.053167Z","shell.execute_reply":"2022-02-16T23:14:03.065595Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for attr in dir(info):\n if not attr.startswith(\"_\"):\n print(f\"{attr:<16}= {getattr(info, attr)}\")","metadata":{"execution":{"iopub.status.busy":"2022-02-16T23:20:10.764053Z","iopub.execute_input":"2022-02-16T23:20:10.764634Z","iopub.status.idle":"2022-02-16T23:20:10.770384Z","shell.execute_reply.started":"2022-02-16T23:20:10.764587Z","shell.execute_reply":"2022-02-16T23:20:10.769459Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Play the sound!","metadata":{}},{"cell_type":"code","source":"from IPython.display import Audio","metadata":{"execution":{"iopub.status.busy":"2022-02-16T23:24:38.519034Z","iopub.execute_input":"2022-02-16T23:24:38.51986Z","iopub.status.idle":"2022-02-16T23:24:38.524151Z","shell.execute_reply.started":"2022-02-16T23:24:38.519811Z","shell.execute_reply":"2022-02-16T23:24:38.523203Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ~2.4 seconds\nAudio(waveform, rate=sample_rate)","metadata":{"execution":{"iopub.status.busy":"2022-02-16T23:24:40.477068Z","iopub.execute_input":"2022-02-16T23:24:40.477578Z","iopub.status.idle":"2022-02-16T23:24:40.490204Z","shell.execute_reply.started":"2022-02-16T23:24:40.477529Z","shell.execute_reply":"2022-02-16T23:24:40.489252Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### I already knew that sound was a 1D array, but now that I see it as an `np.array` it feels great. It seems easy to manipulate as well...\n\nLet's try some stuff","metadata":{}},{"cell_type":"code","source":"import numpy as np\n\nrandom_sound = np.random.rand(sample_rate * 2) / 10 # 2 seconds \npd.Series(random_sound).plot();\nAudio(random_sound, rate=sample_rate)","metadata":{"execution":{"iopub.status.busy":"2022-02-17T00:18:56.676596Z","iopub.execute_input":"2022-02-17T00:18:56.677254Z","iopub.status.idle":"2022-02-17T00:18:58.927359Z","shell.execute_reply.started":"2022-02-17T00:18:56.677201Z","shell.execute_reply":"2022-02-17T00:18:58.926477Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"square_sound = np.ones(sample_rate * 2) / 10\nsquare_sound[16000:32000] = 0\nsquare_sound[32000+16000:] = 0\ndisplay(pd.Series(square_sound).plot())\nAudio(square_sound, rate=sample_rate)","metadata":{"execution":{"iopub.status.busy":"2022-02-16T23:29:08.139322Z","iopub.execute_input":"2022-02-16T23:29:08.139994Z","iopub.status.idle":"2022-02-16T23:29:08.327957Z","shell.execute_reply.started":"2022-02-16T23:29:08.139924Z","shell.execute_reply":"2022-02-16T23:29:08.327229Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Let's play a sin wave to \"synthetize\" sound\n\nThis is how a bare musical note can be created from the math world.","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nplt.rcParams[\"figure.figsize\"] = (20,5)","metadata":{"execution":{"iopub.status.busy":"2022-02-18T16:46:57.35029Z","iopub.execute_input":"2022-02-18T16:46:57.351577Z","iopub.status.idle":"2022-02-18T16:46:57.380773Z","shell.execute_reply.started":"2022-02-18T16:46:57.351455Z","shell.execute_reply":"2022-02-18T16:46:57.380087Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# One sin cycle happens between 0 and 2 pi\nx = np.linspace(0, 2*np.pi, 10000)\nsin = np.sin(x)\nplt.plot(x, sin);\n# Cannot hear it. What about you? :P\nAudio(sin, rate=sample_rate)","metadata":{"execution":{"iopub.status.busy":"2022-02-18T16:46:58.046627Z","iopub.execute_input":"2022-02-18T16:46:58.047384Z","iopub.status.idle":"2022-02-18T16:46:58.114585Z","shell.execute_reply.started":"2022-02-18T16:46:58.047347Z","shell.execute_reply":"2022-02-18T16:46:58.113493Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Increase cycle frequency\nx = np.linspace(0, 2*np.pi, sample_rate)\nsin = np.sin(10*x)\nplt.plot(x, sin);\n# Still cannot hear it, but it's below the human perception frequency range\nAudio(sin, rate=sample_rate)","metadata":{"execution":{"iopub.status.busy":"2022-02-17T00:03:18.760877Z","iopub.execute_input":"2022-02-17T00:03:18.76126Z","iopub.status.idle":"2022-02-17T00:03:19.024337Z","shell.execute_reply.started":"2022-02-17T00:03:18.761218Z","shell.execute_reply":"2022-02-17T00:03:19.023333Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The low-pitch/high-pitch spectre in which musical notes live are related to the frequency of the wave. The frequency is: how many cycles fit in one second?\n\n440 Hz means there are 440 cycles in one second\n\n![](https://149695847.v2.pressablecdn.com/wp-content/uploads/2021/07/image-293.png)\n\nLet's create musical notes with numpy!\n\n**References and useful resources**\n* [A (musical note)][1]\n* [E (musical note)][2]\n* [Online tone generator](https://www.szynalski.com/tone-generator/) - This is awesome but be careful with high pitches tones. \n\n[1]: https://en.wikipedia.org/wiki/A_(musical_note)\n[2]: https://en.wikipedia.org/wiki/E_(musical_note)","metadata":{}},{"cell_type":"code","source":"# Increase cycle frequency to 440 Hz, which is the musical note A\nx = np.linspace(0, 2*np.pi, 32000)\nsin = np.sin(440*x)\nplt.plot(x, sin);\n# Play it\nAudio(sin, rate=sample_rate)","metadata":{"execution":{"iopub.status.busy":"2022-02-18T16:47:45.21206Z","iopub.execute_input":"2022-02-18T16:47:45.21249Z","iopub.status.idle":"2022-02-18T16:47:45.227687Z","shell.execute_reply.started":"2022-02-18T16:47:45.212438Z","shell.execute_reply":"2022-02-18T16:47:45.226836Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Musical notes generator\n\nWe can add duration and volume and we have a musical note generator :D :D\n\nThe volume is the amplitude of the wave, so it's a multipler of the `sin` output.\nThe duration of the sound, on the other hand, affects both the `domain` space and the number of samples, so it affects both `linspace` arguments as a factor.","metadata":{}},{"cell_type":"code","source":"def get_note(frequency=440, volume=1, duration=2, sample_rate=32000, plot_wave=False, display_audio=False):\n x = np.linspace(0, 2*np.pi*duration, int(sample_rate*duration))\n sin = volume * np.sin(frequency * x)\n \n if plot_wave: \n plt.plot(x, sin)\n plt.show()\n \n if display_audio:\n display(Audio(sin, rate=sample_rate))\n return sin","metadata":{"execution":{"iopub.status.busy":"2022-02-17T00:08:16.141544Z","iopub.execute_input":"2022-02-17T00:08:16.141905Z","iopub.status.idle":"2022-02-17T00:08:16.148745Z","shell.execute_reply.started":"2022-02-17T00:08:16.141857Z","shell.execute_reply":"2022-02-17T00:08:16.147777Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# This is an A\nget_note(frequency=440, display_audio=True);","metadata":{"execution":{"iopub.status.busy":"2022-02-17T00:09:12.481092Z","iopub.execute_input":"2022-02-17T00:09:12.481647Z","iopub.status.idle":"2022-02-17T00:09:12.493218Z","shell.execute_reply.started":"2022-02-17T00:09:12.48159Z","shell.execute_reply":"2022-02-17T00:09:12.492451Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# This is an E\nget_note(frequency=329.63, display_audio=True);","metadata":{"execution":{"iopub.status.busy":"2022-02-17T00:09:52.760335Z","iopub.execute_input":"2022-02-17T00:09:52.760972Z","iopub.status.idle":"2022-02-17T00:09:52.772558Z","shell.execute_reply.started":"2022-02-17T00:09:52.760928Z","shell.execute_reply":"2022-02-17T00:09:52.771668Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Let's play a musical scale now","metadata":{}},{"cell_type":"code","source":"# Get a table with scales from a web page\ndef get_scale_notes():\n df_notes = pd.read_html(\"https://pages.mtu.edu/~suits/notefreqs.html\")[1]\n df_notes.columns = ['note', 'frequency', 'waveform']\n df_notes = df_notes[['note', 'frequency']]\n mask = (df_notes['note'].str.contains(\"4\") & ~df_notes['note'].str.contains(\"#\")) | (df_notes['note'] == 'C5')\n return df_notes[mask]\n\nscale = get_scale_notes()\nscale","metadata":{"execution":{"iopub.status.busy":"2022-02-17T00:15:59.015649Z","iopub.execute_input":"2022-02-17T00:15:59.016422Z","iopub.status.idle":"2022-02-17T00:15:59.348327Z","shell.execute_reply.started":"2022-02-17T00:15:59.016383Z","shell.execute_reply":"2022-02-17T00:15:59.347375Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"scale_sound = np.concatenate([get_note(frequency=frequency, duration=0.3) for frequency in scale['frequency'].tolist()])\n\n# Up!\nAudio(scale_sound, rate=sample_rate)","metadata":{"execution":{"iopub.status.busy":"2022-02-17T00:21:12.341346Z","iopub.execute_input":"2022-02-17T00:21:12.342003Z","iopub.status.idle":"2022-02-17T00:21:12.35659Z","shell.execute_reply.started":"2022-02-17T00:21:12.341954Z","shell.execute_reply":"2022-02-17T00:21:12.355638Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# And down!\nAudio(scale_sound[::-1], rate=sample_rate)","metadata":{"execution":{"iopub.status.busy":"2022-02-17T00:21:17.240736Z","iopub.execute_input":"2022-02-17T00:21:17.241524Z","iopub.status.idle":"2022-02-17T00:21:17.253628Z","shell.execute_reply.started":"2022-02-17T00:21:17.241471Z","shell.execute_reply":"2022-02-17T00:21:17.252714Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Please _DO_ upvote if you found this useful or interesting!\n\n# What's next?\nYou can check the next notebook of the series, [๐Ÿฆ Audio 101. 2- Detailed EDA](https://www.kaggle.com/julian3833/audio-101-2-detailed-eda/), in which we will do some EDA of the competition.","metadata":{}}]}