{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
\n# Introduction \n* In this kernel, we will learn the basics of sound theory, how to use **librosa**, sound waves and training neural networks with sound.\n\n\n\n* We will use [Freesound General-Purpose Audio Tagging Challenge](https://www.kaggle.com/c/freesound-audio-tagging) dataest for this kernel.\n\n\n \n### Thanks to This Great Channel: https://www.youtube.com/channel/UCZPFjMe1uRSirmSpznqvJfQ \n \n## Content\n* [Imports](#1)\n* [What is Sound](#2)\n * [What is Waveform](#2.1)\n * [Simple Sine Wave](#2.2)\n * [Coding Simple Sine Wave](#2.21)\n * [Pitch](#2.3)\n * [MIDI Notes](#2.31)\n * [Coding Mapping Pitch to Frequency](#2.32)\n * [Complex Waves (Multiple sine waves)](#2.4)\n * [Coding Complex Waves](#2.41)\n* [Analyzing Sound Data](#3)\n * [Converting Analog Signal to Digital Signal](#3.1)\n * [Preparing Files](#3.2)\n * [Analyzing](#3.3)\n * [Signal Domains](#3.4)\n * [Fast Fourier Transform (FFT)](#3.41)\n * [Spectrogram](#3.42)\n * [MFCCs](#3.5)\n* [Deep Learning with Audio](#4)\n * [Preparing Data](#4.1)\n * [ANN](#4.2)\n * [Results](#4.21)\n * [CNN](#4.3)\n * [Results](#4.31)\n* [Conclusion](#5)","metadata":{}},{"cell_type":"markdown","source":"
\n# Imports\n[Back to Introduction](#0)\n\n* **librosa** is a python package for music and audio analysis. It provides the building blocks necessary to create music information retrieval systems.\n\n","metadata":{}},{"cell_type":"code","source":"# data analysis and visualization\nimport numpy as np \nimport pandas as pd \nimport librosa \nimport librosa.display\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport IPython\nimport IPython.display as ipd\nfrom scipy.io import wavfile\nimport glob\n\n# keras and scikitlearn/deep learning\nfrom sklearn.model_selection import train_test_split\nimport keras\nfrom keras.layers import Dense, Dropout, Input, ReLU, LeakyReLU, BatchNormalization\nfrom keras.models import Model, Sequential\nfrom keras.optimizers import Adam\nfrom keras.utils.np_utils import to_categorical\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-10-07T16:52:09.286718Z","iopub.execute_input":"2021-10-07T16:52:09.287269Z","iopub.status.idle":"2021-10-07T16:52:55.620307Z","shell.execute_reply.started":"2021-10-07T16:52:09.287233Z","shell.execute_reply":"2021-10-07T16:52:55.619459Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n# What is Sound\n[Back to Introduction](#0)\n\n* In physics, sound is a vibration that propagates as an acoustic wave, through a transmission medium such as a gas, liquid or solid. [**Reference**](https://en.wikipedia.org/wiki/Sound)\n\n* When we are talking we are generating sound waves and these waves have **compressions** and **rarefactions**.\n\n\n\n
\n## What is Waveform\n[Back to Introduction](#0)\n\n* We will learn 2 types of waveforms: **Simple(single sinewaves)** and **Complex(multiples sinewaves)**.\n\n
\n## Simple Sine Wave\n[Back to Introduction](#0)\n\n* Sine wave is the simplest form of wave.\n\n\n\n* **Let's look at the features of sine wave:**\n\n\n1. **Period:** Period is the distnace between 2 peaks. Showing with **T**.\n1. **Frequency:** Frequency is the reverse of period (1/T [T is period] ). Expressed in **Hz**. Showing with **f**.\n1. **Amplitude:** How high perturbation in air pressure goes. If it is higher then amplitude is higher too. Showing with **A**. **Note:** Larger amplitude = louder \n1. **Phase:** The first peak of waves crest's time. Showing with **φ**.\n\n\n\n* **Frequency and amplitude comparisons:**\n\n\n\n \n* **Formula of a sine wave:**\n\n","metadata":{}},{"cell_type":"markdown","source":"
\n### Coding Simple Sine Waves\n[Back to Introduction](#0)","metadata":{}},{"cell_type":"code","source":"def sine_wave(amp, freq, time, phase):\n \n y_list = []\n for t in (time):\n y = amp * np.sin(2 * np.pi * freq * t + phase)\n y_list.append(y)\n \n return np.array(y_list)\n\nwave = sine_wave(3, .5, np.arange(0,6, 0.0001), 0)\n\nplt.figure(figsize = (15,8))\nplt.plot(wave, label = \"Sine Wave\")\nplt.legend()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-10-07T16:52:55.623627Z","iopub.execute_input":"2021-10-07T16:52:55.624032Z","iopub.status.idle":"2021-10-07T16:52:56.031086Z","shell.execute_reply.started":"2021-10-07T16:52:55.623987Z","shell.execute_reply":"2021-10-07T16:52:56.030328Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* **Not that necessary, but here is how cosine wave look like:**","metadata":{}},{"cell_type":"code","source":"def cos_wave(amp, freq, time, phase):\n \n y_list = []\n for t in (time):\n y = amp * np.cos(2 * np.pi * freq * t + phase)\n y_list.append(y)\n \n return np.array(y_list)\n\ns_wave = sine_wave(3, 0.5, np.arange(0,6, 0.0001), 0)\nc_wave = cos_wave(3, 0.5, np.arange(0,6, 0.0001), 0) \n\nplt.figure(figsize = (15,8))\nplt.plot(s_wave, label = \"Sine Wave\")\nplt.plot(c_wave, label = \"Cosine Wave\")\nplt.text(100, c_wave[0], \"Cos\", fontsize=15, color = \"orange\")\nplt.text(100, s_wave[0], \"Sin\", fontsize=15, color = \"blue\")\nplt.legend()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-10-07T16:52:56.032082Z","iopub.execute_input":"2021-10-07T16:52:56.032456Z","iopub.status.idle":"2021-10-07T16:52:56.664992Z","shell.execute_reply.started":"2021-10-07T16:52:56.032420Z","shell.execute_reply":"2021-10-07T16:52:56.664285Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n## Pitch\n[Back to Introduction](#0)\n\n* We do not perceive frequency in a linear way, we percieve frequency in a **logarithmic** way.\n\n
\n### MIDI Notes\n[Back to Introduction](#0)\n\n* Before pitch we have to understand **MIDI Notes**. MIDI is a way to represent notes as numbers. \n\n\n\n\n* If you look at the keyboard you can see a pattern that repeats itself. That pattern named **octave**.\n\n\n\n* In an ocatve there are **12 notes**. In every octave (12 notes) **frequency doubles**. \n\n\n\n* **Mapping pitch to frequency formula:**\n\n","metadata":{}},{"cell_type":"markdown","source":"
\n### Coding Mapping Pitch to Frequency\n[Back to Introduction](#0)\n","metadata":{}},{"cell_type":"code","source":"def pitch_to_frequency(p):\n \n freqs = []\n \n for i in p:\n f = 2**(((i-69)/12))*440\n freqs.append(f)\n \n return freqs\n\npitches = np.arange(21, 105, 0.001) # 21, 33, 45, 57, 69, 81, 93, 105\nfreqs = pitch_to_frequency(pitches)\n\nplt.figure(figsize = (15,8))\nplt.plot(freqs, pitches)\nplt.grid(alpha = 0.6)\nplt.ylabel(\"Pitch\")\nplt.xlabel(\"Frequency\")\nplt.title(\"Mapping Pitch to Frequency\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-10-07T16:52:56.665933Z","iopub.execute_input":"2021-10-07T16:52:56.666321Z","iopub.status.idle":"2021-10-07T16:52:56.972294Z","shell.execute_reply.started":"2021-10-07T16:52:56.666290Z","shell.execute_reply":"2021-10-07T16:52:56.971587Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n## Complex Waves (Multiple sine waves)\n[Back to Introduction](#0)\n\n* Complex waves are made of multiple waves. \n\n\n\n* In complex waves if peaks overlap in new wave the amplitude increases. If a peak and a dip overlap then they cancel eachother. \n\n\n\n* **Formula of multiple sine waves:**\n\n","metadata":{}},{"cell_type":"markdown","source":"
\n### Coding Complex Waves\n[Back to Introduction](#0)","metadata":{}},{"cell_type":"code","source":"s1 = sine_wave(2,0.8, np.arange(0,6,0.0001), 0)\ns2 = sine_wave(2,2, np.arange(0,6,0.0001), 0)\ns3 = sine_wave(2,4, np.arange(0,6,0.0001), 0)\n\ns1, s2, s3 = np.array(s1), np.array(s2), np.array(s3) \n\nplt.figure(figsize = (15,8))\n\nax1 = plt.subplot2grid((2, 3), (0, 0), colspan=1)\nax2 = plt.subplot2grid((2, 3), (0, 1), colspan=1)\nax3 = plt.subplot2grid((2, 3), (0, 2), colspan=1)\nax4 = plt.subplot2grid((2, 3), (1, 0), colspan=3)\n\nax4.plot(s1 + s2 + s3, color = \"red\")\nax4.set_title(\"Waves Added (Complex wave)\")\n\nax1.plot(s1)\nax1.set_title(\"Wave 1\")\n\nax2.plot(s2, color = \"orange\")\nax2.set_title(\"Wave 2\")\n\nax3.plot(s3, color = \"green\")\nax3.set_title(\"Wave 3\")\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-10-07T16:52:56.974449Z","iopub.execute_input":"2021-10-07T16:52:56.974856Z","iopub.status.idle":"2021-10-07T16:52:58.040126Z","shell.execute_reply.started":"2021-10-07T16:52:56.974826Z","shell.execute_reply":"2021-10-07T16:52:58.038996Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* **Canceling Waves:**","metadata":{}},{"cell_type":"code","source":"s1 = sine_wave(2,0.8, np.arange(0,6,0.0001), 0)\ns2 = sine_wave(2,0.8, np.arange(0,6,0.0001), 0)\n\ns2 = np.array(s2)\n \nreverser = np.full(\n shape=len(s2),\n fill_value=-1,\n dtype=np.int)\n\ns2 = s2*reverser\n\nplt.figure(figsize = (15,6))\nplt.plot(s1, alpha = 0.4)\nplt.plot(s2, alpha = 0.4)\nplt.plot(s2 + s1, color = \"black\")\nplt.title(\"Canceling Waves\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-10-07T16:52:58.042307Z","iopub.execute_input":"2021-10-07T16:52:58.042582Z","iopub.status.idle":"2021-10-07T16:52:58.560667Z","shell.execute_reply.started":"2021-10-07T16:52:58.042555Z","shell.execute_reply":"2021-10-07T16:52:58.559901Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n# Analyzing Sound Data\n[Back to Introduction](#0)\n\n
\n## Converting Analog Signal to Digital Signal\n[Back to Introduction](#0)\n\n* In real world sound is **continuous**. But we cannot convert continuous things to digital because to do that we need infinite memory. So store analog signals as digital signals we are using a technique called **sampling**.\n\n\n\n* In sampling there is a term named **sampling rate**. Since analog waves are continuous we cannot get every sample of a analog wave (because there are infinite samples). So while converting them to digital we are taking samples from that wave using a certain period. You can define that period yourself but there is a common sampling rate: **44100 Hz**. If you go up or down you will face with deformations. In audio processing you can use sampling rates like 22050 or 16000 Hz. ","metadata":{}},{"cell_type":"markdown","source":"
\n## Preparing Files\n[Back to Introduction](#0)\n\n* In the dataset we have a csv file that shows files' labels(wich instrument) we will use that csv file for loading a spesific instrument's files.","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv(\"../input/freesound-audio-tagging/train.csv\")\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2021-10-07T16:52:58.561998Z","iopub.execute_input":"2021-10-07T16:52:58.562309Z","iopub.status.idle":"2021-10-07T16:52:58.746284Z","shell.execute_reply.started":"2021-10-07T16:52:58.562278Z","shell.execute_reply":"2021-10-07T16:52:58.745277Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* We will use **Saxophone** as a example instrument.","metadata":{}},{"cell_type":"code","source":"df_saxophone = df[df[\"label\"] == \"Saxophone\"]\ndf_saxophone.head()","metadata":{"execution":{"iopub.status.busy":"2021-10-07T16:52:58.747412Z","iopub.execute_input":"2021-10-07T16:52:58.747674Z","iopub.status.idle":"2021-10-07T16:52:58.763926Z","shell.execute_reply.started":"2021-10-07T16:52:58.747648Z","shell.execute_reply":"2021-10-07T16:52:58.763007Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Loading files.","metadata":{}},{"cell_type":"code","source":"files = []\n\nfor path in df_saxophone[\"fname\"]: # we will use saxophone samples\n file, _ = librosa.core.load(\"../input/freesound-audio-tagging/audio_train/\" + str(path), sr=44100) # 44100Hz in dataset\n files.append(file)","metadata":{"execution":{"iopub.status.busy":"2021-10-07T16:52:58.765227Z","iopub.execute_input":"2021-10-07T16:52:58.765483Z","iopub.status.idle":"2021-10-07T16:53:03.551395Z","shell.execute_reply.started":"2021-10-07T16:52:58.765458Z","shell.execute_reply":"2021-10-07T16:53:03.550446Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample = files[3] # the smaple file we will use","metadata":{"execution":{"iopub.status.busy":"2021-10-07T16:53:03.552717Z","iopub.execute_input":"2021-10-07T16:53:03.552975Z","iopub.status.idle":"2021-10-07T16:53:03.557161Z","shell.execute_reply.started":"2021-10-07T16:53:03.552949Z","shell.execute_reply":"2021-10-07T16:53:03.556168Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n## Analyzing\n[Back to Introduction](#0)\n\n* Let's listen to our sample.","metadata":{}},{"cell_type":"code","source":"IPython.display.display(ipd.Audio(data = sample, rate = 44100)) # the original sample rate of recording","metadata":{"execution":{"iopub.status.busy":"2021-10-07T16:53:03.558586Z","iopub.execute_input":"2021-10-07T16:53:03.558995Z","iopub.status.idle":"2021-10-07T16:53:03.591158Z","shell.execute_reply.started":"2021-10-07T16:53:03.558953Z","shell.execute_reply":"2021-10-07T16:53:03.589962Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"reverser = np.full(\n shape=len(sample),\n fill_value=-1,\n dtype=np.int)\n\n\nsample_xd = sample[::-1]\n\nIPython.display.display(ipd.Audio(data = sample_xd, rate = 44100))","metadata":{"execution":{"iopub.status.busy":"2021-10-07T16:53:03.592483Z","iopub.execute_input":"2021-10-07T16:53:03.592754Z","iopub.status.idle":"2021-10-07T16:53:03.615403Z","shell.execute_reply.started":"2021-10-07T16:53:03.592727Z","shell.execute_reply":"2021-10-07T16:53:03.614272Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize = (15, 8))\nplt.plot(sample_xd)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-10-07T16:53:03.616951Z","iopub.execute_input":"2021-10-07T16:53:03.617360Z","iopub.status.idle":"2021-10-07T16:53:03.899270Z","shell.execute_reply.started":"2021-10-07T16:53:03.617322Z","shell.execute_reply":"2021-10-07T16:53:03.898399Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Let's play with the **sample rate** and see what is happening.","metadata":{}},{"cell_type":"code","source":"sample_rates = [60000, 22050, 16000, 10000]\n\nfor sr in sample_rates:\n IPython.display.display(ipd.Audio(data = sample, rate = sr))","metadata":{"execution":{"iopub.status.busy":"2021-10-07T16:53:03.900764Z","iopub.execute_input":"2021-10-07T16:53:03.901177Z","iopub.status.idle":"2021-10-07T16:53:03.963431Z","shell.execute_reply.started":"2021-10-07T16:53:03.901132Z","shell.execute_reply":"2021-10-07T16:53:03.962454Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* As you can see our sample file's playing speed and time length changed.\n* **Note:** Normally if sample rate is higher we should get a smoother result. But in this example our file recorded in 44100 Hz so when we change it we are facing with deformation (normally there is no sample rate limit in nature).","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize = (15, 8))\nplt.plot(sample)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-10-07T16:53:03.964749Z","iopub.execute_input":"2021-10-07T16:53:03.965027Z","iopub.status.idle":"2021-10-07T16:53:04.232064Z","shell.execute_reply.started":"2021-10-07T16:53:03.965000Z","shell.execute_reply":"2021-10-07T16:53:04.231355Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* **Zooming into the sample:**","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize = (20, 8))\nplt.plot(sample[:500])\nplt.plot(sample[:500], \".\", color = \"red\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-10-07T16:53:04.233163Z","iopub.execute_input":"2021-10-07T16:53:04.233593Z","iopub.status.idle":"2021-10-07T16:53:04.415913Z","shell.execute_reply.started":"2021-10-07T16:53:04.233549Z","shell.execute_reply":"2021-10-07T16:53:04.415165Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* As you can see this is a **complex wave**.","metadata":{}},{"cell_type":"markdown","source":"
\n## Signal Domains\n[Back to Introduction](#0)\n\n* There are 3 different domains for signal.\n\n\n1. **Time Domain:** We saw that while analyzing the dataset. We have time on the x axis and amplitude on the y axis.\n\n\n1. **Frequency Domain:** In frequency domain we have frequency on the x axis and magnitude on the y axis. To do that we are using a technique called **Fast Fourier Transform (FFT)**. \n\n\n1. **Time-Frequency Domain:** As you can see we are loosing information both of these techniques. For all information we are using **Spectrograms**. To achive Spectrograms we use **Short Time Fourier Transform (STFT)**. In STFT we are taking just a small frame of signal. So after taking STFTs we are projecting them to ceate Spectrograms. In Spectrograms we have time on the x axis, frequency on the y axis and decibels as colors.\n","metadata":{}},{"cell_type":"markdown","source":"
\n### Fast Fourier Transform (FFT)\n[Back to Introduction](#0)","metadata":{}},{"cell_type":"code","source":"fft = np.fft.fft(sample)\n\nmagnitude = np.abs(fft)\nfreq = np.linspace(0, 44100, len(magnitude))\n\nplt.figure(figsize = (15, 8))\nplt.plot(freq, magnitude)\nplt.title(\"Fast Fourier Transform\")\nplt.xlabel(\"Frequency\")\nplt.ylabel(\"Magnitude\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-10-07T16:53:04.416997Z","iopub.execute_input":"2021-10-07T16:53:04.417434Z","iopub.status.idle":"2021-10-07T16:53:04.635749Z","shell.execute_reply.started":"2021-10-07T16:53:04.417392Z","shell.execute_reply":"2021-10-07T16:53:04.635020Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* As you can see our FFT is **symetrical**. This is an issue of FFT so we can **cut the second half because we do not need it**. ","metadata":{}},{"cell_type":"code","source":"left_freq = freq[:int(len(freq)/2)]\nleft_magnitude = magnitude[:int(len(magnitude)/2)]\n\nplt.figure(figsize = (15, 8))\nplt.plot(left_freq, left_magnitude)\nplt.title(\"Fixed Fast Fourier Transform\")\nplt.xlabel(\"Frequency\")\nplt.ylabel(\"Magnitude\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-10-07T16:53:04.636793Z","iopub.execute_input":"2021-10-07T16:53:04.637202Z","iopub.status.idle":"2021-10-07T16:53:04.816555Z","shell.execute_reply.started":"2021-10-07T16:53:04.637159Z","shell.execute_reply":"2021-10-07T16:53:04.815679Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* **Let's zoom in to our FFT:**","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(3,1, figsize = (18,12))\n\nax[0].plot(left_freq[:500], left_magnitude[:500])\nax[0].set_title(\"FFT Zoom: 0-500\")\nax[0].set_ylabel(\"Magnitude\")\n\nax[1].plot(left_freq[1000:1500], left_magnitude[1000:1500])\nax[1].set_title(\"FFT Zoom: 1000-1500\")\nax[1].set_ylabel(\"Magnitude\")\n\nax[2].plot(left_freq[5000:5500], left_magnitude[5000:5500])\nax[2].set_title(\"FFT Zoom: 5000-5500\")\nax[2].set_xlabel(\"Frequency\")\nax[2].set_ylabel(\"Magnitude\")\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-10-07T16:53:04.817975Z","iopub.execute_input":"2021-10-07T16:53:04.818277Z","iopub.status.idle":"2021-10-07T16:53:05.219174Z","shell.execute_reply.started":"2021-10-07T16:53:04.818248Z","shell.execute_reply":"2021-10-07T16:53:05.218177Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n### Spectrogram\n[Back to Introduction](#0)","metadata":{}},{"cell_type":"code","source":"n_fft = 2048\nhop_length = 512\n\nstft = librosa.core.stft(sample, hop_length = hop_length, n_fft = n_fft)\nspectrogram = np.abs(stft)\n\nplt.figure(figsize = (15,8))\nlibrosa.display.specshow(spectrogram, sr = 44100, hop_length = hop_length)\nplt.title(\"Spectrogram\")\nplt.xlabel(\"Time\")\nplt.ylabel(\"Frequency\")\nplt.colorbar()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-10-07T16:53:05.220764Z","iopub.execute_input":"2021-10-07T16:53:05.221395Z","iopub.status.idle":"2021-10-07T16:53:05.662274Z","shell.execute_reply.started":"2021-10-07T16:53:05.221350Z","shell.execute_reply":"2021-10-07T16:53:05.661255Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* As you can see most of the spectogram colored with black (low decibels). The reason of that is we do not perceive sound in a **linear** way as we plotted but in a **logarithmic** way. So let's code it again with log. ","metadata":{}},{"cell_type":"code","source":"log_spectrogram = librosa.amplitude_to_db(spectrogram) # applying logarithm\n\nplt.figure(figsize = (15,8))\nlibrosa.display.specshow(log_spectrogram, sr = 44100, hop_length = hop_length)\nplt.title(\"Log Spectrogram\")\nplt.xlabel(\"Time\")\nplt.ylabel(\"Frequency\")\nplt.colorbar()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-10-07T16:53:05.663480Z","iopub.execute_input":"2021-10-07T16:53:05.663747Z","iopub.status.idle":"2021-10-07T16:53:06.117157Z","shell.execute_reply.started":"2021-10-07T16:53:05.663720Z","shell.execute_reply":"2021-10-07T16:53:06.116072Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n## MFCCs\n[Back to Introduction](#0)\n\n* **MFCC** is a complicated topic and we will not cover everything here but in machine learning it is very important. Briefly it enables us to distinguish different instruments easily. It has time on the x axis, **MFCC coefficents** on the x axis. Most of the time in machine learning we set the number of MFCCs to **13**. \n\n","metadata":{}},{"cell_type":"code","source":"MFCCs = librosa.feature.mfcc(sample, n_fft = n_fft, hop_length = hop_length, n_mfcc = 13)\n\nplt.figure(figsize = (15,8))\nlibrosa.display.specshow(MFCCs, sr = 44100, hop_length = hop_length)\nplt.title(\"MFCCs\")\nplt.xlabel(\"Time\")\nplt.ylabel(\"MFCC Coefficents\")\nplt.colorbar()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-10-07T16:53:06.118720Z","iopub.execute_input":"2021-10-07T16:53:06.119120Z","iopub.status.idle":"2021-10-07T16:53:06.312218Z","shell.execute_reply.started":"2021-10-07T16:53:06.119067Z","shell.execute_reply":"2021-10-07T16:53:06.311431Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n# Deep Learning with Audio\n[Back to Introduction](#0)\n\n* In our dataset there are samples that are taken from different instruments. We will choose 10 different instruments and train different models.\n\n","metadata":{}},{"cell_type":"markdown","source":"
\n## Preparing Data\n[Back to Introduction](#0)","metadata":{}},{"cell_type":"code","source":"print(df.label.unique())\nprint(\"Number of Instruments: \", len(df.label.unique()))","metadata":{"execution":{"iopub.status.busy":"2021-10-07T16:53:06.313227Z","iopub.execute_input":"2021-10-07T16:53:06.313625Z","iopub.status.idle":"2021-10-07T16:53:06.321809Z","shell.execute_reply.started":"2021-10-07T16:53:06.313595Z","shell.execute_reply":"2021-10-07T16:53:06.321080Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* There are **41** different instruments we will use: **Hi-hat, Trumpet, Cello, Clarinet, Snare_drum, Flute, Bass_drum, Electric_piano, Acoustic_guitar and Violin_or_fiddle**.","metadata":{}},{"cell_type":"code","source":"DURATION = 4\nSAMPLE_RATE = 16000\nAUDIO_SHAPE = SAMPLE_RATE*DURATION\n\n# Path\nDATASET_PATH = \"../input/freesound-audio-tagging/\"\n\n# Load \ndef load_train_data(label, input_length=AUDIO_SHAPE):\n train = pd.read_csv(DATASET_PATH + \"train.csv\")\n if label != None:\n train_list = train.loc[train.label == label]\n else: \n train_list = train\n cur_batch_size = len(train_list)\n train_fname_list = train_list.fname\n X = np.empty((cur_batch_size, input_length))\n for i, train_fname in enumerate(train_fname_list):\n file_path = DATASET_PATH + \"audio_train/\" + train_fname\n \n # Read and Resample the audio\n data, _ = librosa.core.load(file_path, sr=SAMPLE_RATE, res_type='kaiser_fast')\n\n # Random offset / Padding\n if len(data) > input_length:\n max_offset = len(data) - input_length\n offset = np.random.randint(max_offset)\n data = data[offset:(input_length+offset)]\n else:\n if input_length > len(data):\n max_offset = input_length - len(data)\n offset = np.random.randint(max_offset)\n else:\n offset = 0\n data = np.pad(data, (offset, input_length - len(data) - offset), \"constant\")\n X[i,] = data\n print(\"{} Files Loaded\".format(label))\n return X\n\ndef normalization(X):\n mean = X.mean(keepdims=True)\n std = X.std(keepdims=True)\n X = (X - mean) / std\n return X\n\n# Rescale Data to be in range [rangeMin, rangeMax]\ndef rescale(X, rangeMin=-1, rangeMax=+1):\n maxi = X.max()\n mini = X.min()\n X = np.interp(X, (mini, maxi), (rangeMin, rangeMax))\n return X","metadata":{"execution":{"iopub.status.busy":"2021-10-07T16:53:06.322859Z","iopub.execute_input":"2021-10-07T16:53:06.323268Z","iopub.status.idle":"2021-10-07T16:53:06.336922Z","shell.execute_reply.started":"2021-10-07T16:53:06.323238Z","shell.execute_reply":"2021-10-07T16:53:06.335934Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"instruments = [\"Hi-hat\", \"Trumpet\", \"Cello\", \"Clarinet\", \"Snare_drum\", \"Flute\", \"Bass_drum\", \"Electric_piano\", \"Acoustic_guitar\", \"Violin_or_fiddle\"]\n\nX_train = []\nY_train = []\n\nfor label, instrument in enumerate(instruments, start = 0):\n \n x = load_train_data(instrument)\n x = normalization(x)\n x = rescale(x)\n \n for file in x:\n \n X_train.append(file)\n Y_train.append(label)","metadata":{"execution":{"iopub.status.busy":"2021-10-07T16:53:06.338328Z","iopub.execute_input":"2021-10-07T16:53:06.338771Z","iopub.status.idle":"2021-10-07T16:57:06.550391Z","shell.execute_reply.started":"2021-10-07T16:53:06.338737Z","shell.execute_reply":"2021-10-07T16:57:06.548917Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(len(X_train))\nprint(len(Y_train))","metadata":{"execution":{"iopub.status.busy":"2021-10-07T16:57:06.552619Z","iopub.execute_input":"2021-10-07T16:57:06.552956Z","iopub.status.idle":"2021-10-07T16:57:06.559368Z","shell.execute_reply.started":"2021-10-07T16:57:06.552917Z","shell.execute_reply":"2021-10-07T16:57:06.558151Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train = np.array(X_train)\nY_train = np.array(Y_train)","metadata":{"execution":{"iopub.status.busy":"2021-10-07T16:57:06.560791Z","iopub.execute_input":"2021-10-07T16:57:06.561072Z","iopub.status.idle":"2021-10-07T16:57:07.135868Z","shell.execute_reply.started":"2021-10-07T16:57:06.561043Z","shell.execute_reply":"2021-10-07T16:57:07.134960Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = plt.figure(figsize = (12,8))\nsns.countplot(Y_train,palette = \"Blues\")\nplt.title(\"Label Count\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-10-07T16:57:07.137353Z","iopub.execute_input":"2021-10-07T16:57:07.137638Z","iopub.status.idle":"2021-10-07T16:57:07.332570Z","shell.execute_reply.started":"2021-10-07T16:57:07.137609Z","shell.execute_reply":"2021-10-07T16:57:07.331561Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x_train, x_test, y_train, y_test = train_test_split(X_train, Y_train, test_size = 0.2, random_state = 42)","metadata":{"execution":{"iopub.status.busy":"2021-10-07T16:57:07.336034Z","iopub.execute_input":"2021-10-07T16:57:07.336357Z","iopub.status.idle":"2021-10-07T16:57:07.790371Z","shell.execute_reply.started":"2021-10-07T16:57:07.336329Z","shell.execute_reply":"2021-10-07T16:57:07.789340Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_train = to_categorical(y_train,num_classes = 10)\ny_test = to_categorical(y_test,num_classes = 10)","metadata":{"execution":{"iopub.status.busy":"2021-10-07T16:57:07.792961Z","iopub.execute_input":"2021-10-07T16:57:07.793265Z","iopub.status.idle":"2021-10-07T16:57:07.798518Z","shell.execute_reply.started":"2021-10-07T16:57:07.793236Z","shell.execute_reply":"2021-10-07T16:57:07.797573Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"x_train shape: \",x_train.shape)\nprint(\"y_train shape: \",y_train.shape)\nprint(\"x_test shape: \",x_test.shape)\nprint(\"y_test shape: \",y_test.shape)","metadata":{"execution":{"iopub.status.busy":"2021-10-07T16:57:07.799945Z","iopub.execute_input":"2021-10-07T16:57:07.800285Z","iopub.status.idle":"2021-10-07T16:57:07.811877Z","shell.execute_reply.started":"2021-10-07T16:57:07.800255Z","shell.execute_reply":"2021-10-07T16:57:07.810847Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n## ANN\n[Back to Introduction](#0)\n\n* In first part we will use a standard ANN. As input we will give raw audio files.\n\n\n\n* If you do not know what is ANN you can check out this kernel: https://www.kaggle.com/mrhippo/data-science-notes5-deep-learning-ann","metadata":{}},{"cell_type":"code","source":"model = Sequential()\n\nmodel.add(Dense(512, kernel_regularizer = keras.regularizers.l2(0.001), input_shape = (x_train.shape[0], x_train.shape[1])))\nmodel.add(ReLU())\nmodel.add(Dropout(0.4))\n\nmodel.add(Dense(256, kernel_regularizer = keras.regularizers.l2(0.001)))\nmodel.add(ReLU())\nmodel.add(Dropout(0.3))\n\nmodel.add(Dense(64, kernel_regularizer = keras.regularizers.l2(0.001)))\nmodel.add(ReLU())\nmodel.add(Dropout(0.2))\n\nmodel.add(Dense(10, activation = \"softmax\"))\n\nmodel.compile(loss = \"categorical_crossentropy\",\n optimizer = \"adam\",\n metrics = [\"accuracy\"])\n\nhistory = model.fit(x_train,y_train,epochs = 20, batch_size = 100,validation_data = (x_test,y_test))","metadata":{"execution":{"iopub.status.busy":"2021-10-07T16:57:07.818839Z","iopub.execute_input":"2021-10-07T16:57:07.819574Z","iopub.status.idle":"2021-10-07T16:59:48.236783Z","shell.execute_reply.started":"2021-10-07T16:57:07.819474Z","shell.execute_reply":"2021-10-07T16:59:48.235526Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n### Results\n[Back to Introduction](#0)","metadata":{}},{"cell_type":"code","source":"history.history.keys()","metadata":{"execution":{"iopub.status.busy":"2021-10-07T16:59:48.238825Z","iopub.execute_input":"2021-10-07T16:59:48.239276Z","iopub.status.idle":"2021-10-07T16:59:48.246030Z","shell.execute_reply.started":"2021-10-07T16:59:48.239231Z","shell.execute_reply":"2021-10-07T16:59:48.245135Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize = (13,8))\nplt.plot(history.history[\"accuracy\"], label = \"Train Accuracy\")\nplt.plot(history.history[\"val_accuracy\"],label = \"Validation Accuracy\")\nplt.title(\"Accuracies\")\nplt.xlabel(\"Epochs\")\nplt.ylabel(\"Accuracy\")\nplt.legend()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-10-07T16:59:48.247799Z","iopub.execute_input":"2021-10-07T16:59:48.248207Z","iopub.status.idle":"2021-10-07T16:59:48.488235Z","shell.execute_reply.started":"2021-10-07T16:59:48.248165Z","shell.execute_reply":"2021-10-07T16:59:48.487127Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize = (13,8))\nplt.plot(history.history[\"loss\"], label = \"Train Loss\")\nplt.plot(history.history[\"val_loss\"],label = \"Validation Loss\")\nplt.title(\"Losses\")\nplt.xlabel(\"Epochs\")\nplt.ylabel(\"Loss\")\nplt.legend()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-10-07T16:59:48.489968Z","iopub.execute_input":"2021-10-07T16:59:48.490402Z","iopub.status.idle":"2021-10-07T16:59:48.672608Z","shell.execute_reply.started":"2021-10-07T16:59:48.490358Z","shell.execute_reply":"2021-10-07T16:59:48.671565Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* As you can see our model is not that good and has overfitting. Next part we will train CNN on MFCCs.","metadata":{}},{"cell_type":"markdown","source":"
\n## CNN\n[Back to Introduction](#0)\n\n* In this part we will use CNN. We will give MFCCs of audios' as input.\n\n\n\n* If you do not know what is CNN you can check out this kernel: https://www.kaggle.com/mrhippo/data-science-notes6-deep-learning-cnn","metadata":{}},{"cell_type":"code","source":"mfcc_train = []\n\nfor file in X_train:\n MFCCs = librosa.feature.mfcc(file, n_fft = n_fft, hop_length = hop_length, n_mfcc = 15)\n mfcc_train.append(MFCCs.T) ","metadata":{"execution":{"iopub.status.busy":"2021-10-07T16:59:48.673892Z","iopub.execute_input":"2021-10-07T16:59:48.674196Z","iopub.status.idle":"2021-10-07T17:00:16.563758Z","shell.execute_reply.started":"2021-10-07T16:59:48.674165Z","shell.execute_reply":"2021-10-07T17:00:16.562835Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"mfcc_train = np.array(mfcc_train)","metadata":{"execution":{"iopub.status.busy":"2021-10-07T17:00:16.565419Z","iopub.execute_input":"2021-10-07T17:00:16.566186Z","iopub.status.idle":"2021-10-07T17:00:16.593027Z","shell.execute_reply.started":"2021-10-07T17:00:16.566129Z","shell.execute_reply":"2021-10-07T17:00:16.592185Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"mfcc_train.shape","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-10-07T17:00:16.594397Z","iopub.execute_input":"2021-10-07T17:00:16.594940Z","iopub.status.idle":"2021-10-07T17:00:16.601490Z","shell.execute_reply.started":"2021-10-07T17:00:16.594906Z","shell.execute_reply":"2021-10-07T17:00:16.600201Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x_train, x_test, y_train, y_test = train_test_split(mfcc_train, Y_train, test_size = 0.2, random_state = 42)","metadata":{"execution":{"iopub.status.busy":"2021-10-07T17:00:16.603641Z","iopub.execute_input":"2021-10-07T17:00:16.603975Z","iopub.status.idle":"2021-10-07T17:00:16.629158Z","shell.execute_reply.started":"2021-10-07T17:00:16.603945Z","shell.execute_reply":"2021-10-07T17:00:16.628219Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_train = to_categorical(y_train,num_classes = 10)\ny_test = to_categorical(y_test,num_classes = 10)","metadata":{"execution":{"iopub.status.busy":"2021-10-07T17:00:16.630725Z","iopub.execute_input":"2021-10-07T17:00:16.631023Z","iopub.status.idle":"2021-10-07T17:00:16.636948Z","shell.execute_reply.started":"2021-10-07T17:00:16.630993Z","shell.execute_reply":"2021-10-07T17:00:16.635780Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"x_train shape: \",x_train.shape)\nprint(\"y_train shape: \",y_train.shape)\nprint(\"x_test shape: \",x_test.shape)\nprint(\"y_test shape: \",y_test.shape)","metadata":{"execution":{"iopub.status.busy":"2021-10-07T17:00:16.638557Z","iopub.execute_input":"2021-10-07T17:00:16.639002Z","iopub.status.idle":"2021-10-07T17:00:16.650095Z","shell.execute_reply.started":"2021-10-07T17:00:16.638971Z","shell.execute_reply":"2021-10-07T17:00:16.648898Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* **We have to add a 4th dimention for keras:**","metadata":{}},{"cell_type":"code","source":"x_train = x_train.reshape(x_train.shape[0], 126, 15, 1)\nx_test = x_test.reshape(x_test.shape[0], 126, 15, 1)\nprint(x_train.shape)\nprint(x_test.shape)","metadata":{"execution":{"iopub.status.busy":"2021-10-07T17:00:16.652495Z","iopub.execute_input":"2021-10-07T17:00:16.652830Z","iopub.status.idle":"2021-10-07T17:00:16.663764Z","shell.execute_reply.started":"2021-10-07T17:00:16.652800Z","shell.execute_reply":"2021-10-07T17:00:16.662841Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = Sequential()\n\nmodel.add(keras.layers.Conv2D(32, (3, 3), activation = \"relu\", input_shape = (x_train.shape[1], x_train.shape[2], x_train.shape[3])))\nmodel.add(keras.layers.MaxPool2D((3, 3), strides = (2, 2), padding = \"same\"))\nmodel.add(keras.layers.BatchNormalization())\n\nmodel.add(keras.layers.Conv2D(32, (3, 3), activation = \"relu\"))\nmodel.add(keras.layers.MaxPool2D((3, 3), strides = (2, 2), padding = \"same\"))\nmodel.add(keras.layers.BatchNormalization())\n\nmodel.add(keras.layers.Conv2D(32, (3, 3), activation = \"relu\"))\nmodel.add(keras.layers.MaxPool2D((3, 3), strides = (2, 2), padding = \"same\"))\nmodel.add(keras.layers.BatchNormalization())\n\nmodel.add(keras.layers.Flatten())\nmodel.add(Dense(64, activation = \"relu\"))\nmodel.add(Dropout(0.3))\n\nmodel.add(Dense(10, activation = \"softmax\"))\n\nmodel.compile(loss = \"categorical_crossentropy\",\n optimizer = \"adam\",\n metrics = [\"accuracy\"])\n\nhistory = model.fit(x_train,y_train,epochs = 30, batch_size = 100,validation_data = (x_test,y_test))","metadata":{"execution":{"iopub.status.busy":"2021-10-07T17:00:16.665214Z","iopub.execute_input":"2021-10-07T17:00:16.665646Z","iopub.status.idle":"2021-10-07T17:01:22.684153Z","shell.execute_reply.started":"2021-10-07T17:00:16.665600Z","shell.execute_reply":"2021-10-07T17:01:22.682696Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n### Results\n[Back to Introduction](#0)","metadata":{}},{"cell_type":"code","source":"history.history.keys()","metadata":{"execution":{"iopub.status.busy":"2021-10-07T17:01:22.685773Z","iopub.execute_input":"2021-10-07T17:01:22.686071Z","iopub.status.idle":"2021-10-07T17:01:22.692645Z","shell.execute_reply.started":"2021-10-07T17:01:22.686043Z","shell.execute_reply":"2021-10-07T17:01:22.691402Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize = (13,8))\nplt.plot(history.history[\"accuracy\"], label = \"Train Accuracy\")\nplt.plot(history.history[\"val_accuracy\"],label = \"Validation Accuracy\")\nplt.title(\"Accuracies\")\nplt.xlabel(\"Epochs\")\nplt.ylabel(\"Accuracy\")\nplt.legend()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-10-07T17:01:22.693869Z","iopub.execute_input":"2021-10-07T17:01:22.694256Z","iopub.status.idle":"2021-10-07T17:01:22.897869Z","shell.execute_reply.started":"2021-10-07T17:01:22.694196Z","shell.execute_reply":"2021-10-07T17:01:22.897024Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize = (13,8))\nplt.plot(history.history[\"loss\"], label = \"Train Loss\")\nplt.plot(history.history[\"val_loss\"],label = \"Validation Loss\")\nplt.title(\"Losses\")\nplt.xlabel(\"Epochs\")\nplt.ylabel(\"Loss\")\nplt.legend()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-10-07T17:01:22.899338Z","iopub.execute_input":"2021-10-07T17:01:22.899738Z","iopub.status.idle":"2021-10-07T17:01:23.074136Z","shell.execute_reply.started":"2021-10-07T17:01:22.899695Z","shell.execute_reply":"2021-10-07T17:01:23.073166Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* You can increase accuracy by changing parameters.","metadata":{}},{"cell_type":"markdown","source":"
\n# Conclusion\n[Back to Introduction](#0)\n\n* **We learned how to process sound.**\n* **If there is something wrong with this kernel please let me know in the comments.**\n\n### Here is a notebook about generating audio with GANs: https://www.kaggle.com/mrhippo/audio-generation-with-simple-gans\n\n### My other kernels: https://www.kaggle.com/mrhippo/notebooks\n\n* **References:**\n* https://www.youtube.com/playlist?list=PL-wATfeyAMNqIee7cH3q1bh4QJFAaeNv0\n* https://www.kaggle.com/fizzbuzz/beginner-s-guide-to-audio-data\n* https://github.com/rafalencar1997/Audio_Generation","metadata":{}}]}