{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"### At the end of this Notebook you can generate and download the whole 128x128 image (mel-spectrograms) training dataset for short audio sounds, as well as the whole cropped 128x128 image training soundscape dataset","metadata":{}},{"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport os\n# for dirname, _, filenames in os.walk('/kaggle/input'):\n#     for filename in filenames:\n#         print(os.path.join(dirname, filename))\n\nimport matplotlib.pyplot as plt\nimport IPython.display as ipd\nimport warnings\nwarnings.filterwarnings(action='ignore')\nimport pathlib\nfrom pathlib import Path\nfrom PIL import Image\nfrom tqdm import tqdm\n\n# Audio library for Python\nimport librosa\nimport librosa.display","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Sources\n\nhttps://www.kaggle.com/stefankahl/birdclef2021-processing-audio-data\n\n# Data\n\n* **train_short_audio** - The bulk of the training data consists of short recordings of individual bird calls generously uploaded by users of xenocanto.org. These files have been downsampled to 32 kHz where applicable to match the test set audio and converted to the ogg format. The training data should have nearly all relevant files; we expect there is no benefit to looking for more on xenocanto.org.\n* **train_soundscapes** - Audio files that are quite comparable to the test set. They are all roughly ten minutes long and in the ogg format. The test set also has soundscapes from the two recording locations represented here.\n* **test_soundscapes** - When you submit a notebook, the test_soundscapes directory will be populated with approximately 80 recordings to be used for scoring. These will be roughly 10 minutes long and in ogg audio format. The file names include the date the recording was taken, which can be especially useful for identifying migratory birds.\n\nThis folder also contains text files with the name and approximate coordinates of the recording location plus a csv with the set of dates the test set soundscapes were recorded.\n\n* **test.csv** - Only the first three rows are available for download; the full test.csv is in the hidden test set.\n    * row_id: ID code for the row.\n    * site: Site ID.\n    * seconds: the second ending the time window\n    * audio_id: ID code for the audio file.\n* **train_metadata.csv** - A wide range of metadata is provided for the training data. The most directly relevant fields are:\n    * primary_label: a code for the bird species. You can review detailed information about the bird codes by appending the code to https://ebird.org/species/, such as https://ebird.org/species/amecro for the American Crow.\n    * recodist: the user who provided the recording.\n    * latitude & longitude: coordinates for where the recording was taken. Some bird species may have local call 'dialects,' so you may want to seek geographic diversity in your training data.\n    * date: while some bird calls can be made year round, such as an alarm call, some are restricted to a specific season. You may want to seek temporal diversity in your training data.\n\n    * filename: the name of the associated audio file.\n* **train_soundscape_labels.csv** -\n    * row_id: ID code for the row.\n    * site: Site ID.\n    * seconds: the second ending the time window\n    * audio_id: ID code for the audio file.\n    * birds: space delimited list of any bird songs present in the 5 second window. The label nocall means that no call occurred.\n* **sample_submission.csv** - A properly formed sample submission file. Only the first three rows are public, the remainder will be provided to your notebook as part of the hidden test set.\n    * row_id\n    * birds: space delimited list of any bird songs present in the 5 second window. If there are no bird calls, use the label nocall.\n    \n","metadata":{}},{"cell_type":"code","source":"path = Path('/kaggle/working')\ntrain_short_audio_path = Path('../input/birdclef-2021/train_short_audio')\ntrain_soundscapes = Path('../input/birdclef-2021/train_soundscapes')\ntrain_path = path/'train'\ntrain_soundscapes_path = path/'train_soundscapes'\ntrain_path.mkdir(exist_ok=True)\ntrain_soundscapes_path.mkdir(exist_ok=True)\n\ntraining_df = pd.read_csv(\"../input/birdclef-2021/train_metadata.csv\")\ntraining_soundscapes_df = pd.read_csv(\"../input/birdclef-2021/train_soundscape_labels.csv\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"training_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Explore data\n\n## Listen to files\n\n### short train file\n\nThe extracts are short, and you can often hear the bird singing several times in a row. When you listen to the files of the same folder, you realize that a single bird can be assimilated to many different calls.","metadata":{}},{"cell_type":"code","source":"# Pick a file\naudio_path = '../input/birdclef-2021/train_short_audio/rubwre1/XC236057.ogg'\n\n# Listen to it\nipd.Audio(audio_path)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Long train file\n\nThe extracts are long, noisy, and sometimes there is no call during the whole extract.","metadata":{}},{"cell_type":"code","source":"# Pick a file\naudio_path_long = '../input/birdclef-2021/train_soundscapes/11254_COR_20190904.ogg'\n\n# Listen to it\n# ipd.Audio(audio_path_long)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Transform audio into a numpy array\n\nWe need to feed sound waves into a computer. But sound is transmitted as waves. How do we turn sound waves into numbers?\n\n\nSound waves are one-dimensional. At every moment in time, they have a single value based on the height of the wave. To turn this sound wave into numbers, we just record of the height of the wave at equally-spaced points.\n\nThis is called **sampling**. We are taking a reading thousands of times a second and recording a number representing the height of the sound wave at that point in time. That’s basically all an uncompressed .wav audio file is.\n\nHere, audio files are sampled at 32khz (32000 readings per second).","metadata":{}},{"cell_type":"code","source":"sig, rate = librosa.load(audio_path, sr=32000, offset=None)\n\n# The result is a 1D numpy array that contains audio samples. shape = seconds * sample rate = 5 * 32000 == 160000)\nprint('SIGNAL SHAPE:', sig.shape)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see, we get a 1D array of 5x32000=160000 numbers, because 1/32000 second, we have a number which represent he height of the sound wave at that point in time. We can visualize this:","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(15, 5))\nlibrosa.display.waveplot(sig, sr=32000)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Transform audio 1D array into 2D array (spectrogram) for image recognition\n\nWe now have an array of numbers with each number representing the sound wave’s amplitude at 1/44100th of a second intervals.\n\nWe could feed these numbers right into a neural network. But trying to recognize speech patterns by processing these samples directly is difficult. We’ve seen in the past that convolutional neural networks (CNN) perform particularly well for sound classification. But CNN need 2D inputs. Luckily, we can transform an audio signal into a 2D representation.\n\nTo make this data easier for a neural network to process, we are going to break apart this complex sound wave into it’s component parts. We’ll break out the low-pitched parts, the next-lowest-pitched-parts, and so on. Then by adding up how much energy is in each of those frequency bands (from low to high), we create a fingerprint of sorts for this audio snippet.\n\nWe do this using a mathematic operation called a Fourier transform. It breaks apart the complex sound wave into the simple sound waves that make it up. Once we have those individual sound waves, we add up how much energy is contained in each one.\n\nThe end result is a score of how important each frequency range is, from low pitch (i.e. bass notes) to high pitch.\n\n![](https://cdn-images-1.medium.com/max/1200/1*A4CxgdyqYd_nrF3e-7ETWA.png)\n\nIf we repeat this process on every 20 millisecond chunk of audio, we end up with a spectrogram.","metadata":{}},{"cell_type":"code","source":"# First, compute the spectrogram using the \"short-time Fourier transform\" (stft)\nspec = librosa.stft(sig)\n\n# Scale the amplitudes according to the decibel scale\nspec_db = librosa.amplitude_to_db(spec, ref=np.max)\n\n# Plot the spectrogram\nplt.figure(figsize=(15, 5))\nlibrosa.display.specshow(spec_db, \n                         sr=32000, \n                         x_axis='time', \n                         y_axis='hz', \n                         cmap=plt.get_cmap('viridis'))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('SPEC SHAPE:', spec_db.shape)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Very nice! However, that's still a very large input for a CNN. Let's change the *“window length”* and *\"hop length\"*\nA good final size could be 128x128 or 256x256 (if the second number is higher than 128 or 256, we can then split the sample into multiple smaller samples).\n\nWe could use the so-called mel scale (https://en.wikipedia.org/wiki/Mel_scale) to scale the frequency axis of our spectrogram. In the past, this attempt (even though it was initially designed for human speech) worked well for bird sound recognition. Luckily, Librosa supports this transformation. We can set the number of mel bins we want to use and that number would eventually be our vertical resolution of the spectrogram. We also know that the hop length we choose is key for the width of the spectrogram, so we have to settle on a certain value. On top of that, we should probably process 5-second chunks of audio (since that’s the submission segment duration).\n\nWe should probably also consider the vocal and auditory range of birds. We know that most songbirds vocalize between 1 and 4 kHz. Yet, some species vocalize below that, and some significantly above. In general, we can probably limit the frequency range we want to include in a spectrogram between 500 Hz and 12.5 kHz. Not many birds will vocalize outside this range.\n\n","metadata":{}},{"cell_type":"code","source":"# Desired shape of the input spectrogram for a 5s time window\nSPEC_HEIGHT = 128\nSPEC_WIDTH = 128\n\n# Derive num_mels and hop_length from desired spec shape\n# num_mels is easy, that's just spec_height\n# hop_length is a bit more complicated\nNUM_MELS = SPEC_HEIGHT\nHOP_LENGTH = int(32000 * 5 / (SPEC_WIDTH - 1)) # sample rate * duration / spec width - 1 == 627\n\n# High- and low-pass frequencies\n# For many birds, these are a good choice\nFMIN = 20\nFMAX = 16000\n\n# Compute the spectrogram and apply the mel scale\nmel_spec = librosa.feature.melspectrogram(y=sig, \n                                      sr=32000, \n                                      n_fft=2048, \n                                      hop_length=HOP_LENGTH, \n                                      n_mels=NUM_MELS, \n                                      fmin=FMIN, \n                                      fmax=FMAX)\n\nmel_spec_db = librosa.power_to_db(mel_spec, ref=np.max)\n\n# Show the spec\nplt.figure()\nplt.title('Shape: ' + str(mel_spec_db.shape))\nplt.imshow(mel_spec_db)\n# librosa.display.specshow(mel_spec_db, \n#                              sr=32000, \n#                              hop_length=HOP_LENGTH, \n#                              x_axis='time', \n#                              y_axis='mel',\n#                              fmin=FMIN, \n#                              fmax=FMAX, \n#                              cmap=plt.get_cmap('viridis'))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Last but not least, we can convert this 2D array into a 3D array.\nThe mono_to_color function takes as an input the spectrogram of our sound. \n* It stacks it three times, so that it has the same shape as a classic RGB image.\n* Then it standardize the array (take a matrix and change it so that its mean is equal to 0 and variance is 1). This improves performance.\n* Then it normalizes each value between 0 and 255 (gray scale).","metadata":{}},{"cell_type":"code","source":"def mono_to_color(X: np.ndarray,\n                  mean=None,\n                  std=None,\n                  norm_max=None,\n                  norm_min=None,\n                  eps=1e-6):\n    \"\"\"\n    Code from https://www.kaggle.com/daisukelab/creating-fat2019-preprocessed-data\n    \"\"\"\n    # Stack X as [X,X,X]\n    X = np.stack([X, X, X], axis=-1)\n\n    # Standardize\n    mean = mean or X.mean()\n    X = X - mean\n    std = std or X.std()\n    Xstd = X / (std + eps)\n    _min, _max = Xstd.min(), Xstd.max()\n    norm_max = norm_max or _max\n    norm_min = norm_min or _min\n    if (_max - _min) > eps:\n        # Normalize to [0, 255]\n        V = Xstd\n        V[V < norm_min] = norm_min\n        V[V > norm_max] = norm_max\n        V = 255 * (V - norm_min) / (norm_max - norm_min)\n        V = V.astype(np.uint8)\n    else:\n        # Just zero\n        V = np.zeros_like(Xstd, dtype=np.uint8)\n    return V","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image = mono_to_color(mel_spec_db)\nplt.title('Shape: ' + str(image.shape))\nplt.imshow(image)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Prepare Data\n\nLet's use everything we've done above to define:\n* A function which convert an ogg file into a 3D array image.\n* A function which split a soundscape into 5 second splits and then convert it into a 3D array image.","metadata":{}},{"cell_type":"code","source":"SPEC_HEIGHT = 128\nSPEC_WIDTH = 128\nNUM_MELS = SPEC_HEIGHT\nHOP_LENGTH = int(32000 * 5 / (SPEC_WIDTH - 1)) # sample rate * duration / spec width - 1 == 627\nFMIN = 20\nFMAX = 16000\nSAMPLE_RATE = 32000\nN_FFT = 2048\nDURATION = 5\n\ndef ogg_to_image(audio_path):\n    # Load the ogg file\n    sig, rate = librosa.load(audio_path, sr=SAMPLE_RATE, offset=None, duration=DURATION)\n    # Get start and stop sample\n    s_start = 0\n    s_end = DURATION * 32000\n    # Compute the spectrogram and apply the mel scale\n    mel_spec = librosa.feature.melspectrogram(y=sig[s_start:s_end], sr=SAMPLE_RATE, n_fft=N_FFT,\n                                              hop_length=HOP_LENGTH, n_mels=NUM_MELS, fmin=FMIN, fmax=FMAX)\n    mel_spec_db = librosa.power_to_db(mel_spec, ref=np.max)\n    # Convert the spectrogram into a 128x128x3 image\n    image = mono_to_color(mel_spec_db)\n    return image\n\ndef soundscape_to_images(audio_path):\n    images_soundscape = []\n    # Load the ogg file\n    sig, rate = librosa.load(audio_path, sr=SAMPLE_RATE, offset=None)\n    # Compute the spectrogram and apply the mel scale\n    \n    for second in tqdm(range(DURATION, 605, DURATION)):  \n        # Get start and stop sample\n        s_start = (second - 5) * 32000\n        s_end = second * 32000\n        # Compute the spectrogram and apply the mel scale\n        mel_spec = librosa.feature.melspectrogram(y=sig[s_start:s_end], sr=SAMPLE_RATE, n_fft=N_FFT,\n                                                  hop_length=HOP_LENGTH, n_mels=NUM_MELS, fmin=FMIN, fmax=FMAX)\n        mel_spec_db = librosa.power_to_db(mel_spec, ref=np.max)\n        # Convert the spectrogram into a 128x128x3 image\n        image = mono_to_color(mel_spec_db)\n        images_soundscape.append(image)\n    return images_soundscape","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let's test if everything works","metadata":{}},{"cell_type":"code","source":"audio_path = '../input/birdclef-2021/train_short_audio/rubwre1/XC236057.ogg'\nimage = ogg_to_image(audio_path)\nplt.imshow(image)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"audio_path_soundscape = '../input/birdclef-2021/train_soundscapes/10534_SSW_20170429.ogg'\nimages_soundscape = soundscape_to_images(audio_path_soundscape)\nprint('Shape: ' + str(np.array(images_soundscape).shape))\nfor i in range(0,5):\n    plt.imshow(images_soundscape[i])\n    plt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Get all processed data\n\nFor now I just take the first 5s of audio for short audio samples.","metadata":{}},{"cell_type":"code","source":"#Creating every image in training set for short audio  \nfor i, row in tqdm(enumerate(training_df.values)):\n    dir = train_path/str(row[0])\n    dir.mkdir(exist_ok=True)\n    audio_path = train_short_audio_path/str(row[0])/str(row[9])\n    img = ogg_to_image(audio_path)\n    filename = str(row[9])[:-4]\n    Image.fromarray(img, mode='RGB').save(dir/f\"{filename}.png\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Here it is fast because there are not many audio files, and the slow part is loading the audio file \nfor i in tqdm(range(0, len(training_soundscapes_df), 120)):\n    #Find the filename in the list of files in the directory\n    audio_filename = [j for j in os.listdir(train_soundscapes) if os.path.isfile(os.path.join(train_soundscapes,j)) and \n                      training_soundscapes_df.iloc[i]['row_id'][:-2] in j][0]\n    audio_path_soundscape = train_soundscapes/audio_filename\n    imgs = soundscape_to_images(audio_path_soundscape)\n    for j in range(len(imgs)):\n        dir = train_soundscapes_path/training_soundscapes_df.iloc[i+j]['birds']\n        dir.mkdir(exist_ok=True)\n        filename = str(training_soundscapes_df.iloc[i+j]['row_id'])\n        Image.fromarray(imgs[j], mode='RGB').save(dir/f\"{filename}.png\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Profit: download the processed dataset!\n\nOn the top right of your screen zip files will be generated, just download it.","metadata":{}},{"cell_type":"code","source":"# Sorry fot this.\n!tar -zcvf train_a.tar.gz /kaggle/working/train/a*\n!tar -zcvf train_b.tar.gz /kaggle/working/train/b*\n!tar -zcvf train_c.tar.gz /kaggle/working/train/c*\n!tar -zcvf train_d.tar.gz /kaggle/working/train/d*\n!tar -zcvf train_e.tar.gz /kaggle/working/train/e*\n!tar -zcvf train_f.tar.gz /kaggle/working/train/f*\n!tar -zcvf train_g.tar.gz /kaggle/working/train/g*\n!tar -zcvf train_h.tar.gz /kaggle/working/train/h*\n!tar -zcvf train_i.tar.gz /kaggle/working/train/i*\n!tar -zcvf train_j.tar.gz /kaggle/working/train/j*\n!tar -zcvf train_k.tar.gz /kaggle/working/train/k*\n!tar -zcvf train_l.tar.gz /kaggle/working/train/l*\n!tar -zcvf train_m.tar.gz /kaggle/working/train/m*\n!tar -zcvf train_n.tar.gz /kaggle/working/train/n*\n!tar -zcvf train_o.tar.gz /kaggle/working/train/o*\n!tar -zcvf train_p.tar.gz /kaggle/working/train/p*\n!tar -zcvf train_q.tar.gz /kaggle/working/train/q*\n!tar -zcvf train_r.tar.gz /kaggle/working/train/r*\n!tar -zcvf train_s.tar.gz /kaggle/working/train/s*\n!tar -zcvf train_t.tar.gz /kaggle/working/train/t*\n!tar -zcvf train_u.tar.gz /kaggle/working/train/u*\n!tar -zcvf train_v.tar.gz /kaggle/working/train/v*\n!tar -zcvf train_w.tar.gz /kaggle/working/train/w*\n!tar -zcvf train_x.tar.gz /kaggle/working/train/x*\n!tar -zcvf train_y.tar.gz /kaggle/working/train/y*\n!tar -zcvf train_z.tar.gz /kaggle/working/train/z*\n\n# !tar -zcvf train_soundscapes.tar.gz /kaggle/working/train_soundscapes","metadata":{"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]}]}