{"cells":[{"metadata":{},"cell_type":"markdown","source":"<center><img src=\"https://i.imgur.com/bIw8deA.jpg\" width=\"1000px\"></center>","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Introduction\n\nWelcome to the \"Cornell Birdcall Identification\" challenge on Kaggle! In this challenge, contestants need to identify the species of birds involved in audio clips of them calling. In this kernel, I will generate <code>melspectrograms</code> from all the training audio clips and save them as images, so that training can be speeded up!","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Acknowledgements\n\n1. [LibROSA](https://librosa.org/librosa/) ~ by the librosa team\n2. [Audio Data Analysis Using librosa 📈](https://www.kaggle.com/hamditarek/audio-data-analysis-using-librosa) ~ by Tarek Hamdi\n3. [Understanding the Mel Spectrogram](https://medium.com/analytics-vidhya/understanding-the-mel-spectrogram-fca2afa2ce53) ~ by Leland Roberts\n4. [Bidirectional LSTM for audio labeling with Keras](https://www.kaggle.com/carlolepelaars/bidirectional-lstm-for-audio-labeling-with-keras) ~ by Carlo Lepelaars","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Preparing the ground <a id=\"1\"></a>\n\nIn this section, we will prepare the ground to train and test the model by installing packages, setting hyperparameters, and loading the data.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Install additional packages <a id=\"1.1\"></a>\n\n* We will now install <code>pydub</code>\n* <code>pydub</code> will help us load audio data from <code>.mp3</code> files much faster than the <code>librosa</code> command: <code>librosa.load</code>","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install -q pydub","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Import necessary libraries <a id=\"1.2\"></a>\n\n* Now, we import all the libraries we need.\n* <code>matplotlib</code> and <code>tqdm</code> for data analysis and visualization.\n* <code>librosa</code>, <code>pydub</code> and <code>keras</code> for model training and inference.\n* <code>numpy</code>, <code>pandas</code>, and <code>sklearn</code> for data processing and manipulation.","execution_count":null},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"import os\nimport gc\nimport cv2\nimport time\nimport numpy as np\nimport pandas as pd\nfrom tqdm.notebook import tqdm\nfrom joblib import Parallel, delayed\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\nimport pydub\nimport librosa\nimport librosa.display\nfrom pydub import AudioSegment as AS\nfrom librosa.feature import melspectrogram\nfrom librosa.core import power_to_db as ptdb\n\nfrom keras.utils import to_categorical\nfrom keras.preprocessing.sequence import pad_sequences as pad","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Define key hyperparameters and paths <a id=\"1.3\"></a>\n\n* Here we define the key hyperparameters: sequence length, train/val split, batch size, epochs, LR.\n* We also specify the correct paths for loading data, training, inference, and finally submission to this competition.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Define hyperparameters\n\nThe hyperparameters used are documented below ~\n\n1. Data processing related\n\n    * <code>N_MELS</code> is the number of <code>melspectrogram</code> features per time step.\n    * <code>AMPLITUDE</code> represents the default signal amplitude applied to unreadable files.\n    * <code>SR</code> is the sampling rate at which the audio is loaded (readings per second). It defaults to <code>44100 Hz</code>.\n    * <code>TSR</code> is the sampling rate at which the test audio clips are loaded. It defaults to <code>32000 Hz</code>.\n    * <code>MAXLEN</code> is the maximum number of readings. Longer clips will be trimmed and shorter ones will be padded.\n    * <code>SPLIT</code> represents the fraction of data to be used for training. The rest of the data is used for validation.\n    * <code>CHUNKS</code> represents the number of chunks that will be extracted from each signal. It defaults to <code>1</code> per signal.\n    * <code>CHUNK_SIZE</code> represents the sequence length of each audio chunk to be fed into the <code>melspectrogram</code> function.\n    * <code>POP_FRAC</code> is the maximum proportion of signal information to be ignored per chunk (defaults to <code>0.25</code>).","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"N = 8\nSR = 44100\nCHUNKS = 1\nTSR = 32000\nN_MELS = 128\nPOP_FRAC = 0.25\nMAXLEN = 1000000\nAMPLITUDE = 1000\nCHUNK_SIZE = 500000","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Check data available\n\nWe have two datasets at our disposal: <code>birdsong-recognition</code> and <code>prepare-check-dataset</code>.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"os.listdir('../input')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Define paths\n\nThe paths used are documented below ~\n\n1. Metadata related\n\n    * <code>../input/birdsong-recognition/test.csv</code> contains the test metadata used for submission.\n    * <code>../input/birdsong-recognition/train.csv</code> contains the train metadata used for submission.\n    * <code>../input/prepare-check-dataset/test.csv</code> contains the test metadata used for committing.\n\n\n2. Audio data related\n\n    * <code>../input/birdsong-recognition/test_audio</code> contains the test audio used for submission.\n    * <code>../input/birdsong-recognition/train_audio</code> contains the train audio used for submission.\n    * <code>../input/prepare-check-dataset/test_audio</code> contains the test audio used for committing.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"TEST_DATA_PATH = '../input/birdsong-recognition/test.csv'\nTRAIN_DATA_PATH = '../input/birdsong-recognition/train.csv'\nTEST_AUDIO_PATH = '../input/birdsong-recognition/test_audio/'\nTRAIN_AUDIO_PATH = '../input/birdsong-recognition/train_audio/'\nCHECKING_PATH = '../input/prepare-check-dataset/birdcall-check/'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub = os.path.exists(TEST_AUDIO_PATH)\nTEST_DATA_PATH = TEST_DATA_PATH if sub else CHECKING_PATH + 'test.csv'\nTEST_AUDIO_PATH = TEST_AUDIO_PATH if sub else CHECKING_PATH + 'test_audio/'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load metadata from .csv files <a id=\"1.4\"></a>\n\n* Now we load the training and testing metadata.\n\n* We can see that the testing dataframe has only <code>3</code> rows in it.\n\n* This is only a dummy test dataframe. The actual testing data will be used during submission.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df = pd.read_csv(TEST_DATA_PATH)\ntrain_df = pd.read_csv(TRAIN_DATA_PATH)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Prepare the label dictionary\n\n* Next we prepare a dictionary linking each bird species to a unique integer.\n* This dictionary will help us when we need to one-hot encode our targets later.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"keys = set(train_df.ebird_code)\nvalues = np.arange(0, len(keys))\ncode_dict = dict(zip(sorted(keys), values))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data processing <a id=\"3.1\"></a>\n\nThe first step to define important function to process the data and generate features.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Define utility function to read audio\n\n* Now we define a function using <code>pydub</code> to read audio files into <code>numpy</code> arrays.\n* This implementation is significantly faster than <code>librosa.load</code> and <code>torchaudio.load</code>.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def normalize(x):\n    return np.float32(x)/2**15\n\ndef read(file, norm=False):\n    try:\n        a = AS.from_mp3(file)\n        a = a.set_frame_rate(TSR)\n    except:\n        return TSR, np.zeros(MAXLEN)\n\n    y = np.array(a.get_array_of_samples())\n    if a.channels == 2: y = y.reshape((-1, 2))\n    if norm: return a.frame_rate, normalize(y)\n    if not norm: return a.frame_rate, np.float32(y)\n\ndef write(file, sr, x, normalized=False):\n    birds_audio_bitrate, file_format = '320k', 'mp3'\n    ch = 2 if (x.ndim == 2 and x.shape[1] == 2) else 1\n    y = np.int16(x * 2 ** 15) if normalized else np.int16(x)\n    song = AS(y.tobytes(), frame_rate=sr, sample_width=2, channels=ch)\n    song.export(file, format=file_format, bitrate=birds_audio_bitrate)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Define functions to process audio signals\n\nThese are a set of functions which process the audio before the <code>melspectrogram</code> transformation.\n\nThe functions used are documented below ~\n\n* <code>get_idx</code> selects the start and end index of a given audio chunk.\n* <code>get_chunk</code> takes indices from <code>get_idx</code> and outputs a chunk of data between those indices.\n* <code>get_len</code> is a helper function which is used to decide possible chunk indices based on <code>POP_FRAC</code>.\n  \n  --> <code>If</code> the signal is longer than <code>MAXLEN</code>, it sets the maximum index to <code>MAXLEN</code>.<br>\n  --> <code>Else</code> it uses <code>POP_FRAC</code> to ensure chunks are centered around audio signal and not padding.\n\n\n* <code>get_signal</code> flattens the signal, pads it to <code>MAXLEN</code>, and stacks multiple chunks into one array.","execution_count":null},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"def get_idx(length):\n    length = get_len(length)\n    max_idx = MAXLEN - CHUNK_SIZE\n    idx = np.random.randint(length + 1)\n    chunk_range = idx, idx + CHUNK_SIZE\n    chunk_idx = max([0, chunk_range[0]])\n    chunk_idx = min([chunk_range[1], max_idx])\n    return (chunk_idx, chunk_idx + CHUNK_SIZE)\n\ndef get_len(length):\n    if length > MAXLEN: return MAXLEN\n    if length <= MAXLEN: return int(length*POP_FRAC)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_chunk(data, length):\n    index = get_idx(length)\n    return data[index[0]:index[1]]\n\ndef get_signal(data):\n    length = max(data.shape)\n    data = data.T.flatten().reshape(1, -1)\n    data = np.float32(pad(data, maxlen=MAXLEN).reshape(-1))\n    return [get_chunk(data, length) for _ in range(CHUNKS)]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Define functions to calculate melspectrogram features\n\nBelow we define some functions to calculate the <code>melspectrogram</code> features from audio signals.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def to_imagenet(X, mean=None, std=None, norm_max=None, norm_min=None, eps=1e-6):\n    mean = mean or X.mean()\n    X = X - mean\n    std = std or X.std()\n    Xstd = X / (std + eps)\n    _min, _max = Xstd.min(), Xstd.max()\n    norm_max = norm_max or _max\n    norm_min = norm_min or _min\n    if (_max - _min) > eps:\n        # Normalize to [0, 255]\n        V = Xstd\n        V[V < norm_min] = norm_min\n        V[V > norm_max] = norm_max\n        V = 255*((V - norm_min) / (norm_max - norm_min))\n    else:\n        # Just zero\n        V = np.zeros_like(Xstd, dtype=np.uint8)\n    return np.stack([V]*3, axis=-1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_melsp(data):\n    melsp = melspectrogram(data, n_mels=N_MELS)\n    return to_imagenet(librosa.power_to_db(melsp))\n\ndef get_melsp_img(data):\n    data = get_signal(data)\n    return np.stack([get_melsp(point) for point in data])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Define spectrogram loading function\n\nNow we define a function that generates a <code>spectrogram</code> at a list of indices.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def save(indices, path):\n    folder = TRAIN_AUDIO_PATH\n\n    for index in tqdm(indices):\n        file_name = train_df.filename[index]\n        ebird_code = train_df.ebird_code[index]\n\n        default_signal = np.random.random(MAXLEN)*AMPLITUDE\n        default_values = SR, np.int32(np.round(default_signal))\n\n        values = read(folder + ebird_code + '/' + file_name)\n        _, data = values if len(values) == 2 else default_values\n        \n        image = np.nan_to_num(get_melsp_img(data))[0]\n        cv2.imwrite(path + file_name + '.jpg', image); del image; gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Load all training spectrograms and save with parallel processing\n\nNext we will use multi-threading to generate all the spectorgrams and save them quickly.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_ids = np.array_split(np.arange(len(train_df)), 5)\ntrain_ids_1, train_ids_2, train_ids_3, train_ids_4, train_ids_5 = train_ids","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_ids_1 = np.array_split(np.array(train_ids_1), N)\ntrain_ids_2 = np.array_split(np.array(train_ids_2), N)\ntrain_ids_3 = np.array_split(np.array(train_ids_3), N)\ntrain_ids_4 = np.array_split(np.array(train_ids_4), N)\ntrain_ids_5 = np.array_split(np.array(train_ids_5), N)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!mkdir train_1\npath = \"train_1/\"\nparallel = Parallel(n_jobs=N, backend=\"threading\")\nparallel(delayed(save)(ids, path) for ids in train_ids_1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"!zip -r train_1.zip train_1\n!rm -rf train_1","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"!mkdir train_2\npath = \"train_2/\"\nparallel = Parallel(n_jobs=N, backend=\"threading\")\nparallel(delayed(save)(ids, path) for ids in train_ids_2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"!zip -r train_2.zip train_2\n!rm -rf train_2","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"!mkdir train_3\npath = \"train_3/\"\nparallel = Parallel(n_jobs=N, backend=\"threading\")\nparallel(delayed(save)(ids, path) for ids in train_ids_3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"!zip -r train_3.zip train_3\n!rm -rf train_3","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"!mkdir train_4\npath = \"train_4/\"\nparallel = Parallel(n_jobs=N, backend=\"threading\")\nparallel(delayed(save)(ids, path) for ids in train_ids_4)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"!zip -r train_4.zip train_4\n!rm -rf train_4","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"!mkdir train_5\npath = \"train_5/\"\nparallel = Parallel(n_jobs=N, backend=\"threading\")\nparallel(delayed(save)(ids, path) for ids in train_ids_5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"!zip -r train_5.zip train_5\n!rm -rf train_5","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}