{"cells":[{"metadata":{},"cell_type":"markdown","source":"
","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Introduction\n\nWelcome to the \"Cornell Birdcall Identification\" challenge on Kaggle! In this challenge, contestants need to identify the species of birds involved in audio clips of them calling. In this kernel, I will build a comprehensive pipeline based on melspectrogram audio transformation and the ResNet pretrained image model to identify bird species from audio samples. I'm very excited about this challenge because I love birds and I'm dealing with audio data for the first time, so let's get straight to it!","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Acknowledgements\n\n1. [LibROSA](https://librosa.org/librosa/) ~ by the librosa team\n2. [Audio Data Analysis Using librosa 📈](https://www.kaggle.com/hamditarek/audio-data-analysis-using-librosa) ~ by Tarek Hamdi\n3. [Understanding the Mel Spectrogram](https://medium.com/analytics-vidhya/understanding-the-mel-spectrogram-fca2afa2ce53) ~ by Leland Roberts\n4. [Bidirectional LSTM for audio labeling with Keras](https://www.kaggle.com/carlolepelaars/bidirectional-lstm-for-audio-labeling-with-keras) ~ by Carlo Lepelaars","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Contents\n\n1. [Preparing the ground](#1)\n * [Install additional packages](#1.1)\n * [Import necessary libraries](#1.2)\n * [Define key hyperparameters and paths](#1.3)\n * [Load metadata from .csv files](#1.4)\n\n\n2. [EDA](#2)\n * [ebird_code](#2.1)\n * [duration](#2.2)\n * [pitch](#2.3)\n * [channels](#2.4)\n\n \n3. [Modeling](#3)\n * [Data processing](#3.1)\n * [Define PyTorch data loading pipeline](#3.2)\n * [Define PyTorch modelling pipeline](#3.3)\n * [Define categorical cross entropy and accuracy](#3.4)\n * [Train the model on GPU](#3.5)\n * [Define PyTorch inference pipeline](#3.6)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Preparing the ground \n\nIn this section, we will prepare the ground to train and test the model by installing packages, setting hyperparameters, and loading the data.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Install additional packages \n\n* We will now install two other packages: colored and pydub\n* colored helps add foreground colors to text output in Python to make training logs more readable.\n* pydub will help us load audio data from .mp3 files much faster than the librosa command: librosa.load","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install -q pydub\n!pip install -q colored","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Import necessary libraries \n\n* Now, we import all the libraries we need.\n* colored, matplotlib and tqdm for data analysis and visualization.\n* numpy, pandas, and sklearn for data processing and manipulation.\n* librosa, pydub, keras and torch for model training and inference.","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport gc\nimport cv2\nimport time\nimport numpy as np\nimport pandas as pd\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\nimport IPython\nimport IPython.display as ipd\nfrom tqdm.notebook import tqdm\nimport matplotlib.pyplot as plt\nfrom colored import fg, bg, attr\nfrom sklearn.utils import shuffle\n\nimport plotly.express as px\nimport plotly.graph_objects as go\nimport plotly.figure_factory as ff\nfrom plotly.subplots import make_subplots\n\nimport pydub\nimport librosa\nimport librosa.display\nfrom pydub import AudioSegment as AS\nfrom librosa.feature import melspectrogram\nfrom librosa.core import power_to_db as ptdb\n\nimport torch\nimport torch.nn as nn\nfrom torch.optim import Adam\nfrom albumentations import Normalize\nfrom torchvision.models import resnet34\nfrom torch.utils.data import Dataset, DataLoader\nfrom torch import FloatTensor, LongTensor, DoubleTensor\n\nfrom keras.utils import to_categorical\nfrom keras.preprocessing.sequence import pad_sequences as pad","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"torch.backends.cudnn.benchmark = True","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Define key hyperparameters and paths \n\n* Here we define the key hyperparameters: sequence length, train/val split, batch size, epochs, LR.\n* We also specify the correct paths for loading data, training, inference, and finally submission to this competition.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Define hyperparameters\n\nThe hyperparameters used are documented below ~\n\n1. Data processing related\n\n * N_MELS is the number of melspectrogram features per time step.\n * MEL_LEN is the total number of time steps in each melspectrogram.\n * AMPLITUDE represents the default signal amplitude applied to unreadable files.\n * SR is the sampling rate at which the audio is loaded (readings per second). It defaults to 44100 Hz.\n * TSR is the sampling rate at which the test audio clips are loaded. It defaults to 32000 Hz.\n * MAXLEN is the maximum number of readings. Longer clips will be trimmed and shorter ones will be padded.\n * SPLIT represents the fraction of data to be used for training. The rest of the data is used for validation.\n * CHUNKS represents the number of chunks that will be extracted from each signal. It defaults to 1 per signal.\n * CHUNK_SIZE represents the sequence length of each audio chunk to be fed into the melspectrogram function.\n * POP_FRAC is the maximum proportion of signal information to be ignored per chunk (defaults to 0.25).\n\n\n2. Modelling related\n \n * F represents the number of total number of features output by the ResNet model (defaults to 512).\n * DROP represents the dropout rate between the feature layer and final 264D output layer (0.2).\n * LEARNING_RATE represents the learning rate of the model. It defaults to a low value of 1e-3.\n * BATCH_SIZE is the size of each minibatch to be fed into the model while training (defaults to 64).\n * VAL_BATCH_SIZE is the size of each minibatch to be fed into the model while validating (defaults to 64).\n * EPOCHS represents the number of times the training dataset is fed into the model (defaults to 100).\n\n \n3. Submission related\n\n * MAX_OUTPUTS is the maximum number of bird species to be output per audio clip (defaults to 3).\n * THRESHOLD is the minimum probability required for a bird species to be selected (defaults to 6e-3).\n * MIN_THRESHOLD is the minimum probability required in a list of probabilities to assign any bird species.\n\n (*Note*: If the maximum probability falls below MIN_THESHOLD, nocall will be predicted.)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"F = 512\nDROP = 0.2\nEPOCHS = 100\nLR = 1e-3, 1e-2\nVAL_BATCH_SIZE = 64\nTRAIN_BATCH_SIZE = 64\n\nMAX_OUTPUTS = 3\nTHRESHOLD = 6e-3\nMIN_THRESHOLD = 5e-3\n\nSR = 44100\nCHUNKS = 1\nTSR = 32000\nSPLIT = 0.8\nN_MELS = 256\nMEL_LEN = 313\nPOP_FRAC = 0.25\nMAXLEN = 2000000\nAMPLITUDE = 1000\nCHUNK_SIZE = 160000","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Check data available\n\nWe have three datasets at our disposal\n\n1. birdsong-recognition\n2. prepare-check-dataset\n3. birdcall-identification-spectrogram-loader","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"os.listdir('../input')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Unzip .zip files wth spectrogram images","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"!unzip ../input/birdcall-identification-spectrogram-loader/train_1.zip\n!unzip ../input/birdcall-identification-spectrogram-loader/train_2.zip\n!unzip ../input/birdcall-identification-spectrogram-loader/train_3.zip\n!unzip ../input/birdcall-identification-spectrogram-loader/train_4.zip\n!unzip ../input/birdcall-identification-spectrogram-loader/train_5.zip","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Define paths\n\nThe paths used are documented below ~\n\n1. Metadata related\n\n * ../input/birdsong-recognition/test.csv contains the test metadata used for submission.\n * ../input/birdsong-recognition/train.csv contains the train metadata used for submission.\n * ../input/prepare-check-dataset/test.csv contains the test metadata used for committing.\n\n\n2. Audio data related\n\n * ../input/birdsong-recognition/test_audio contains the test audio used for submission.\n * ../input/birdsong-recognition/train_audio contains the train audio used for submission.\n * ../input/prepare-check-dataset/test_audio contains the test audio used for committing.\n\n\n3. Spectrogram related\n\n * IMG_PATHS contains a list of folders with the already generated training spectrograms.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"TEST_DATA_PATH = '../input/birdsong-recognition/test.csv'\nTRAIN_DATA_PATH = '../input/birdsong-recognition/train.csv'\nTEST_AUDIO_PATH = '../input/birdsong-recognition/test_audio/'\nTRAIN_AUDIO_PATH = '../input/birdsong-recognition/train_audio/'\nCHECKING_PATH = '../input/prepare-check-dataset/birdcall-check/'\nIMG_PATHS = ['train_1', 'train_2', 'train_3', 'train_4', 'train_5']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"PATH_DICT = {}\nfor folder_path in tqdm(IMG_PATHS):\n for img_path in os.listdir(folder_path):\n PATH_DICT[img_path] = folder_path + '/'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub = os.path.exists(TEST_AUDIO_PATH)\nTEST_DATA_PATH = TEST_DATA_PATH if sub else CHECKING_PATH + 'test.csv'\nTEST_AUDIO_PATH = TEST_AUDIO_PATH if sub else CHECKING_PATH + 'test_audio/'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load metadata from .csv files \n\n* Now we load the training and testing metadata.\n\n* We can see that the testing dataframe has only 3 rows in it.\n\n* This is only a dummy test dataframe. The actual testing data will be used during submission.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df = pd.read_csv(TEST_DATA_PATH)\ntrain_df = pd.read_csv(TRAIN_DATA_PATH)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Prepare the label dictionary\n\n* Next we prepare a dictionary linking each bird species to a unique integer.\n* This dictionary will help us when we need to one-hot encode our targets later.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"keys = set(train_df.ebird_code)\nvalues = np.arange(0, len(keys))\ncode_dict = dict(zip(sorted(keys), values))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# EDA \n\nNow let us carry out some basic visualization to get a better idea of the data.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## ebird_code \n\nThe ebird_code is a code assigned to each bird species.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"counts = [list(train_df.ebird_code).count(code) for code in set(train_df.ebird_code)]\nidx = np.argsort(counts)\ndf = pd.DataFrame(np.transpose([np.array(list(set(train_df.ebird_code)))[idx], np.array(counts)[idx]]))\ndf.columns = [\"Bird species\", \"Count\"]\nfig = px.bar(df, x=\"Count\", y=\"Bird species\", title=\"Bird species vs. Count\", template=\"simple_white\")\nfig.data[0].orientation = 'h'\nfig.update_layout(height=1800, paper_bgcolor=\"#edebeb\")\nfig.update_traces(textfont=dict(\n color=\"white\"\n ))\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the above plot, we can see that the maximum number of audio samples per species in the training data is 100. There are many species with less than 100 samples in the training data, such as redhea, buffle, and coshum with 9, 15, and 19 samples respectively.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## duration \n\nThe duration is the length of the audio clip in seconds.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"nums_1 = train_df.duration\nnums_1 = nums_1.fillna(nums_1.mean())\n\nfig = ff.create_distplot(hist_data=[nums_1],\n group_labels=[\"0\"],\n colors=[\"seagreen\"], show_hist=False)\n\nfig.update_layout(title_text=\"Duration distribution\", xaxis_title=\"Duration\",\n template=\"plotly_white\", paper_bgcolor=\"#edebeb\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the distribution plot above, we can see that most clips last between 5 to 35 seconds. Overall, the distribution has a very clear positive/rightward skew as the probability density narrows down towards the right.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## pitch \n\nThe pitch describes the change in frequency of the bird's call over time (increasing, descreasing, etc).","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"counts = [list(train_df.pitch).count(code) for code in set(train_df.pitch)]\ndf = pd.DataFrame(np.transpose([list(set(train_df.pitch)), counts]))\ndf.columns = [\"Pitch\", \"Count\"]\nfig = px.pie(df, names=\"Pitch\", values=\"Count\", title=\"Pitch\", color_discrete_sequence=list(reversed(px.colors.cyclical.Edge)))\nfig.update_layout(paper_bgcolor=\"#edebeb\")\nfig.update_traces(textfont=dict(\n color=\"white\"\n ))\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the pie chart above, we can see that most audio clips do not have a specified pitch. The level and both pitches are second in frequency to not specified, followed by decreasing and increasing.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## channels \n\nThe channels indicates the number of channels in the audio clip (1: mono or 2: stereo).","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"counts = [list(train_df.channels).count(code) for code in set(train_df.channels)]\ndf = pd.DataFrame(np.transpose([list(set(train_df.channels)), counts]))\ndf.columns = [\"Channels\", \"Count\"]\ncolors = px.colors.qualitative.Plotly\nfig = px.pie(df, names=\"Channels\", values=\"Count\", title=\"Channels\", color_discrete_sequence=[\"darkblue\", \"SteelBlue\"])\nfig.update_layout(paper_bgcolor=\"#edebeb\")\nfig.update_traces(textfont=dict(\n color=\"white\"\n ))\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It can be seen that a somewhat even number of audio clips are mono (53.7%) stereo (46.3%). In this project, we deal with this by concatenating the multiple channels into a single signal.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"
","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Modelling \n\nNow it's time to start with the process of building our audio classification pipeline.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Data processing \n\nThe first step to define important function to process the data and generate features.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Define utility function to read audio (unhide code)\n\n* Now we define a function using pydub to read audio files into numpy arrays.\n* This implementation is significantly faster than librosa.load and torchaudio.load.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def normalize(x):\n return np.float32(x)/2**15\n\ndef read(file, norm=False):\n try:\n a = AS.from_mp3(file)\n a = a.set_frame_rate(TSR)\n except:\n return TSR, np.zeros(MAXLEN)\n\n y = np.array(a.get_array_of_samples())\n if a.channels == 2: y = y.reshape((-1, 2))\n if norm: return a.frame_rate, normalize(y)\n if not norm: return a.frame_rate, np.float32(y)\n\ndef write(file, sr, x, normalized=False):\n birds_audio_bitrate, file_format = '320k', 'mp3'\n ch = 2 if (x.ndim == 2 and x.shape[1] == 2) else 1\n y = np.int16(x * 2 ** 15) if normalized else np.int16(x)\n song = AS(y.tobytes(), frame_rate=sr, sample_width=2, channels=ch)\n song.export(file, format=file_format, bitrate=birds_audio_bitrate)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Define helper function to convert NumPy arrays to PyTorch tensors\n\n* This is a function that takes a list of numpy arrays as input and outputs a list of torch tensors.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def to_tensor(data):\n return [FloatTensor(point) for point in data]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Define functions to process audio signals (unhide code)\n\nThese are a set of functions which process the audio before the melspectrogram transformation.\n\nThe functions used are documented below ~\n\n* get_idx selects the start and end index of a given audio chunk.\n* get_chunk takes indices from get_idx and outputs a chunk of data between those indices.\n* get_len is a helper function which is used to decide possible chunk indices based on POP_FRAC.\n \n --> If the signal is longer than MAXLEN, it sets the maximum index to MAXLEN.
\n --> Else it uses POP_FRAC to ensure chunks are centered around audio signal and not padding.\n\n\n* get_signal flattens the signal, pads it to MAXLEN, and stacks multiple chunks into one array.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def get_idx(length):\n length = get_len(length)\n max_idx = MAXLEN - CHUNK_SIZE\n idx = np.random.randint(length + 1)\n chunk_range = idx, idx + CHUNK_SIZE\n chunk_idx = max([0, chunk_range[0]])\n chunk_idx = min([chunk_range[1], max_idx])\n return (chunk_idx, chunk_idx + CHUNK_SIZE)\n\ndef get_len(length):\n if length > MAXLEN: return MAXLEN\n if length <= MAXLEN: return int(length*POP_FRAC)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def get_chunk(data, length):\n index = get_idx(length)\n return data[index[0]:index[1]]\n\ndef get_signal(data):\n length = max(data.shape)\n data = data.T.flatten().reshape(1, -1)\n data = np.float32(pad(data, maxlen=MAXLEN).reshape(-1))\n return [get_chunk(data, length) for _ in range(CHUNKS)]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Understanding melspectrogram\n\nTo understand the melspectrogram, one needs to first understand the fourier transform. In simple terms, it is a transformation that allows us to analyze the frequency content of a signal effectively. But unfortunately, the fourier transform requires the signal to be periodic to get optimal results.\n\nThis problem can be solved by applying the fourier transform on multiple windowed segments of the signals to capture the frequency content as it changes over time. This is called the short-time fourier transform or STFT for short. When the fourier transform is applied on overlapping windowed segments of the signal, what we get what is called a spectrogram.\n\nA spectrogram can be thought of as several fourier transforms stacked on top of each other. It is a way to visually represent a signal’s loudness, or amplitude, as it varies over time at different frequencies. The y-axis is converted to a log scale, and the color dimension is converted to decibels. This is because humans can only perceive a very small and concentrated range of frequencies and amplitudes.\n\nMoreover, studies have shown that humans do not perceive frequencies on a linear scale. We are better at detecting differences in lower frequencies than higher frequencies. For example, we can easily tell the difference between 500 and 1000 Hz, but we will hardly be able to tell a difference between 10,000 and 10,500 Hz, even though the distance between the two pairs are the same.\nIn 1937, Stevens, Volkmann, and Newmann proposed a unit of pitch such that equal distances in pitch sounded equally distant to the listener. This is called the mel scale. We perform a mathematical operation on frequencies to convert them to the mel scale.\n\nA spectrogram converted to the mel scale is called a melspectrogram and is a great way to convert audio signal data to a visual feature map to train an image model (like ResNet-34) on. Refer to [this article](https://medium.com/analytics-vidhya/understanding-the-mel-spectrogram-fca2afa2ce53) for an excellent explanation of melspectrogram. Now, since we understand how the melspectrogram works, let us move on to the modelling part.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Define functions to calculate melspectrogram features\n\nBelow we define some functions to calculate the melspectrogram features from audio signals.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def to_imagenet(X, mean=None, std=None, norm_max=None, norm_min=None, eps=1e-6):\n mean = mean or X.mean()\n X = X - mean\n std = std or X.std()\n Xstd = X / (std + eps)\n _min, _max = Xstd.min(), Xstd.max()\n norm_max = norm_max or _max\n norm_min = norm_min or _min\n if (_max - _min) > eps:\n # Normalize to [0, 255]\n V = Xstd\n V[V < norm_min] = norm_min\n V[V > norm_max] = norm_max\n V = (V - norm_min) / (norm_max - norm_min)\n else:\n # Just zero\n V = np.zeros_like(Xstd, dtype=np.uint8)\n return np.stack([V]*3, axis=-1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_melsp(data):\n melsp = melspectrogram(data, n_mels=N_MELS)\n return to_imagenet(librosa.power_to_db(melsp))\n\ndef get_melsp_img(data):\n data = get_signal(data)\n return np.stack([get_melsp(point) for point in data])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Visualize melspectrogram\n\nNext we visualize the melspectrogram feature maps for sample signals to get an better understanding. We can see that the melspectrogram contains visual information about the trends (frequency and amplitude) in the audio signal over time.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def viz(species):\n data = train_df.query('ebird_code == \"{}\"'.format(species))\n \n data = data.reset_index(drop=True)\n ebird_code, filename = data.ebird_code.loc[0], data.filename.loc[0]\n \n ipd.Audio(TRAIN_AUDIO_PATH + species + '/' + filename)\n\n f, pth = (30, 5), TRAIN_AUDIO_PATH\n fig, ax = plt.subplots(nrows=CHUNKS, ncols=2, figsize=f)\n\n ax[0].set_title(species + \" signal\", fontsize=16)\n ax[1].set_title(species + \" melspectrogram\", fontsize=16)\n\n sr, data = read(pth + ebird_code + '/' + filename)\n signals, melsp_features = get_signal(data), get_melsp_img(data)\n\n values = zip(signals, melsp_features)\n for i, (signal, melsp_feature) in enumerate(values):\n ax[0].plot(signal, 'crimson'); ax[1].imshow(cv2.resize(melsp_feature, (4096, 1024)))\n \n display(ipd.Audio(pth + ebird_code + '/' + filename))\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Aldfly","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"viz('aldfly')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Canware","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"viz('canwre')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Lesgol","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"viz('lesgol')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Define PyTorch data loading pipeline ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Define PyTorch dataset\n\nNow we need to define a PyTorch dataset to load data for the ResNet-34 model.\n\n* A PyTorch dataset has three fundamental functions: init, len, and getitem.\n* The init function initializes all the components required for data loading (paths, metadata, etc).\n* The len function simply returns the length of the dataset. This indicates the number of retrievable samples.\n* The getitem function returns a data point at a given index idx. The actual logic is written in this function.\n* The getitem function does 2 things: gets the one-hot targets and melspectrogram, and outputs them.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class BirdDataset(Dataset):\n def __init__(self, df, path):\n self.code_dict = code_dict\n self.classes = len(code_dict)\n self.df, self.path = df, path\n self.dataset_length = len(df)\n \n def __len__(self):\n return self.dataset_length\n \n def __getitem__(self, i):\n file_name = self.df.filename[i]\n ebird_code = self.df.ebird_code[i]\n num_code = self.code_dict[ebird_code]\n default_signal = np.random.random(MAXLEN)*AMPLITUDE\n default_values = SR, np.int32(np.round(default_signal))\n\n values = read(self.path + ebird_code + '/' + file_name)\n _, data = values if len(values) == 2 else default_values\n code = to_categorical([num_code], num_classes=self.classes)\n return to_tensor([np.nan_to_num(get_melsp_img(data)), np.repeat(code, CHUNKS, 0)])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Define second PyTorch dataset with pre-generated spectrograms\n\nWe also build another PyTorch dataset that uses already generated spectrograms to speed up training.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class MelDataset(Dataset):\n def __init__(self, df):\n self.aug = Normalize(p=1)\n self.code_dict = code_dict\n self.classes = len(code_dict)\n self.df, self.dataset_length = df, len(df)\n \n def __len__(self):\n return self.dataset_length\n \n def __getitem__(self, i):\n file_name = self.df.filename[i]\n image_name = file_name + '.jpg'\n ebird_code = self.df.ebird_code[i]\n num_code = self.code_dict[ebird_code]\n image = cv2.imread(PATH_DICT[image_name] + image_name)\n code = to_categorical([num_code], num_classes=self.classes)\n return to_tensor([self.aug(image=image)['image'], np.repeat(code, CHUNKS, 0)])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Split train/val (80/20) and declare PyTorch datasets and dataloders\n\n* Now we split the dataset into training (80%) and validation (20%) sets.\n* We then create separate datasets and dataloaders for training and validation.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = shuffle(train_df)\n\nsplit = int(SPLIT*len(train_df))\ntrain_df = train_df.reset_index(drop=True)\nvalid_df = train_df[split:].reset_index(drop=True)\ntrain_df = train_df[:split].reset_index(drop=True)\n\ntrain_set = MelDataset(train_df)\nvalid_set = MelDataset(valid_df)\nvalid_loader = tqdm(DataLoader(valid_set, batch_size=VAL_BATCH_SIZE))\ntrain_loader = DataLoader(train_set, batch_size=TRAIN_BATCH_SIZE, shuffle=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Define PyTorch modelling pipeline \n\nNow we define the model to classify melspectrograms into bird species.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"
","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Understanding ResNet\n\nTo understand ResNet, we first need to understand the process of convolution.\n\n### Convolution \n\nConvolution is a rather simple algorithm which involves a kernel (a 2D matrix) which moves over the entire image, calculating dot products with each window along the way. The GIF below demonstrates convolution in action.\n\n
\n\nThe above process can be summarized with an equation, where *f* is the image and *h* is the kernel. The dimensions of *f* are *(m, n)* and the kernel is a square matrix with dimensions smaller than *f*:\n\n
\n
\n\nIn the above equation, the kernel *h* is moving across the length and breadth of the image. The dot product of *h* with a sub-matrix or window of matrix *f* is taken at each step, hence the double summation (rows and columns). Convolution is very good at extracting features such as edges and corners from images. It is very popular in most modern computer vision applications (image classification, object detection, etc).\n\n### Back to ResNet\n\nResNet is essentially a deep neural network that uses convolution to generate features and accurately classify images. The key innovation in ResNet was skip connections, or shortcuts to jump over some layers. This helped combat the vanishing gradient problem while training deep CNNs and maximize performance in several tasks. This architecture was presented back in 2015 and caused a massive stir in the computer vision community with excellent performance on several common tasks at the time. For this project, we will use ResNet to extract features from melspectrograms and identify bird species.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Define ResNet model\n\n* We then build a simple model using a ResNet-34 backbone.\n* We first extract a 512D feature vector from the ResNet-34 model.\n* We then pass the vector through a Dropout + Dense head to get the logits for each class.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"class BirdNet(nn.Module):\n def __init__(self, f, o):\n super(BirdNet, self).__init__()\n self.dropout = nn.Dropout(p=DROP)\n self.dense_output = nn.Linear(f, o)\n self.resnet = resnet34(pretrained=True)\n self.resnet_head = list(self.resnet.children())\n self.resnet_head = nn.Sequential(*self.resnet_head[:-1])\n\n def forward(self, x):\n x = self.resnet_head(x)\n return self.dense_output(self.dropout(x.view(-1, F)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Declare model and optimizer\n\n* The next step is to declare the model and define an Adam optimizer to train the model.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"O = len(code_dict)\nnetwork = BirdNet(f=F, o=O)\noptimizer = Adam([{'params': network.resnet.parameters(), 'lr': LR[0]},\n {'params': network.dense_output.parameters(), 'lr': LR[1]}])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Define categorical cross entropy and accuracy \n\n* Next, we define our loss function (categorical cross entropy) and evaluation metric (accuracy).","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def cel(y_true, y_pred):\n y_true = torch.argmax(y_true, axis=-1)\n return nn.CrossEntropyLoss()(y_pred, y_true.squeeze())\n\ndef accuracy(y_true, y_pred):\n y_true = torch.argmax(y_true, axis=-1).squeeze()\n y_pred = torch.argmax(y_pred, axis=-1).squeeze()\n return (y_true == y_pred).float().sum()/len(y_true)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def print_metric(data, batch,\n epoch, start,\n end, metric, typ):\n\n t = typ, metric, \"%s\", data, \"%s\"\n if typ == \"Val\": pre = \"\\nEPOCH %s\" + str(epoch+1) + \"%s \"\n if typ == \"Train\": pre = \"BATCH %s\" + str(batch-1) + \"%s \"\n time = np.round(end - start, 1); time = \"Time: %s{}%s s\".format(time)\n fonts = [(fg(211), attr('reset')), (fg(212), attr('reset')), (fg(213), attr('reset'))]\n print(pre % fonts[0] + \"{} {}: {}{}{}\".format(*t) % fonts[1] + \" \" + time % fonts[2])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Train the model on GPU \n\n* Now, we train the model on Kaggle's NVIDIA Tesla P100 GPU.\n* First, we do a **training loop**, where we train the model with back-prop to adjust the parameters.\n* Second, we do a **validation loop**, to check the model's performance on unseen data after each epoch.\n* The training loop uses **forward-prop and back-prop**, while the validation loop uses **only forward-prop.**\n* We use the torch.no_grad() flag for the validation loop as no gradients are needed for forward-prop.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_shuffle_idx(tensor):\n return shuffle(np.arange(len(tensor)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"D = (3, N_MELS, MEL_LEN)\nnetwork = network.cuda()\ndevice = torch.device('cuda')\n\nstart = time.time()\nprint(\"STARTING TRAINING ...\\n\")\n\nfor epoch in range(EPOCHS):\n fonts = (fg(48), attr('reset'))\n print((\"EPOCH %s\" + str(epoch+1) + \"%s\") % fonts)\n \n batch = 1\n network.train()\n for minibatch in train_loader:\n train_X, train_y = minibatch\n train_y = train_y.view(-1, O)\n train_X = train_X.view(-1, *D)\n idx = get_shuffle_idx(train_X)\n train_X = train_X[idx].to(device)\n train_y = train_y[idx].to(device)\n train_preds = network.forward(train_X)\n train_loss = cel(train_y, train_preds)\n train_accuracy = accuracy(train_y, train_preds)\n \n optimizer.zero_grad()\n train_loss.backward()\n\n optimizer.step()\n end = time.time()\n batch = batch + 1\n is_print = batch % 100 == 1\n acc = np.round(train_accuracy.item(), 3)\n if is_print: print_metric(acc, batch, 0, start, end, \"Acc\", \"Train\")\n \n valid_loss = 0\n valid_points = 0\n valid_accuracy = 0\n \n network.eval()\n with torch.no_grad():\n for minibatch in valid_loader:\n valid_X, valid_y = minibatch\n valid_y = valid_y.view(-1, O)\n valid_X = valid_X.view(-1, *D)\n idx = get_shuffle_idx(valid_X)\n valid_X = valid_X[idx].to(device)\n valid_y = valid_y[idx].to(device)\n valid_preds = network.forward(valid_X)\n valid_points = valid_points + len(valid_y)\n valid_loss += cel(valid_y, valid_preds).item()*len(valid_y)\n valid_accuracy += accuracy(valid_y, valid_preds).item()*len(valid_y)\n \n end = time.time()\n valid_loss /= valid_points\n valid_accuracy /= valid_points\n acc = np.round(valid_accuracy, 3)\n print_metric(acc, 0, epoch, start, end, \"Acc\", \"Val\"); print(\"\")\n \nprint(\"ENDING TRAINING ...\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Define PyTorch inference pipeline \n\nNow we define the datasets and functions required to perform inference and make a submission.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Define PyTorch test dataset\n\nNext we define the dataset for retrieving data samples for inference.\n\n* The init function initializes all the components required for data loading (paths, metadata, etc).\n* The len function simply returns the length of the dataset. This indicates the number of retrievable samples.\n* The getitem function returns a data point at a given index idx. The actual logic is written in this function.\n* The getitem function gets the audio clip, crops it to a given range and outputs the melspectrogram at index i.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_time(site, start_time):\n if site == 'site_3': return 0, None\n if site == 'site_1' or site == 'site_2':\n return int((start_time - 5)*TSR), int(start_time*TSR)\n\nclass BirdTestDataset(Dataset):\n def __init__(self, df, path):\n self.df, self.path = df, path\n self.dataset_length = len(df)\n self.normalize = Normalize(p=1)\n \n def __len__(self):\n return self.dataset_length\n \n def __getitem__(self, i):\n site = self.df.site[i]\n start_time = self.df.seconds[i]\n s, e = get_time(site, start_time)\n default_values = TSR, np.zeros(MAXLEN)\n values = read(self.path + self.df.audio_id[i])\n out = values if len(values) == 2 else default_values\n return FloatTensor(self.normalize(image=get_melsp_img(out[1][s:e]))['image'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Define functions for decision making\n\nNow we define functions required to convert predictions into a final submission.\n\n* softmax performs the well-known softmax function on logits to get probabilities.\n* decision_fn takes a list of probabilities and outputs predicted bird species using simple thresholding.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def softmax(x):\n return np.exp(x)/np.sum(np.exp(x), axis=1)[:, None]\n\ndef decision_fn(probs):\n probs[np.argsort(probs)[:-MAX_OUTPUTS]] = 0\n if np.max(probs) < MIN_THRESHOLD: return [-1]\n condition = probs == np.max(probs), probs >= THRESHOLD\n return np.where(np.logical_or(*condition))[0].tolist()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Run the inference loop\n\nNow we define the test dataloader and run the inference loop over the training audio clips.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"network.eval()\ntest_preds = []\ntest_set = BirdTestDataset(test_df, TEST_AUDIO_PATH)\ntest_loader = DataLoader(test_set, batch_size=VAL_BATCH_SIZE)\n\nif os.path.exists(TEST_AUDIO_PATH):\n for test_X in tqdm(test_loader):\n test_pred = network.forward(test_X.view(-1, *D).to(device))\n test_preds.extend(softmax(test_pred.detach().cpu().numpy()).flatten())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Convert model predictions to bird species\n\n* We use a simple averaging function over CHUNKS to get final predictions for each test sample.\n* Next, we convert the test predictions into lists of bird species predictions using decision_fn.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"code_dict[-1] = 'nocall'\nkeys = code_dict.keys()\nvalues = code_dict.values()\ncode_dict = dict(zip(values, keys))\n\nif os.path.exists(TEST_AUDIO_PATH):\n test_preds = np.array(test_preds)\n test_preds = test_preds.reshape(-1, CHUNKS, O).mean(axis=1)\n test_preds = [decision_fn(test_pred) for test_pred in test_preds]\n test_preds = [[code_dict[pred] for pred in test_pred] for test_pred in test_preds]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Prepare submission\n\nWe finally load the sample_submission and add our predictions.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.read_csv('../input/birdsong-recognition/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if os.path.exists(TEST_AUDIO_PATH):\n submission.birds = [' '.join(pred) for pred in test_preds][:len(submission)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission.head(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Make submission and save trained model\n\n* To wrap it all up, we export our predictions to the .csv format expected by the competition.\n* We save our trained model for future use. With this, we can avoid training the model repeatedly.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"submission.to_csv('submission.csv', index=False)\ntorch.save(network.cpu().state_dict(), 'resnet.pt')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### This is my first audio classification project. I hope you enjoyed it and found it useful!","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}