{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Kaggle BirdCLEF 2021\n\n## Data Preparation\nBefore we get working on concrete algorithms, let's load our data and take a look at it to see what data cleaning is necessary.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\n\ninput_folder = '/kaggle/input/birdclef-2021/'\noutput_folder = '/kaggle/working/'\n\n# Python f-strings allow us to do variable interpolation\ntest_soundscapes_folder = f\"{input_folder}test_soundscapes/\"\ntrain_short_audio_folder = f\"{input_folder}train_short_audio/\"\ntrain_soundscapes_folder = f\"{input_folder}train_soundscapes/\"\n\nsample_submission = pd.read_csv(f\"{input_folder}sample_submission.csv\")\ntest = pd.read_csv(f\"{input_folder}test.csv\")\ntrain_metadata = pd.read_csv(f\"{input_folder}train_metadata.csv\")\ntrain_soundscape_labels = pd.read_csv(f\"{input_folder}train_soundscape_labels.csv\")","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, let's take at the first few rows of our training metadata set to see what kind of data we are working with.","metadata":{}},{"cell_type":"code","source":"train_metadata.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see by inspecting the columns that the \"secondary labels\" and \"type\" attributes are multi-valued. This means that our data set is not currently [normalized](https://en.wikipedia.org/wiki/Database_normalization). To address this and thus make working with our data set a bit easier further down the line, let's normalize our data set into [first normal form](https://en.wikipedia.org/wiki/Database_normalization#Satisfying_1NF). Let's first create one DataFrame `filename_metadata` containing only the atomic attributes currently in our DataFrame and split off the \"secondary labels\" and \"type\" attributes into separate DataFrames, making sure to strip the brackets from the strings in the 'secondary_labels' and 'type' columns.","metadata":{}},{"cell_type":"code","source":"import re # to be able to use regular expressions\n\ndef remove_brackets_quotes_spaces(string: str) -> str:\n    return re.sub(r'[\\[\\]\\' ]', '', string)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"filenames = train_metadata['filename']\nfile_metadata = train_metadata.drop(columns=['secondary_labels', 'type']).rename(columns={'primary_label': 'primary label'})\n# Note that indexing into train_metadata with a column name will return a Pandas Series object\nsecondary_birdcall_labels = train_metadata['secondary_labels'].map(remove_brackets_quotes_spaces)\nbirdsound_labels = train_metadata['type'].map(remove_brackets_quotes_spaces)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Great! Now, let's [explode](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.explode.html) the `secondary_birdcall_labels` and `birdsound_labels` DataFrames so that every valid tuple of (filename, secondary birdcall label) and likewise every tuple of (filename, birdsound label) get expanded into distinct rows within their respective DataFrames. However, we will need to convert the secondary birdcall labels and birdsound labels into iterable Python objects so that we can actually apply the `explode` method.","metadata":{}},{"cell_type":"code","source":"from typing import Sequence # to specify a parameter as a sequence type\n\ndef convert_to_list(string: str) -> Sequence[str]:\n    \"\"\"Turns a comma-delimited string into a list\n\n    Parameters\n    ----------\n    string: str\n        A comma-delimited string\n\n    Returns\n    -------\n    str\n        A list containing each of the words from the original string\"\"\"\n    return re.split(r'\\W+', string)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"secondary_birdcall_labels = secondary_birdcall_labels.map(convert_to_list)\nbirdsound_labels = birdsound_labels.map(convert_to_list)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Looking good so far. Now, let's convert `secondary_birdcall_labels` and `birdsound_labels` into actual DataFrames and then apply the `explode` method on them.","metadata":{}},{"cell_type":"code","source":"secondary_birdcall_labels = pd.concat([filenames, secondary_birdcall_labels], axis=1).rename(columns={'secondary_labels': 'secondary label'})\nbirdsound_labels = pd.concat([filenames, birdsound_labels], axis=1).rename(columns={'type': 'birdsound label'})\n\n# Pass the option ignore_index=True so that the original indices aren't duplicated in the process of expanding the DataFrame\nsecondary_birdcall_labels = secondary_birdcall_labels.explode('secondary label', ignore_index=True)\nbirdsound_labels = birdsound_labels.explode('birdsound label', ignore_index=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Finally, let's print out the `secondary_birdcall_labels` DataFrame to make sure it looks reasonable.","metadata":{}},{"cell_type":"code","source":"secondary_birdcall_labels.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It seems like we've forgotton to remove rows that have no secondary label. Let's fix that.","metadata":{}},{"cell_type":"code","source":"# Use the reindex() method to renumber the rows after selecting a subset of the original DataFrame\n# We pass the option drop=True in order to avoid having the old row indices being added as a column to the DataFrame\nsecondary_birdcall_labels = secondary_birdcall_labels[secondary_birdcall_labels['secondary label'] != ''].reset_index(drop=True)\nsecondary_birdcall_labels.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Fantastic! Let's do the same with our birdsound_labels DataFrame.","metadata":{}},{"cell_type":"code","source":"birdsound_labels = birdsound_labels[birdsound_labels['birdsound label'] != ''].reset_index(drop=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Finally, let's clean up the `filename_metadata` DataFrame to remove the attributes we don't need. Let's take a have quick refresher of the attributes in the DataFrame.","metadata":{}},{"cell_type":"code","source":"file_metadata.columns","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The \"license\" and \"url\" fields will certainly not help with our prediction task. The \"common_name\" and \"scientific_name\" fields will similarly be not too helpful since our target labels will be found with the \"primary_label\" attribute. The \"author\" attribute may indeed exhibit some non-negligible correlation with the birdcall labels (i.e. some authors may be interested in observing certain birds over others), but that will not help us in our goal of creating a machine learning model capable of identifying birdcalls strictly from soundscape recordings.\n\nThe \"rating\" attribute should definitely be preserved as it can help us focus on training on the recordings that have the best audio quality. The \"latitude\", \"longitude\" and \"date\" attributes will also be of importance as those attributes will also be present along with the test data during our notebook submission.\n\nFinally, let's also make sure to split off the \"primary label\" attribute into its own DataFrame and also make a DataFrame combining the primary and secondary labels into a single data set. This will be convenient for us later.\n\nAll that said, let's finish tidying up the data.","metadata":{}},{"cell_type":"code","source":"file_metadata = file_metadata.drop(columns=['scientific_name', 'common_name', 'author', 'license', 'url'])\n\n# Let's create the following DataFrame for our future convenience\nprimary_birdcall_labels = file_metadata[['filename', 'primary label']]\n\nfile_metadata = file_metadata.drop(columns=['primary label'])\n\n# https://pandas.pydata.org/docs/getting_started/comparison/comparison_with_sql.html#group-by\nbirdcall_labels = pd.concat([primary_birdcall_labels.rename(columns={'primary label': 'birdcall label'}), secondary_birdcall_labels.rename(columns={'secondary label': 'birdcall label'})])\n\ndel train_metadata","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Exploratory Data Analysis\n\nOur next step towards training a good model for identifying bird calls will be to get a good understanding of the data that we are working with, so let's visualize our data to see if we can gain any insights that will help us in training our models.  ","metadata":{}},{"cell_type":"code","source":"# How many unique bird species are there in this data set?\nNUM_CLASSES = len(pd.unique(birdcall_labels['birdcall label']))\nNUM_CLASSES","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Before moving on, we need to make sure that `secondary_bircall_labels` DataFrame not contain a label that is not found in the `primary_birdcall_labels` DataFrame","metadata":{}},{"cell_type":"code","source":"set(pd.unique(secondary_birdcall_labels['secondary label'])) - set(pd.unique(primary_birdcall_labels['primary label']))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"That's unexpected. Let's see how many files have \"rocpig1\" as a label.","metadata":{}},{"cell_type":"code","source":"birdcall_labels[birdcall_labels['birdcall label'] == 'rocpig1'].shape[0]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"How many tuples of (filename, birdcall label) are there total?","metadata":{}},{"cell_type":"code","source":"birdcall_labels.shape[0]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Since there are only 9 instances of a file being associated with the label \"rocpig1\" among the many tens of thousands of training examples, we will ignore it for the purposes of making our model.","metadata":{}},{"cell_type":"code","source":"import numpy as np # for working with arrays and numerical data\nimport seaborn as sns # from plotting functionality\n\n# What is the distribution of each of the bird species found in this data set?\nspecies_distribution = birdcall_labels.groupby('birdcall label')['birdcall label'].count().sort_values(ascending=False).values\nplt = sns.lineplot(x=np.arange(NUM_CLASSES), y=species_distribution)\nplt.set_xlabel('Species (by index)')\nplt.set_ylabel('Count')\nplt.set_title('Count by Species'); # suppressing standard output in an IPython notebook: https://stackoverflow.com/questions/42635806/how-to-remove-matplotlib-output-lines-from-showing-in-jupyter-notebook-when-plot","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Since our training set is quite large, we'll choose the most frequent 50 classes as the cut-off point for training our models. This will certainly not yield the best performance, but it will give us a reasonable strating point for exploring different ideas. Let's also take this opportunity to trim our training set down to the examples that have only a primary birdcall label and no secondary birdcall label.","metadata":{}},{"cell_type":"code","source":"# Create a DataFrame with all files that don't have a secondary birdcall label\nsingular_birdcall_files = pd.DataFrame({'filename': list(set(primary_birdcall_labels['filename']) - set(secondary_birdcall_labels['filename']))})\n\n# Merge the resulting DataFrame with the file metadata DataFrame by joining on the \"filename\" attribute\nsingular_birdcall_file_metadata = pd.merge(singular_birdcall_files, file_metadata, on='filename')\n# Merge with the primary birdcall labels DataFrame by joining on the \"filename\" attribute\nsingular_birdcall_file_metadata = pd.merge(singular_birdcall_file_metadata, primary_birdcall_labels, on='filename')\n\n# Identify the top fifty most frequently occurring birdcall classes\nmost_common_birdcalls = singular_birdcall_file_metadata.groupby('primary label')['primary label'].count()\nmost_common_birdcalls = pd.DataFrame({'primary label': most_common_birdcalls.index, 'Count': most_common_birdcalls.values})\nmost_common_birdcalls = most_common_birdcalls.sort_values(by='Count', ascending=False).head(50).drop(columns=['Count'])\n\ntraining_files = most_common_birdcalls.merge(primary_birdcall_labels, on='primary label')\n# Merge on all key attributes in order to avoid duplicating columns\ntraining_files = singular_birdcall_file_metadata.merge(training_files, on=['filename', 'primary label']).rename(columns={'primary label': 'birdcall label'}) # Reference: https://www.pauldesalvo.com/how-to-remove-or-prevent-duplicate-columns-from-a-pandas-merge/","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, let us identify the highest audio quality recording to which we can restrict the audio samples of our training set while still retaining at least one example for every class of birdcall.","metadata":{}},{"cell_type":"code","source":"minmax_audio_rating = np.min(training_files.groupby('birdcall label')['rating'].agg(np.max))\nminmax_audio_rating","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Great! Now, let's assure ourselves that there are still a sufficient number of samples from each of the birdcall classes if we are to restrict ourselves to only using audio samples of rating 5.0.","metadata":{}},{"cell_type":"code","source":"high_quality_training_files = training_files[np.isclose(training_files['rating'], 5.0)]\nmin_audio_sample_count = np.min(high_quality_training_files.groupby('birdcall label')['birdcall label'].count())\nmin_audio_sample_count","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Well if that isn't an auspicious number! It seems we'll have enough audio samples from every birdcall class in order to train, at the least, some moderately performant models. Let's go ahead and update our training set to use only the highest quality samples.","metadata":{}},{"cell_type":"code","source":"training_files = high_quality_training_files.drop(columns=['rating'])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"One last thing. Let's replace our \"date\" and \"time\" attributes with a single, normalized \"timestamp\" attribute.","metadata":{}},{"cell_type":"code","source":"from sklearn.preprocessing import normalize\n\ndates = training_files['date'].values\ntimes = training_files['time'].values\n\n# Keep track of which files have dates and times that cannot be parsed so that we can filter them from the data set.\ninvalid_date_indices = []\n\n# In case the dates and times cannot be parseed\nfor i, (date, time) in enumerate(zip(dates, times)):\n    try:\n        pd.Timestamp(f\"{date} {time}\")\n    except:\n        invalid_date_indices.append(i)\n\nvalid_date_indices = list(set(range(training_files.shape[0])) - set(invalid_date_indices))\n\ntraining_files = training_files.iloc[valid_date_indices, :]\ndates = dates[valid_date_indices]\ntimes = times[valid_date_indices]\n\ntimestamps = list(map(lambda pair: [pd.Timestamp(f\"{pair[0]} {pair[1]}\").timestamp()], zip(dates, times)))\ntimestamps = normalize(np.array(timestamps).reshape(-1, 1), axis=0).reshape(-1)\ntimestamps = pd.DataFrame({'timestamp': timestamps}, index=training_files.index)\n\ntraining_files = pd.concat([training_files.drop(columns=['date', 'time']), timestamps], axis=1)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, before we move on, let's just do a sanity check to make sure we still have a sufficient number of examples from each of the birdcall classes.","metadata":{}},{"cell_type":"code","source":"min_audio_sample_count = np.min(training_files.groupby('birdcall label')['birdcall label'].count())\nmin_audio_sample_count","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training Setup","metadata":{}},{"cell_type":"markdown","source":"Now that our data is relatively tidied up, we can now focus on preparing it for model training. Our only goal will be to create an appropriate train and cross-validation split using our training data. Ideally, we would create a separate training sample from every possible disjoint five-second long segment from each of our files (or perhaps even every ovelapping five-second long segment using a stride of one second over each file), but, due to our limited time and memory constraints, we will limit ourselves to treating an entire file as a single training example.","metadata":{}},{"cell_type":"code","source":"# Check the current number of training files\ntraining_files.shape[0]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\nX = training_files\ny = training_files['birdcall label']\n\n# Pass the train_size=0.70 argument to get a roughly 70-30 split of train and cross-validation data and pass the stratify=y argument to get a similar distribution of birdcall labels between the train and cross-validation sets\nX_train, X_cv = train_test_split(X, train_size=0.70, stratify=y)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# See how many examples are in our training set\nX_train.shape[0]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Helper Functions\n\nLet's do some preparatory work by defining a couple of functions that will be useful as we move on to training our models.","metadata":{}},{"cell_type":"code","source":"import random\n\nRANDOM_SEED = 1234\n# Set the seed for the random number generator provided by the random library\nrandom.seed(RANDOM_SEED)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import soundfile as sf # for working with the .ogg audio files\n\ndef read_ogg_data(filename: str, birdcall_label: str) -> np.ndarray:\n    \"\"\"Returns the audio data from a specified file as a NumPy array\n\n    Parameters\n    ----------\n    filename: str\n        The name of the file containing the audio signal to be returned\n    birdcall_label: str\n        The birdcall label of the specified file; used to determine the parent folder of the specified file within the train_short_audio_folder\n\n    Returns\n    -------\n    numpy.ndarray\n        A NumPy array representing the audio signal\n    \"\"\"\n    # Use the soundfile library to read in the audio data\n    data, _ = sf.read(f\"{train_short_audio_folder}{birdcall_label}/{filename}\")\n    # Reshape `data` into two dimensions since the soundfile library will return a one-dimensional NumPy array by default\n    reshaped_data = data.reshape((-1, data.shape[0]))\n    return reshaped_data","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def extract_time_interval(audio_signal: np.ndarray, sample_rate: int, start_time_in_seconds: int, interval_length_in_seconds: int, pad: bool=True) -> np.ndarray:\n    \"\"\"Returns a NumPy array corresponding to a specified time interval from a given NumPy array audio sample\n\n    Parameters\n    ----------\n    audio_signal: np.ndarray\n        The input NumPy array audio signal\n    sample_rate: int\n        The sample rate in Hertz of the inputted audio\n    start_time_in_seconds: int\n        The offset in seconds from the beginning of the audio sample from which the desired interval of audio is extracted\n    interval_length_in_seconds: int\n        The length of the desired duration in seconds of the audio sample to be extracted\n    pad: bool\n        A bool indicating whether the output array should be zero-padded to satisfy the desired interval length\n\n    Returns\n    -------\n    numpy.ndarray\n        A NumPy array representing the extracted audio signal\n    \"\"\"\n    # Use the start time in seconds and the sample rate to calculate the actual index at which the desired interval of audio begins\n    start_index = start_time_in_seconds * sample_rate\n    # Use the interval in seconds and the sample rate to calculate actual the interval in indices in indices of the input NumPy array that must be extracted\n    index_interval = interval_length_in_seconds * sample_rate\n    sample = audio_signal[:, start_index:start_index + index_interval]\n    # If the caller does not wish to pad the data should it be shorter than the desired interval\n    if not pad:\n        return sample\n    # If padding is required\n    else:\n        # Using the sample length that was required appropriate number of zeros so that it has the desired interval length\n        num_indices_to_pad = index_interval - sample.shape[1]\n        # If the length of the sample is of the desired length\n        if num_indices_to_pad == 0:\n            return sample\n        # Else if the length of the sample is shorter than desired\n        else:\n            return np.concatenate([sample, np.zeros((1, num_indices_to_pad))], axis=1)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def extract_random_time_interval(audio_signal: np.ndarray, sample_rate: int, interval_length_in_seconds: int, pad: bool) -> np.ndarray:\n    \"\"\"Returns a random interval of the specified length from the inputted audio signal\n\n    Parameters\n    ----------\n    audio_signal: np.ndarray\n        A NumPy array representing an audio signal\n    sample_rate: int\n        The sample rate in Hertz of the inputted audio\n    interval_length_in_seconds: int\n        The length of the desired duration in seconds of the audio sample to be extracted\n    pad: bool\n        A bool indicating whether the output array should be zero-padded to satisfy the desired interval length\n\n    Returns\n    -------\n    numpy.ndarray\n        A NumPy array representing the extracted audio signal\"\"\"\n    num_elements = audio_signal.shape[1]\n    if pad:\n        num_segments = int(np.ceil(num_elements / (interval_length_in_seconds * sample_rate)))\n    else:\n        num_segments = num_elements // (interval_length_in_seconds * sample_rate)\n\n    return extract_time_interval(audio_signal, sample_rate, random.choice(range(num_segments)) * interval_length_in_seconds, interval_length_in_seconds, pad)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Model Definitions\n\nNow that we the preparatory work out of the way, let's move on to defining our neural network models. First, we'll test out the performance of convolutional neural networks (CNNs). To be able to assign a variable number of birdcall labels to a given sound sample, we will train a separate neural network model to recognize each of the 397 distinct classes of birds in our data set. We will use some of the code provided by the [fast.ai tutorial](https://pytorch.org/tutorials/beginner/nn_tutorial.html) at [pytorch.org](pytorch.org) to set up our initial boilerplate and then further modify the neural networks per our needs. Note that we will take a slightly unconvential approach since, in addition to testing simple one-dimensional convolutional neural networks over the audio samples, we will also try reshaping the one-dimensional audio signals into two-dimensional arrays so that we can relatively cheaply extract temporal information over entire audio samples using two-dimensional convolutions. After testing out the convolutional neural networks, we will test the performance of vanilla recurrent neural networks (RNNs) on the data set.","metadata":{}},{"cell_type":"code","source":"import torch\nfrom torch import nn # to expose the Module class from which we can subclass to create our own, custom neural network layers\nfrom typing import Callable # to specify a parameter as a function type\nfrom typing import Union # to specify a parameter as a union type\n\nclass Lambda(nn.Module):\n    \"\"\"A class for creating activation layers for PyTorch neural networks\n\n    ...\n\n    Attributes\n    ----------\n    f: function\n        The activation function\n    \"\"\"\n    def __init__(self, f: Callable[[torch.tensor], torch.tensor]):\n        \"\"\"\n        Parameters\n        ----------\n        f: function\n            The activation function\n        \"\"\"\n        super().__init__()\n        self.f = f\n\n    def forward(self, xs: Union[torch.tensor, Sequence[torch.tensor]]) -> torch.tensor:\n        \"\"\"A function that applies the activation function `self.f` to the inputted sequence of data `xs`\n\n        Parameters\n        ----------\n        x: torch.tensor\n            The inputs to this activation layer\n\n        Returns\n        -------\n        torch.tensor\n            A PyTorch tensor representing the result of applying `self.f` to each element of the inputted tensor x\n        \"\"\"\n        if type(xs) == torch.Tensor:\n            return self.f(xs)\n        else:\n            return self.f(*xs)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def reshape_into_image(x: torch.tensor) -> torch.tensor:\n    \"\"\"Reshapes an inputted PyTorch tensor representing five seconds of audio sampled at 32kHz into a square, 400 x 400 pixel image\n\n    Parameters\n    ----------\n    x: torch.tensor\n        A PyTorch tensor representing a five-second audio signal sampled at 32kHz\n\n    Returns\n    -------\n    torch.tensor\n        A PyTorch tensor in the shape of a 400 x 400 image\n\n    Notes\n    -----\n    This function takes advantage of the fact that a five-second audio clip sampled at 32kHz will have 32,000 * 5 = 160,000 data points, which yields a perfect square.\n    \"\"\"\n    return x.view(-1, 1, 400, 400)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Set our device to be the GPU if available else the CPU\nDEVICE = torch.device(\"cuda\") if torch.cuda.is_available() else torch.device(\"cpu\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class OneDimensionalConvolutionalNeuralNetwork(nn.Module):\n    \"\"\"A class for creating a 1D convolutional neural network birdcall classifier for inputs containing a latitude, longitude, timestamp, and a five-second audio signal sampled at 32kHz\"\"\"\n    def __init__(self):\n        super(OneDimensionalConvolutionalNeuralNetwork, self).__init__()\n        self.splitters = nn.ModuleDict({\n            'non-audio': Lambda(lambda x: x[0, :3].view(1, 3)),\n            'audio': Lambda(lambda x: x[0, 3:])\n        })\n        # Create the convolutional neural network for the audio signal\n        self.cnn = nn.Sequential(\n            Lambda(lambda x: x.view(-1, 1, 160_000)),\n            nn.Conv1d(1, 1, kernel_size=16, stride=16),\n            nn.ReLU(),\n            nn.Conv1d(1, 1, kernel_size=100, stride=100),\n            nn.ReLU(),\n            nn.AdaptiveAvgPool1d(10),\n            Lambda(lambda x: x.view(-1, 10))\n        )\n        self.combiner = Lambda(lambda x1, x2: torch.cat([x1, x2], dim=1))\n        self.output = nn.Linear(13, 2)\n\n    def forward(self, x):\n        non_audio_x = self.splitters['non-audio'](x)\n        audio_x = self.splitters['audio'](x)\n        cnn_output = self.cnn(audio_x)\n        recombined_input = self.combiner([non_audio_x, cnn_output])\n        return self.output(recombined_input)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from torch import nn # to expose the nn.Module class\n\nclass TwoDimensionalConvolutionalNeuralNetwork(nn.Module):\n    \"\"\"A class for creating a 2D convolutional neural network birdcall classifier for inputs containing a latitude, longitude, timestamp, and a five-second audio signal sampled at 32kHz\"\"\"\n    def __init__(self):\n        super(TwoDimensionalConvolutionalNeuralNetwork, self).__init__()\n        # Create a dictionary of `nn.Module`s for splitting the input into its audio and non-audio components\n        self.splitters = nn.ModuleDict({\n            'non-audio': Lambda(lambda x: x[0, :3].view(1, 3)),\n            'audio': Lambda(lambda x: x[0, 3:])\n        })\n        # Create the convolutional neural network for the audio signal\n        self.cnn = nn.Sequential(\n            # Reshape the audio signal into a square image\n            Lambda(reshape_into_image),\n            nn.Conv2d(1, 1, kernel_size=20, stride=20),\n            nn.ReLU(),\n            nn.Conv2d(1, 1, kernel_size=10, stride=10),\n            nn.ReLU(),\n            nn.AdaptiveAvgPool2d(1),\n            Lambda(lambda x: x.view(-1, 1))\n        )\n        # For combining the CNN output and non-audio input into a single PyTorch tensor\n        self.combiner = Lambda(lambda x1, x2: torch.cat([x1, x2], dim=1))\n        # For yielding the final output\n        self.output = nn.Linear(4, 2)\n\n    def forward(self, x):\n        non_audio_x = self.splitters['non-audio'](x)\n        audio_x = self.splitters['audio'](x)\n        cnn_output = self.cnn(audio_x)\n        recombined_input = self.combiner([non_audio_x, cnn_output])\n        return self.output(recombined_input)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class RecurrentNeuralNetwork(nn.Module):\n    \"\"\"A class for creating a recurrent neural network birdcall classifier for inputs containing a latitude, longitude, timestamp, and a five-second audio signal sampled at 32kHz\"\"\"\n    def __init__(self):\n        super(RecurrentNeuralNetwork, self).__init__()\n        self.splitters = nn.ModuleDict({\n            'non-audio': Lambda(lambda x: x[:, :3]),\n            'audio': Lambda(lambda x: x[:, 3:])\n        })\n        # Create the recurrent neural network for the audio signal\n        self.rnn_one = nn.RNNCell(10000, 5)\n        self.rnn_two = nn.RNNCell(5, 10)\n        self.combiner = Lambda(lambda x1, x2: torch.cat([x1, x2], dim=1))\n        self.output = nn.Linear(13, 2)\n\n    def forward(self, x):\n        non_audio_x = self.splitters['non-audio'](x)\n        audio_x = self.splitters['audio'](x)\n        hx_1 = torch.randn(1, 5).to(DEVICE)\n        hx_2 = torch.randn(1, 10).to(DEVICE)\n        for i in range(16):\n            hx_1 = self.rnn_one(audio_x[:, i * 10000:i * 10000 + 10000], hx_1)\n            hx_2 = self.rnn_two(hx_1, hx_2)\n        recombined_input = self.combiner([non_audio_x, hx_2])\n        return self.output(recombined_input)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, that we've got our models, let's make sure that we can run them on a real training sample.","metadata":{}},{"cell_type":"code","source":"def get_randomized_model_input_as_tensor(example: pd.DataFrame, sample_rate: int, sample_interval_length_in_seconds: int, device: torch.device, pad: bool) -> torch.Tensor:\n    \"\"\"Produces a tensor corresponding to a random audio segment of the desired duration from the specified file along with the audio example's metadata attributes\n\n    Parameters\n    ----------\n    example: pd.DataFrame\n        A Pandas DataFrame containing the name of the file from which the audio segment should be extracted along with its associated metadata attributes\n    sample_rate: int\n        The sample rate of the audio file\n    sample_interval_length_in_seconds: int\n        The desired duration in seconds of the audio segment to be extracted\n    device: torch.device\n        The device (i.e. CPU or GPU) on which the returned PyTorch tensor should reside\n    pad: bool\n        Whether or not the function should permit returning an audio segment that is originally shorter than the desired length by means of zero-padding\n\n    Returns\n    -------\n    torch.Tensor\n        The PyTorch tensor representing an input vector to be fed into a neural network\"\"\"\n    filename = example['filename']\n    latitude = example['latitude']\n    longitude = example['longitude']\n    timestamp = example['timestamp']\n    birdcall_label = example['birdcall label']\n    audio_signal = read_ogg_data(filename, birdcall_label)\n    audio_data = extract_random_time_interval(audio_signal, sample_rate, sample_interval_length_in_seconds, pad=pad)\n    # Ensure that the elements of the input are of type torch.float32 instead of torch.float64 so that they are compatible with the convolution layers of our neural network\n# See the following StackOverflow article for reference: https://stackoverflow.com/questions/66074684/runtimeerror-expected-scalar-type-double-but-found-float-in-pytorch-cnn-train\n    non_audio_data = torch.tensor([latitude, longitude, timestamp]).view(1, -1).float().to(DEVICE)\n    audio_data = torch.tensor(audio_data).view(1, -1).float().to(DEVICE)\n    return torch.cat([non_audio_data, audio_data], dim=1).float().to(DEVICE)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import torch.nn.functional as F\n\n# Use cross-entropy as our classification loss function\nLOSS_FUNCTION = F.cross_entropy\n\nSAMPLE_INTERVAL_LENGTH_IN_SECONDS = 5\nSAMPLE_RATE = 32_000\n\nLEARNING_RATE = 0.01\nMOMENTUM = 0.90","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from torch import optim # to expose the optim module and the Optimizer class\n\ncnn_1d = OneDimensionalConvolutionalNeuralNetwork().to(DEVICE)\ncnn_2d = TwoDimensionalConvolutionalNeuralNetwork().to(DEVICE)\nrnn = RecurrentNeuralNetwork().to(DEVICE)\n\n# Get a random training file\ntraining_file = training_files.iloc[random.choice(range(training_files.shape[0])), :]\nx = get_randomized_model_input_as_tensor(training_file, SAMPLE_RATE, SAMPLE_INTERVAL_LENGTH_IN_SECONDS, DEVICE, False)\n\ncnn_1d_optim = optim.SGD(cnn_1d.parameters(), lr=LEARNING_RATE, momentum=MOMENTUM)\ncnn_2d_optim = optim.SGD(cnn_2d.parameters(), lr=LEARNING_RATE, momentum=MOMENTUM)\nrnn_optim = optim.SGD(rnn.parameters(), lr=LEARNING_RATE, momentum=MOMENTUM)\n\ncnn_1d_out = cnn_1d(x)\ncnn_2d_out = cnn_2d(x)\nrnn_out = rnn(x)\n\n# Assume the true label to be 0 and propagate gradients backward through the networks\ncnn_1d_loss = LOSS_FUNCTION(cnn_1d_out, torch.zeros(1).long().to(DEVICE))\ncnn_1d_loss.backward()\n\ncnn_2d_loss = LOSS_FUNCTION(cnn_2d_out, torch.zeros(1).long().to(DEVICE))\ncnn_2d_loss.backward()\n\nrnn_label_prediction_loss = LOSS_FUNCTION(rnn_out, torch.zeros(1).long().to(DEVICE))\nrnn_label_prediction_loss.backward()\n\n# Try making a gradient descent step\ncnn_1d_optim.step()\ncnn_2d_optim.step()\nrnn_optim.step()\n\n# Zero out the error derivatives over the computation graph\ncnn_1d_optim.zero_grad()\ncnn_2d_optim.zero_grad()\nrnn_optim.zero_grad()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training","metadata":{}},{"cell_type":"code","source":"from typing import Dict # to specify a parameter as a dict\n\ndef train_models_on_audio_file(training_file: pd.DataFrame, loss_function: Callable[[torch.tensor, torch.tensor], torch.tensor], class_label_to_model_optimizer_dict: Dict[str, optim.Optimizer], class_label_to_model_dict: Dict[str, nn.Module], sample_rate: int, sample_interval_length_in_seconds: int, device: torch.device) -> Dict[str, torch.Tensor]:\n    \"\"\"Trains a set of binary neural network classifiers on a set of training files\n\n    Parameters\n    ----------\n    training_files: pd.DataFrame\n        A Pandas DataFrame containing the filename of a training file along with its associated metadata attributes\n    loss_function: Callable[[torch.tensor, torch.tensor], torch.tensor]\n        The loss function applied to each of the classifiers\n    class_label_to_model_optimizer_dict: Dict[str, optim.Optimizer]\n        A dictionary mapping each label from the list of class labels to the optimizer of its corresponding model instance\n    class_label_to_model_dict: Dict[str, nn.Module]\n        A dictionary that maps each label from the list of class labels to its associated model instance\n    audio_file_sample_rate: int\n        The sampling rate of each audio file on which the models are to be trained\n    sample_interval_length_in_seconds: int\n        The duration in seconds of the audio samples on which the models should be trained\n    device: torch.device\n        The type of device (CPU or GPU) on which to store the PyTorch tensors\n\n    Returns\n    -------\n    Dict[str, torch.Tensor]\n        A dictionary mapping each birdcall label to the cross-entropy classification loss of the corresponding model\n    \"\"\"\n    x_train = get_randomized_model_input_as_tensor(training_file, sample_rate, sample_interval_length_in_seconds, device, False)\n    class_label_to_loss_dict = {}\n    for class_label, model in class_label_to_model_dict.items():\n        # Creata a one-dimensional PyTorch tensor consisting of all zeros if the birdcall label is not equal to the parent folder else a one-dimensional PyTorch tensor consisting of all ones with the number of elements equal to the number of samples\n        y_train = torch.ones(1) if (class_label == training_file['birdcall label']) else torch.zeros(1)\n        # Ensure that the elements of y_train are integral values of type torch.int64\n        y_train = y_train.long()\n        # Make sure that the label vector resides on the appropriate device\n        y_train = y_train.to(device)\n        predictions = model(x_train)\n        loss = loss_function(predictions, y_train)\n        # Backpropagate the gradients over the graph representing the neural network\n        loss.backward()\n        # Make a gradient descent step using the model's associated optimizer\n        class_label_to_model_optimizer_dict[class_label].step()\n        # Reset all model parameter gradients to zero\n        class_label_to_model_optimizer_dict[class_label].zero_grad()\n        model.eval()\n        with torch.no_grad():\n            class_label_to_loss_dict[class_label] = loss_function(predictions, y_train).item()\n    return class_label_to_loss_dict","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Use the tqdm object from the tqdm.notebook module so that the progress bar does not result in the creation of multiple lines of output in the IPython notebook\n# Suggestion taken from the following StackOverflow article: https://stackoverflow.com/questions/42212810/tqdm-in-jupyter-notebook-prints-new-progress-bars-repeatedly\nfrom tqdm.notebook import tqdm # to be able to show a progress bar during training\n\ndef train_models_on_examples(training_files: pd.DataFrame, loss_function: Callable[[torch.tensor, torch.tensor], torch.tensor], class_label_to_model_optimizer_dict: Dict[str, optim.Optimizer], class_label_to_model_dict: Dict[str, nn.Module], audio_file_sample_rate: int, sample_interval_length_in_seconds: int, device: torch.device, progress_bar: bool=True) -> Dict[str, Sequence[torch.Tensor]]:\n    \"\"\"Trains a set of classifiers to a sequence of labelled audio files\n\n    Parameters\n    ----------\n    training_files: pd.DataFrame\n        A Pandas DataFrame containing a list of files on which the \n    loss_function: Callable[[torch.tensor, torch.tensor], torch.tensor]\n        The loss function to be applied to each of the classifiers\n    class_label_to_model_optimizer_dict: Dict[str, optim.Optimizer]\n        A dictionary that maps each label from the list of class labels to the optimizer of its corresponding model instance\n    class_label_to_model_dict: Dict[str, nn.Module]\n        A dictionary that maps each label from the list of class labels to its associated model instance\n    audio_file_sample_rate: int\n        The sampling rate of each audio file on which the models are to be trained\n    sample_interval_length_in_seconds: int\n        The duration in seconds of the audio samples on which the models should be trained\n    device: torch.device\n        The type of device (CPU or GPU) on which to store the PyTorch tensors\n    progress_bar: bool\n        A bool indicating whether or not a progress bar should be displayed during training\n\n    Returns\n    -------\n    Dict[str, Sequence[torch.Tensor]]\n        A dictionary mapping each birdcall label to a sequence of cross-entropy classification losses for each training file on which the models were trained\n    \"\"\"\n    # Create an appropriate iterable over the provided sequence of identifiers based on whether the user wished to display a progress bar or not\n    indices = tqdm(range(training_files.shape[0])) if progress_bar else audio_sample_id_sequence\n    class_label_to_losses_dict = {}\n    for idx in indices:\n        class_label_to_loss_dict = train_models_on_audio_file(training_files.iloc[idx, :], loss_function, class_label_to_model_optimizer_dict, class_label_to_model_dict, audio_file_sample_rate, sample_interval_length_in_seconds, device)\n        for class_label, loss in class_label_to_loss_dict.items():\n            losses = class_label_to_losses_dict.get(class_label, [])\n            losses.append(loss)\n            class_label_to_losses_dict[class_label] = losses\n    return class_label_to_losses_dict","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"birdcalls = most_common_birdcalls.values.reshape(-1)\n\n# Initialize our neural network models\none_dimensional_convolutional_models = dict(zip(birdcalls, [OneDimensionalConvolutionalNeuralNetwork().to(DEVICE) for _ in birdcalls]))\ntwo_dimensional_convolutional_models = dict(zip(birdcalls, [TwoDimensionalConvolutionalNeuralNetwork().to(DEVICE) for _ in birdcalls]))\nrnn_models = dict(zip(birdcalls, [RecurrentNeuralNetwork().to(DEVICE) for _ in birdcalls]))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create an optimizer to do gradient updates for every set of models that must be trained\none_dimensional_convolutional_network_optimizers = dict(zip(birdcalls, [optim.SGD(model.parameters(), lr=LEARNING_RATE, momentum=MOMENTUM) for model in one_dimensional_convolutional_models.values()]))\ntwo_dimensional_convolutional_network_optimizers = dict(zip(birdcalls, [optim.SGD(model.parameters(), lr=LEARNING_RATE, momentum=MOMENTUM) for model in two_dimensional_convolutional_models.values()]))\nrnn_optimizers = dict(zip(birdcalls, [optim.SGD(model.parameters(), lr=LEARNING_RATE, momentum=MOMENTUM) for model in rnn_models.values()]))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Reproducibility\n\nThough we cannot guarantee with one hundred percent confidence that our experiments will be fully reproducible, we can take some steps to minimize the variation that might occur across runs of our notebook by following the advice provided in the PyTorch [reproducibility article](https://pytorch.org/docs/stable/notes/randomness.html?highlight=reproducible#).","metadata":{}},{"cell_type":"code","source":"# Set the random seed used by PyTorch to help ensure that we get more reproducible results during training\ntorch.manual_seed(RANDOM_SEED)\n# Similarly set the random seed used by NumPy\nnp.random.seed(RANDOM_SEED)\n# Tell PyTorch to use deterministic algorithms instead of non-deterministic ones to also help ensure that we get more reproducible results\ntry:\n    torch.use_deterministic_algorithms(True)\n# In case the currently loaded version of the PyTorch library does not support `use_deterministic_algorithms()`\nexcept AttributeError:\n    print(\"This version of PyTorch does not support the use_deterministic_algorithms() method. Skipping...\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Initiating training...\")\nprint(\"Training 1D convolutional models...\")\none_dimensional_convolutional_losses = train_models_on_examples(X_train, LOSS_FUNCTION, one_dimensional_convolutional_network_optimizers, one_dimensional_convolutional_models, SAMPLE_RATE, SAMPLE_INTERVAL_LENGTH_IN_SECONDS, DEVICE)\nprint(\"Training 2D convolutional models...\")\ntwo_dimensional_convolutional_losses = train_models_on_examples(X_train, LOSS_FUNCTION, two_dimensional_convolutional_network_optimizers, two_dimensional_convolutional_models, SAMPLE_RATE, SAMPLE_INTERVAL_LENGTH_IN_SECONDS, DEVICE)\nprint(\"Training recurrent models...\")\nrecurrent_losses = train_models_on_examples(X_train, LOSS_FUNCTION, rnn_optimizers, rnn_models, SAMPLE_RATE, SAMPLE_INTERVAL_LENGTH_IN_SECONDS, DEVICE)\nprint(\"Training completed!\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Cross-Validation","metadata":{}},{"cell_type":"code","source":"# Set the confidence threshold to be fairly high so that we only append a bircall label to a multi-label classification only if we are fairly certain that that birdcall occurs within a particular audio segment\nCONFIDENCE_THRESHOLD = 0.95","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def compute_f1_score(num_false_positives: int, num_true_positives: int, num_false_negatives: int) -> float:\n    # Referenced from: https://en.wikipedia.org/wiki/F-score\n    precision = num_true_positives / (num_false_positives + num_true_positives)\n    recall = num_true_positives / (num_false_negatives + num_true_positives)\n    return 2.0 * (precision * recall) / (precision + recall)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_types = ['1D CNN', '2D CNN', 'RNN']\n\nmodel_type_to_false_positives_dict = dict(zip(model_types, [0] * 3))\nmodel_type_to_true_positives_dict = dict(zip(model_types, [0] * 3))\nmodel_type_to_false_negatives_dict = dict(zip(model_types, [0] * 3))\n\nmodel_type_to_models_dict = dict(zip(model_types, [one_dimensional_convolutional_models, two_dimensional_convolutional_models, rnn_models]))\n\nprint(\"Initiating cross-validation of models...\")\nfor idx in tqdm((range(X_cv.shape[0]))):\n    cv_file = X_cv.iloc[idx, :]\n    x = get_randomized_model_input_as_tensor(cv_file, SAMPLE_RATE, SAMPLE_INTERVAL_LENGTH_IN_SECONDS, DEVICE, False)\n    for model_type, models in model_type_to_models_dict.items():\n        for birdcall, model in models.items():\n            prediction = model(x)\n            if torch.argmax(prediction, dim=1) == 1 and prediction[:, 1] > CONFIDENCE_THRESHOLD:\n                if birdcall == cv_file['birdcall label']:\n                    model_type_to_true_positives_dict[model_type] += 1\n                else:\n                    model_type_to_false_positives_dict[model_type] += 1\n            else:\n                if birdcall == cv_file['birdcall label']:\n                    model_type_to_false_negatives_dict[model_type] += 1\n\nprint(\"Completed cross-validation!\")\nprint(\"Calculated F1 scores:\")\nfor model_type in model_types:\n    print(f\"{model_type} -> {compute_f1_score(model_type_to_false_positives_dict[model_type], model_type_to_true_positives_dict[model_type], model_type_to_false_negatives_dict[model_type])}\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"one_dimensional_convolutional_losses = pd.DataFrame(one_dimensional_convolutional_losses)\ntwo_dimensional_convolutional_losses = pd.DataFrame(two_dimensional_convolutional_losses)\nrecurrent_losses = pd.DataFrame(recurrent_losses)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"one_dimensional_convolutional_losses_melted = one_dimensional_convolutional_losses.melt(var_name='birdcall', ignore_index=False)\ntwo_dimensional_convolutional_losses_melted = two_dimensional_convolutional_losses.melt(var_name='birdcall', ignore_index=False)\nrecurrent_losses_melted = recurrent_losses.melt(var_name='birdcall', ignore_index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"one_dimensional_convolutional_losses.to_csv(f\"{output_folder}1D_CNN_losses.csv\")\ntwo_dimensional_convolutional_losses.to_csv(f\"{output_folder}2D_CNN_losses.csv\")\nrecurrent_losses.to_csv(f\"{output_folder}recurrent_losses.csv\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}