{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Tensorflow - Ready Notebook\n\nBirdCLEF22 attempt. Did not catch a dataleak until the eleventh hour. Hopefully this public notebook will help others for BirdCLEF23. While this notebook is ready to train off the bat, I highly recommend that the training is done on an external computer. I did not have much luck with kaggle due to the lack of compute resources.\n\n## High-level overview\n\nI trust the user has done their own data exploration. \n\n1. Create a pd.Dataframe of audio files split into 5s chunks. This dataframe will be shuffled, and the filename + time information will be used to construct a spectrogram of the correct time.\n2. Dataframe will be parsed into a tf.Dataset generator yielding stft + label pairs. There are several wrappers to force this process.\n3. Augmentations will be applied via Dataset.map\n4. Finish off by normalising + centering the bins in the spectrogram. This was accomplished by taking a log, scaling and shifting.\n\n## Features\n\nThis notebook contains a few features that are easily customisable and usable in data projects of a similar scope.\n\n### Augmentations\n\nStandard augmentations for image processing. Most of these are given as tf.functions which can be executed in graph mode. Those that cannot will need to be wrapped using tf.py_function.\n\n* Time shift\n* White noise\n* Pink noise\n* Mixing of sounds\n\n### Custom loss function\n\nBinary crossentropy (BCE) suffers when the number present objects in its label vector is few relative to the length of the vector. While using BCE + sigmoid activation is nominally OK in this case (2-3 birds in ~20 vector), we have experimented in using a weighted categorical crossentropy function + softmax activation, where we scale $y_true$ by the number of positive classes. It is shown in https://arxiv.org/pdf/1805.00932.pdf that this may help the training process, however you will need another method to discern the number of objects in the inferrence stage (take top $N$ items from the predict vector).\n\n\n","metadata":{}},{"cell_type":"markdown","source":"## Libraries","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport librosa\nimport librosa.display\nimport IPython.display as ipd\nimport ast\nimport json\nimport glob\nimport os\n\nimport tensorflow as tf\nimport tensorflow_addons as tfa\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\nfrom tensorflow.keras import callbacks\n\nprint(tf.__version__)\nprint(np.__version__)\nprint(pd.__version__)","metadata":{"execution":{"iopub.status.busy":"2022-05-24T07:10:49.768010Z","iopub.execute_input":"2022-05-24T07:10:49.768316Z","iopub.status.idle":"2022-05-24T07:10:49.962463Z","shell.execute_reply.started":"2022-05-24T07:10:49.768283Z","shell.execute_reply":"2022-05-24T07:10:49.961558Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Globals","metadata":{}},{"cell_type":"code","source":"\"\"\"\nall constants here:\n- audio constants, chunking parameters, etc\n- stft constants, n_fft, hop size, etc.\n- tensorflow constants: learning rate, layer depth, etc.\n\"\"\"\n\ndef get_scored_birds(file_name = '../input/birdclef-2022/scored_birds.json'):\n    with open(file_name) as sbfile:\n        scored_birds = json.load(sbfile)\n    return scored_birds\n\nPATH_DIR = \"../input/birdclef-2022/train_audio\"\nSEED = 42\nSCORED_BIRDS = np.asarray(get_scored_birds())\n\n#SINGLE AUDIO PARAMS\nCHUNK_LENGTH = 5\nSR = 32000\nMIN_REMAINDER = 3\nSEC_DUR = 30 #If audio duration > SEC_DUR, ignore secondary birds. Shorter audio files more likely to encapsulate secondary bird in side the 5s chunk.\n\n#STFT GLOBALS\nTOTAL_FRAMES = int(CHUNK_LENGTH * SR)\nN_FFT = 2048\nHOP_DIV = 4\nFREQ_BANDS = (N_FFT//2) + 1\nN_FFT_FRAMES = ((TOTAL_FRAMES - N_FFT)//(N_FFT // HOP_DIV)) + 1\nSTFT_NORM = 1 / 1.5 / FREQ_BANDS #Used in noise augmentations. hop_div > 1 raises the total energy from an STFT. Given the params, N_FFT, HOP_DIV and WINDOW_SIZE = N_FFT, this is our energy norm from STFT -> signal\n\n#DATASET GENERATION PARAMS\nMAX_FN = 50 #max number of files per class\n\n#NET TRAINING PARAMS\n\n#INPUT_SIZE = (N_FFT_FRAMES, FREQ_BANDS, 1) #If spectrogram is used for training\nN_MELS = 128\nINPUT_SIZE = (N_FFT_FRAMES, N_MELS, 1) #If mel-spectrogram is used for training","metadata":{"execution":{"iopub.status.busy":"2022-05-24T06:05:10.357451Z","iopub.execute_input":"2022-05-24T06:05:10.357800Z","iopub.status.idle":"2022-05-24T06:05:10.377007Z","shell.execute_reply.started":"2022-05-24T06:05:10.357765Z","shell.execute_reply":"2022-05-24T06:05:10.376240Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Bird-to-index encoder/decoder","metadata":{}},{"cell_type":"code","source":"\"\"\"\nCreate encoder/decoder to convert between birds to indices.\nAdd a background class represented by 'other', or index 21\n\"\"\"\n\nSCORED_BIRDS = np.concatenate((SCORED_BIRDS, ['other']))\nprint(SCORED_BIRDS)\n\n# encode bird id to each scoring birds \nencode = {}\nbird_id = 0\nfor bird in SCORED_BIRDS:\n    encode[bird] = bird_id\n    bird_id+=1\n\nencode[\"other\"] = 21\ndecode = {v: k for k, v in encode.items()}","metadata":{"execution":{"iopub.status.busy":"2022-05-24T06:05:11.226151Z","iopub.execute_input":"2022-05-24T06:05:11.226995Z","iopub.status.idle":"2022-05-24T06:05:11.234492Z","shell.execute_reply.started":"2022-05-24T06:05:11.226950Z","shell.execute_reply":"2022-05-24T06:05:11.233605Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Metadata for train data","metadata":{}},{"cell_type":"code","source":"\"\"\"\nWe're going to create a df that expands the loaded bird audio files into their chunks.\nWe point our tensorflow datagenerator to load specific \"chunks\" instead of unbatching\nThis allows us to get the exact filenumber, AND for those that are interested in training on kaggle, allows us to get away with smaller\nshuffles on our datagen.\n\"\"\"\nINPUT_DIR = '/kaggle/input/birdclef-2022'\n\ntaxonomy_data = pd.read_csv(f'{INPUT_DIR}/eBird_Taxonomy_v2021.csv')\ntrain_data = pd.read_csv(f'{INPUT_DIR}/train_metadata.csv')\n\n#Manipulate dataset with convenient information\ntrain_data['sound_file'] = INPUT_DIR + '/train_audio/' + train_data.filename\ntrain_data.drop(['author','license','url','filename'],axis=1)","metadata":{"execution":{"iopub.status.busy":"2022-05-24T06:05:15.937156Z","iopub.execute_input":"2022-05-24T06:05:15.937470Z","iopub.status.idle":"2022-05-24T06:05:16.181334Z","shell.execute_reply.started":"2022-05-24T06:05:15.937435Z","shell.execute_reply":"2022-05-24T06:05:16.180466Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Constructing a Dataframe as an input for a tf.Dataset generator","metadata":{}},{"cell_type":"code","source":"\"\"\"\nConstruct a dataframe consisting of our birds, filenames and chunks. We shuffle this dataset and pull a smaller buffer during net training\nto avoid kaggle from freaking out on us with memory issues\n\nMAX_FN is max number of files per bird to pull. Partially mitigates the impact of imbalanced classes.\nAlternatively you can pull all the birds in the desired class and adjust class weights in the loss function.\n\"\"\"\n\nindex_frame = pd.DataFrame(columns = ['birdname', 'secondarybirds', 'filepath', 'duration', 'dur_pos', 'partial'])\n\ndef second_check(x):\n    \"\"\"\n    For birds without any elements in the 'secondary_birds' column. Pass it a placeholder value\n    \"\"\"\n    if len(x) == 0:\n        return np.asarray(['placeholder']).astype(np.unicode_)\n    return np.asarray(x).astype(np.unicode_)\n\ndef encode_second(x):\n    \"\"\"\n    Encodes the names of birds into their respective indices.\n    If I had more time, I would change the final index from 99 to encode['others'] such that it is captured in my final labels\n    \"\"\"\n    return np.asarray([*map(lambda i: encode.get(i, 99), x)])\n\nfor bird in SCORED_BIRDS:\n    bird_df = train_data[train_data.primary_label == bird]\n    n_files = len(bird_df)\n    \n    if n_files > MAX_FN:\n        bird_df = bird_df.sample(MAX_FN, random_state=SEED)\n        \n    for _ , element in bird_df.iterrows():\n\n        y, _ = librosa.load(element.sound_file, sr=None)\n\n        #Load librosa, we'll chunk the data, have another flag that says \"partial\" if the remaining duration of a chunk is > MIN_REMAINDER seconds\n        chunk, remainder = np.divmod(len(y), TOTAL_FRAMES)\n        for i in range(1, chunk+1):\n            index_frame = index_frame.append({\"birdname\":bird, \"secondarybirds\": element.secondary_labels, \"filepath\":element.sound_file, \"duration\": len(y)/SR, \"dur_pos\":i*5, \"partial\": False}, ignore_index=True)\n\n        if remainder >= MIN_REMAINDER * SR:\n            index_frame = index_frame.append({\"birdname\":bird, \"secondarybirds\": element.secondary_labels, \"filepath\":element.sound_file, \"duration\": len(y)/SR, \"dur_pos\":(chunk+1)*5, \"partial\": True}, ignore_index=True)\n\n#Include the 'other' bird category\nother_df = train_data[~train_data['primary_label'].isin(SCORED_BIRDS)].sample(int(MAX_FN), random_state=42)\n\nfor _ , element in other_df.iterrows():\n    y, _ = librosa.load(element.sound_file, sr=None)\n\n    chunk, remainder = np.divmod(len(y), TOTAL_FRAMES)\n    for i in range(1, chunk+1):\n        index_frame = index_frame.append({\"birdname\":\"other\", \"secondarybirds\": element.secondary_labels, \"filepath\":element.sound_file, \"duration\": len(y)/SR, \"dur_pos\":i*5, \"partial\": False}, ignore_index=True)\n\n    if remainder >= MIN_REMAINDER * SR:\n        index_frame = index_frame.append({\"birdname\":\"other\", \"secondarybirds\": element.secondary_labels, \"filepath\":element.sound_file, \"duration\": len(y)/SR, \"dur_pos\":(chunk+1)*5, \"partial\": True}, ignore_index=True)\n\n# Convert all birdname strings into their encoded indices\nindex_frame['birdname'] = index_frame['birdname'].map(encode.get)        \nindex_frame['secondarybirds'] = index_frame['secondarybirds'].map(lambda x: encode_second(second_check(ast.literal_eval(x))))\n\n# Pandas does not take ragged arrays well. Convert the secondary_bird array into a standard format of shape (max_secondary, )\nmax_secondary = index_frame['secondarybirds'].map(len).max()\nindex_frame['secondarybirds'] = index_frame['secondarybirds'].map(lambda x: np.pad(x, (0, max_secondary - len(x)), constant_values=99))\n\nprint(index_frame.head())\n\nindex_frame.to_pickle('training_df.pickle')","metadata":{"execution":{"iopub.status.busy":"2022-05-24T06:14:18.218063Z","iopub.execute_input":"2022-05-24T06:14:18.218359Z","iopub.status.idle":"2022-05-24T06:15:37.041562Z","shell.execute_reply.started":"2022-05-24T06:14:18.218329Z","shell.execute_reply":"2022-05-24T06:15:37.040596Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Utility functions","metadata":{}},{"cell_type":"code","source":"\"\"\"\nUtility functions used in generating dataset and passing it through to the net\n\"\"\"\n\ndef bird_one_hot(bird_elem, birds):\n    \"\"\"\n    Birds is an array of named birds that we would like to score.\n    To exclude secondarybirds, i.e. converting multi-hot multi-label -> one-hot single-label, you can always manually change SEC_DUR into 0.\n    \"\"\"    \n    depth = len(birds)\n    \n    if tf.math.reduce_any(decode[bird_elem['birdname'].numpy()] == \"other\"):\n        return tf.one_hot(int(depth - 1), depth=depth, dtype=tf.float32)\n    \n    if bird_elem['duration'] < SEC_DUR:\n        concat_label = tf.concat([tf.expand_dims(bird_elem['birdname'], axis=0), tf.cast(bird_elem['secondarybirds'], dtype=tf.int32)], axis=0)\n        return tf.reduce_max(tf.one_hot(concat_label, depth=depth, dtype=tf.float32), axis=0)\n        \n    else:\n        return tf.one_hot(bird_elem['birdname'], depth=depth, dtype=tf.float32)\n    \n@tf.function\ndef normalize_audio(stft, *label):\n    \"\"\"\n    Convert to db or just log it.\n    Manually checked that output is more or less between ~[-1,1]\n    \"\"\"\n    shift = 4\n    scale = 6\n    \n    if label:\n        return ((tf.math.log(stft + 1e-10)/tf.math.log(10.0)) + shift)/scale, label[0]\n    \n    else:\n        return ((tf.math.log(stft + 1e-10)/tf.math.log(10.0)) + shift)/scale\n\n@tf.function\ndef add_single_channel(stft, *label):\n    \"\"\"\n    Utility map to add final channel for net input.\n    \"\"\"\n    if label:\n        return tf.expand_dims(stft, axis=-1), label[0]\n    else:\n        return tf.expand_dims(stft, axis=-1)","metadata":{"execution":{"iopub.status.busy":"2022-05-24T06:49:29.094591Z","iopub.execute_input":"2022-05-24T06:49:29.094910Z","iopub.status.idle":"2022-05-24T06:49:29.108778Z","shell.execute_reply.started":"2022-05-24T06:49:29.094879Z","shell.execute_reply":"2022-05-24T06:49:29.107759Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\nAugmentation functions.\nAll of these take in spectrograms, but some can be used with mel-spectrograms.\nYou will need to calculate STFT_NORM yourself if you mess with N_FFT, HOP_DIV and WINDOW_SIZE.\n\nCalculation is straightforward -> divide total energy from signal(either in time or freq domain)/total energy in STFT.\nWith this factor you can compute total energy in the STFT and scale back to find total energy of the underlying signal.\n\"\"\"\n\n@tf.function\ndef time_roll(inp, *label):\n    \"\"\"\n    Roll image in time dimension.\n    \"\"\"\n    random_shift = tf.random.uniform(shape=(), minval=0, maxval=N_FFT_FRAMES, dtype=tf.int32)\n    roll = tf.roll(inp, shift=random_shift, axis=0)\n    \n    if label:\n        return roll, label[0]\n    \n    return roll\n\n@tf.function\ndef white_noise(inp, *label):\n    \"\"\"\n    Create a WN signal between 0.01%-1% of total signal energy.\n    \"\"\"\n    \n    energy_max = 1./100\n    \n    signal_energy = STFT_NORM * tf.reduce_sum(inp ** 2)\n    gain = tf.random.uniform([]) * energy_max\n    \n    #power of white noise is equal to its variance, remember to divide by N samples to conserve energy after FFT\n    std = tf.math.sqrt(signal_energy * gain) / (CHUNK_LENGTH * SR)\n    \n    noise_signal = tf.random.normal([CHUNK_LENGTH * SR], 0, std)\n    noise_stft = tf.abs(tf.signal.stft(noise_signal, frame_length=N_FFT, frame_step=N_FFT//HOP_DIV))\n    \n    if label:\n        return inp + noise_stft, label[0]\n    \n    return inp + noise_stft\n\n@tf.function\ndef pink_noise(inp, *label):\n    \"\"\"\n    Create a PN signal probably up to 100% due to wanting to get some energy in the middle bands.\n    Ideal method is to calc total energy of the pink_noise spectrum, then scale upwards.\n    Probably better to implement a user-defined min-freq cutoff. Here the 0Hz bin is replaced with 1./SR\n    \n    Here the factor of ~2000 comes from SR/2 divided by integral(1/f from ~0 to 16000), i.e. power loss from the 1/f scaling.\n    \"\"\" \n    signal_energy = STFT_NORM * tf.reduce_sum(inp ** 2)\n    gain = tf.random.uniform([])\n    \n    #power of white noise is equal to its variance\n    std = tf.math.sqrt(signal_energy * gain) / (CHUNK_LENGTH * SR) * 2000\n    \n    inv_f = ((tf.cast(tf.linspace(0, SR//2, FREQ_BANDS), tf.float32))) + tf.concat((tf.constant([SR], dtype=tf.float32), tf.zeros(FREQ_BANDS-1)), axis=0)\n    \n    noise_signal = tf.random.normal([CHUNK_LENGTH * SR], 0, std)\n    noise_stft = tf.abs(tf.signal.stft(noise_signal, frame_length=N_FFT, frame_step=N_FFT//HOP_DIV))/inv_f\n    \n    if label:\n        return inp + noise_stft, label[0]\n    \n    return inp + noise_stft\n\n@tf.function\ndef mix_items(inp, label, ds):\n    \"\"\"\n    You need to define another dataset. Take care that this dataset contains the same elements as your training dataset to prevent\n    data leakage.\n    \n    Mixing ratio is currently set to 0.4 - 0.7 of our input.\n    \n    Mix two spectrograms and return a multi-hot label.\n    \"\"\"\n    mix_stft, mix_label = next(iter(ds))\n    \n    alpha = tf.random.uniform([])*0.4 + 0.3\n    beta = 1 - alpha\n    \n    sum_stft = alpha * inp + beta * mix_stft\n    sum_label = tf.reduce_max(tf.concat([label[None,:], mix_label[None,:]], axis=0), axis=0)\n    \n    return sum_stft, sum_label\n\n@tf.function\ndef random_effect(inp, label, fn, base_prob):\n    \"\"\"\n    tf graph mode does not like if-else statements. Wrapper to incorporate random augmentation with a base probability\n    \"\"\"\n    tmp_tuple = tf.cond(tf.random.uniform([]) < base_prob, lambda: (fn(inp, label)), lambda: (inp, label))\n    \n    return tmp_tuple[0], tmp_tuple[1]\n\n@tf.function\ndef random_mix_wrapper(inp, label, ds, base_prob):\n    \"\"\"\n    Much easier to make a separate wrapper for the mixing instead of incorporating it into the random_effect function\n    \"\"\"\n    tmp_tuple = tf.cond(tf.random.uniform([]) < base_prob, lambda: (mix_items(inp, label, ds)), lambda: (inp, label))\n    \n    return tmp_tuple[0], tmp_tuple[1]\n\ndef convert_mel(inp, label):\n    \"\"\"\n    Take in a spectogram, convert to mel-spectrogram. \n    \"\"\"\n    mel_conv = librosa.feature.melspectrogram(S=tf.transpose(inp)**2, sr=SR, n_mels=N_MELS)\n\n    return mel_conv.T, label\n    ","metadata":{"execution":{"iopub.status.busy":"2022-05-24T06:49:29.565450Z","iopub.execute_input":"2022-05-24T06:49:29.566322Z","iopub.status.idle":"2022-05-24T06:49:29.589474Z","shell.execute_reply.started":"2022-05-24T06:49:29.566284Z","shell.execute_reply":"2022-05-24T06:49:29.588756Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## tf.Dataset generator and preprocessing","metadata":{}},{"cell_type":"code","source":"\"\"\"\nBunch of tricks to force tf.Dataset generators to take in a dictionary.\n\"\"\"\ndef dict_py_function(func, inp, Tout):\n    \"\"\"\n    Trick py_function to take in a dict by passing our input off as an array, then reconstructing it inside a wrapped function\n    Completely dumb.\n    \"\"\"\n    def wrapped_func(*flattened_inp):\n        #To reconstruct, pass tf.nest.pack_sequence_as(dict, flattened_dict_of_values, expand_composites=True)\n        reconstructed_inp = tf.nest.pack_sequence_as(inp, flattened_inp, expand_composites=True)\n        return func(*reconstructed_inp)\n        \n    return tf.py_function(func=wrapped_func, inp=tf.nest.flatten(inp, expand_composites=True), Tout=Tout)\n\ndef joint_parser(bird_elem, birds):\n    \"\"\"\n    Load audio['filepath']\n    Construct labels into one/multi-hot format    \n    \"\"\"    \n    y, _ = librosa.load(bird_elem['filepath'].numpy(), sr=None)\n    \n    start_idx = (bird_elem['dur_pos'] - CHUNK_LENGTH) * SR\n    end_idx = bird_elem['dur_pos'] * SR\n    \n    if bird_elem['partial']:\n        if bird_elem['duration'] < CHUNK_LENGTH:\n            y = tf.concat([y, y[:(end_idx - len(y))]], axis=0)\n        else:\n            #just overlap up to 2 seconds. easy.\n            y = y[-(CHUNK_LENGTH * SR):]\n    else:\n        y = y[start_idx:end_idx]\n\n    stft_fn = lambda x: tf.abs(tf.signal.stft(x, frame_length=N_FFT, frame_step=N_FFT//HOP_DIV))\n    label_arr = bird_one_hot(bird_elem, SCORED_BIRDS)\n    \n    return stft_fn(y), label_arr\n\ndef construct_dataset(bird_set, birds=SCORED_BIRDS, fraction=0.20, seed=SEED, second_shuffle=False):\n    \"\"\"\n    ---\n    CAUTION: UPDATED ON 24/05. PREVIOUS IMPLEMENTATION INCURRED DATA LEAK.\n    CORRECT ORDER IS: SHUFFLE -> SAMPLE -> CHUNK INTO 5S\n    \n    As our dictionary is already chunked, we're going to shuffle by group (filepath) and pray that birds with small entries do not end up\n    in the validation. In future, you can probably code something to ensure that birdsounds < some number is forced in the training set.\n    ---\n    New function uses slice_from_tensors instead of list_files -> can shuffle the entire list quickly and call smaller buffer for the actual training\n    \n    WARNING to_dict() changes np,pd objects to python scalars which is subsequently re-interpreted by tensorflow\n    You need to EXPLICITLY CAST dtype INSIDE THE MAPPING FUNCTIONS.\n    \"\"\"\n    \n    #number of unique filepaths, or audio files\n    ids = bird_set.filepath.unique()\n    rng = np.random.default_rng(seed=SEED)\n    rng.shuffle(ids)\n    \n    split = int(len(ids) * fraction)\n    \n    tmp_train_birdset = bird_set.set_index(\"filepath\").loc[ids[split:]].reset_index().set_index('birdname').reset_index()\n    tmp_val_birdset = bird_set.set_index(\"filepath\").loc[ids[:split]].reset_index().set_index('birdname').reset_index()\n    \n    train_dataset  = tf.data.Dataset.from_tensor_slices(tmp_train_birdset.to_dict('list'))\n    val_dataset  = tf.data.Dataset.from_tensor_slices(tmp_val_birdset.to_dict('list'))\n    \n    train_len = train_dataset.cardinality().numpy()\n    val_len = val_dataset.cardinality().numpy()\n    \n    train_dataset = train_dataset.shuffle(train_len)\n    val_dataset = val_dataset.shuffle(val_len)\n    \n    if second_shuffle:\n        train_dataset = train_dataset.shuffle(train_len)\n    \n    #need to map a dict deconstructor as py_function does not play well with dicts\n    #https://github.com/tensorflow/tensorflow/issues/27679\n\n    train_dataset = train_dataset.map(lambda a: dict_py_function(joint_parser, [a, birds], [tf.float32, tf.float32]), num_parallel_calls=tf.data.experimental.AUTOTUNE)\n    val_dataset = val_dataset.map(lambda a: dict_py_function(joint_parser, [a, birds], [tf.float32, tf.float32]), num_parallel_calls=tf.data.experimental.AUTOTUNE)\n    \n    print(f\"-- dataset sizes -- train:{train_len}, val:{val_len} --\")\n\n    return train_dataset, val_dataset, train_len, val_len","metadata":{"execution":{"iopub.status.busy":"2022-05-24T07:00:32.662436Z","iopub.execute_input":"2022-05-24T07:00:32.662726Z","iopub.status.idle":"2022-05-24T07:00:32.683653Z","shell.execute_reply.started":"2022-05-24T07:00:32.662697Z","shell.execute_reply":"2022-05-24T07:00:32.683005Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ResNet models\n\nIncludes Resnet34, Resnet34v2 and Resnet50v2","metadata":{}},{"cell_type":"code","source":"\"\"\"\nresnet34 architecture. adapted + thanks to:\n    https://www.analyticsvidhya.com/blog/2021/08/how-to-code-your-resnet-from-scratch-in-tensorflow/\n    \ntested on mnist datafirst. no problem. resnet has a minimum size, so some padding is needed for MNIST data.\n\nyou need to pre-define the input size of the spectrogram. can be computed in the globals: (N_FFT_FRAMES, FREQ_BANDS, 1)\n\"\"\"\ndef res_block(input_tensor, n_filters, kernel_size, strides=(1,1), activation='relu', padding='same', kernel_initializer=\"he_normal\"):\n    x_skip = input_tensor\n    x = layers.Conv2D(n_filters, kernel_size, strides=strides, activation=activation, padding=padding, kernel_initializer=kernel_initializer)(input_tensor)\n    x = layers.BatchNormalization()(x)\n    x = layers.Conv2D(n_filters, kernel_size, strides=strides, activation=None, padding=padding, kernel_initializer=kernel_initializer)(x)\n    x = layers.BatchNormalization()(x)\n    \n    x = layers.Add()([x, x_skip])\n    x = layers.Activation('relu')(x)\n\n    return x\n\ndef conv_res_block(input_tensor, n_filters, kernel_size, strides=(2,2), activation='relu', padding='same', kernel_initializer=\"he_normal\"):\n    #essentially downsamples by striding instead of pooling\n    x_skip = input_tensor\n    x_skip = layers.Conv2D(n_filters, (1,1), strides=strides, activation=None, padding=padding, kernel_initializer=kernel_initializer)(x_skip)\n    \n    x = layers.Conv2D(n_filters, kernel_size, strides=strides, activation=activation, padding=padding, kernel_initializer=kernel_initializer)(input_tensor)\n    x = layers.BatchNormalization()(x)\n    x = layers.Conv2D(n_filters, kernel_size, strides=(1,1), activation=None, padding=padding, kernel_initializer=kernel_initializer)(x)\n    x = layers.BatchNormalization()(x)\n    \n    x = layers.Add()([x, x_skip])\n    x = layers.Activation('relu')(x)\n    \n    return x\n    \ndef res_net34(input_size=(28,28,1), n_output=10, n_base=64, final_act='softmax'):\n    #mnist image is 28, resnet takes in a minimum of 32,32, so let's pad it by (2,2)\n    img_input = layers.Input(input_size)\n    #x = layers.ZeroPadding2D((2,2))(img_input)\n    \n    #from resnet, 34 layers, take a 7x7 conv with n_base, stride 2\n    x = layers.Conv2D(n_base, kernel_size=(7,7), strides=(2,2), activation='relu', padding='same')(img_input)\n    x = layers.BatchNormalization()(x)\n    x = layers.MaxPool2D(pool_size=(3,3), strides=(2,2), padding='same')(x)\n    \n    block_layers = [3, 4, 6, 3]\n    n_filter = n_base\n    \n    for i in range(4):\n        if i == 0:\n            for j in range(block_layers[i]):\n                x = res_block(x, n_filter, kernel_size=(3,3))\n        else:\n            n_filter = n_filter * 2\n            x = conv_res_block(x, n_filter, kernel_size=(3,3))\n            for j in range(block_layers[i] - 1):\n                x = res_block(x, n_filter, kernel_size=(3,3))\n    \n    #finish with an avgpool, FC and softmax\n    x = layers.GlobalAveragePooling2D()(x)\n    out = layers.Dense(n_output, activation=final_act)(x)\n    \n    model = tf.keras.Model(inputs=[img_input], outputs=[out])\n    return model","metadata":{"execution":{"iopub.status.busy":"2022-05-24T06:56:29.578986Z","iopub.execute_input":"2022-05-24T06:56:29.579290Z","iopub.status.idle":"2022-05-24T06:56:29.600102Z","shell.execute_reply.started":"2022-05-24T06:56:29.579255Z","shell.execute_reply":"2022-05-24T06:56:29.599359Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\nresnet34v2 architecture\n\ntested on MNIST. Unused but here to test the v2 implementation of the resblocks. (reordering activation and concatenation operations)\n\nkey difference between v1 and v2 is BN - Relu - Conv2d instead of Conv - Relu - BN\n\"\"\"\ndef res_blockv2(input_tensor, n_filters, kernel_size, strides=(1,1), activation='relu', padding='same', kernel_initializer=\"he_normal\"):\n    x_skip = input_tensor\n    \n    x = layers.BatchNormalization()(input_tensor)\n    \n    x = layers.Activation(activation)(x)\n    x = layers.Conv2D(n_filters, kernel_size, strides=strides, activation=None, padding=padding, kernel_initializer=kernel_initializer)(x)\n    x = layers.BatchNormalization()(x)\n    x = layers.Activation(activation)(x)\n    x = layers.Conv2D(n_filters, kernel_size, strides=strides, activation=None, padding=padding, kernel_initializer=kernel_initializer)(x)\n    \n    tf.debugging.check_numerics(x, \"x is producing nans!\")\n    tf.debugging.check_numerics(x_skip, \"x_skip is producing nans!\")\n    \n    x = layers.Add()([x, x_skip])\n\n    return x\n\ndef conv_res_blockv2(input_tensor, n_filters, kernel_size, strides=(2,2), activation='relu', padding='same', kernel_initializer=\"he_normal\"):\n    #essentially downsamples by striding instead of pooling\n    x_skip = input_tensor\n    x_skip = layers.Conv2D(n_filters, (1,1), strides=strides, activation=None, padding=padding, kernel_initializer=kernel_initializer)(x_skip)\n    \n    x = layers.BatchNormalization()(input_tensor)\n    x = layers.Activation(activation)(x)\n    x = layers.Conv2D(n_filters, kernel_size, strides=strides, activation=None, padding=padding, kernel_initializer=kernel_initializer)(x)\n    \n    x = layers.BatchNormalization()(x)\n    x = layers.Activation(activation)(x)\n    x = layers.Conv2D(n_filters, kernel_size, strides=(1,1), activation=None, padding=padding, kernel_initializer=kernel_initializer)(x)\n    \n    tf.debugging.check_numerics(x, \"conv x is producing nans!\")\n    tf.debugging.check_numerics(x_skip, \"conv x_skip is producing nans!\")\n    \n    x = layers.Add()([x, x_skip])\n    \n    return x\n    \ndef res_net34v2(input_size=(28,28,1), n_output=10, n_base=64):\n    #mnist image is 28, resnet takes in a minimum of 32,32, so let's pad it by (2,2)\n    img_input = layers.Input(input_size)\n    x = layers.ZeroPadding2D((2,2))(img_input)\n    \n    #from resnet, 34 layers, take a 7x7 conv with n_base, stride 2\n    x = layers.Conv2D(n_base, kernel_size=(7,7), strides=(2,2), activation='relu', padding='same')(x)\n    x = layers.BatchNormalization()(x)\n    x = layers.MaxPool2D(pool_size=(3,3), strides=(2,2), padding='same')(x)\n    \n    block_layers = [3, 4, 6, 3]\n    n_filter = n_base\n    \n    for i in range(4):\n        if i == 0:\n            for j in range(block_layers[i]):\n                x = res_blockv2(x, n_filter, kernel_size=(3,3))\n        else:\n            n_filter = n_filter * 2\n            x = conv_res_blockv2(x, n_filter, kernel_size=(3,3))\n            for j in range(block_layers[i] - 1):\n                x = res_blockv2(x, n_filter, kernel_size=(3,3))\n    \n    #finish with an avgpool, FC and softmax\n    x = layers.GlobalAveragePooling2D()(x)\n    tf.debugging.check_numerics(x, \"final x is producing nans!\")\n    out = layers.Dense(n_output, activation='softmax')(x)\n    \n    model = tf.keras.Model(inputs=[img_input], outputs=[out])\n    return model","metadata":{"execution":{"iopub.status.busy":"2022-05-24T06:56:31.490858Z","iopub.execute_input":"2022-05-24T06:56:31.491685Z","iopub.status.idle":"2022-05-24T06:56:31.513536Z","shell.execute_reply.started":"2022-05-24T06:56:31.491633Z","shell.execute_reply":"2022-05-24T06:56:31.512318Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\nresnet50v2 architecture.\nyou need to pre-define the input size of the spectrogram. can be computed in the globals: (N_FFT_FRAMES, FREQ_BANDS, 1)\n\nkey difference between v1 and v2 is BN - Relu - Conv2d instead of Conv - Relu - BN\n\nThis is our preferred training model. Tested on MNIST data.\n\nFunky interaction with AdamW optimizer (Adam + weight decay).\n\"\"\"\n\n\ndef res50_block(input_tensor, n_filters, kernel_size, strides=(1,1), activation='relu', padding='same', kernel_initializer=\"he_normal\"):\n    \"\"\"\n    slightly different to res32 blocks. (1x1, n_filter) -> (3x3, n_filter) -> (1x1, n_filter * 4)\n    \"\"\"\n    x_skip = input_tensor\n    x_skip = layers.Conv2D(n_filters * 4, (1,1), strides=strides, activation=None, padding=padding, kernel_initializer=kernel_initializer)(x_skip)\n    \n    x = layers.BatchNormalization()(input_tensor)\n    x = layers.Activation(activation)(x)\n    x = layers.Conv2D(n_filters, (1,1), strides=strides, activation=None, padding=padding, kernel_initializer=kernel_initializer)(x)\n    \n    x = layers.BatchNormalization()(x)\n    x = layers.Activation(activation)(x)\n    x = layers.Conv2D(n_filters, kernel_size, strides=strides, activation=None, padding=padding, kernel_initializer=kernel_initializer)(x)\n    \n    x = layers.BatchNormalization()(x)\n    x = layers.Activation(activation)(x)\n    x = layers.Conv2D(n_filters * 4, (1,1), strides=strides, activation=None, padding=padding, kernel_initializer=kernel_initializer)(x)\n    \n    tf.debugging.check_numerics(x, \"x is producing nans!\")\n    tf.debugging.check_numerics(x_skip, \"x_skip is producing nans!\")\n    \n    x = layers.Add()([x, x_skip])\n\n    return x\n\ndef conv_res50_block(input_tensor, n_filters, kernel_size, strides=(2,2), activation='relu', padding='same', kernel_initializer=\"he_normal\"):\n    #essentially downsamples by striding instead of pooling\n    x_skip = input_tensor\n    x_skip = layers.Conv2D(n_filters * 4, (1,1), strides=strides, activation=None, padding=padding, kernel_initializer=kernel_initializer)(x_skip)\n    \n    x = layers.BatchNormalization()(input_tensor)\n    x = layers.Activation(activation)(x)\n    x = layers.Conv2D(n_filters, (1,1), strides=(1,1), activation=None, padding=padding, kernel_initializer=kernel_initializer)(x)\n    \n    x = layers.BatchNormalization()(x)\n    x = layers.Activation(activation)(x)\n    x = layers.Conv2D(n_filters, kernel_size, strides=strides, activation=None, padding=padding, kernel_initializer=kernel_initializer)(x)\n    \n    x = layers.BatchNormalization()(x)\n    x = layers.Activation(activation)(x)\n    x = layers.Conv2D(n_filters * 4, (1,1), strides=(1,1), activation=None, padding=padding, kernel_initializer=kernel_initializer)(x)\n    \n    tf.debugging.check_numerics(x, \"conv x is producing nans!\")\n    tf.debugging.check_numerics(x_skip, \"conv x_skip is producing nans!\")\n    \n    x = layers.Add()([x, x_skip])\n    \n    return x\n    \ndef res_net50(input_size=INPUT_SIZE, n_output=len(SCORED_BIRDS), n_base=64, final_act='softmax'):\n    \n    img_input = layers.Input(input_size)\n    x = layers.ZeroPadding2D((2,2))(img_input)\n    \n    x = layers.Conv2D(n_base, kernel_size=(7,7), strides=(2,2), activation='relu', padding='same')(x)\n    x = layers.BatchNormalization()(x)\n    x = layers.MaxPool2D(pool_size=(3,3), strides=(2,2), padding='same')(x)\n    \n    block_layers = [3, 4, 6, 3]\n    n_filter = n_base\n    \n    for i in range(4):\n        if i == 0:\n            for j in range(block_layers[i]):\n                x = res50_block(x, n_filter, kernel_size=(3,3))\n        else:\n            n_filter = n_filter * 2\n            x = conv_res50_block(x, n_filter, kernel_size=(3,3))\n            for j in range(block_layers[i] - 1):\n                x = res50_block(x, n_filter, kernel_size=(3,3))\n    \n    #finish with an avgpool, FC and softmax\n    \n    x = layers.Activation('relu')(x)\n    x = layers.BatchNormalization()(x)\n    x = layers.GlobalAveragePooling2D()(x)\n    out = layers.Dense(n_output, activation=final_act)(x)\n    \n    model = tf.keras.Model(inputs=[img_input], outputs=[out])\n    return model","metadata":{"execution":{"iopub.status.busy":"2022-05-24T06:56:35.968046Z","iopub.execute_input":"2022-05-24T06:56:35.968728Z","iopub.status.idle":"2022-05-24T06:56:35.994747Z","shell.execute_reply.started":"2022-05-24T06:56:35.968687Z","shell.execute_reply":"2022-05-24T06:56:35.993692Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Running and training the model","metadata":{}},{"cell_type":"code","source":"\"\"\"\nIf you've generated the dataframe earlier, you can load it here.\n\"\"\"\n# index_frame = pd.read_pickle(\"./training_df.pickle\")\n# print(index_frame.head())","metadata":{"execution":{"iopub.status.busy":"2022-05-24T06:57:45.496964Z","iopub.execute_input":"2022-05-24T06:57:45.497243Z","iopub.status.idle":"2022-05-24T06:57:45.524483Z","shell.execute_reply.started":"2022-05-24T06:57:45.497214Z","shell.execute_reply":"2022-05-24T06:57:45.522986Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\nGenerating dataset + mixing dataset. Ensure that the mixing dataset is shuffled a SECOND time via the second_shuffle flag such that\nnot mixing the same audio file with itself.\n\nCache (if possible) and repeat.\n\"\"\"\n\ntrain_ds, val_ds, train_len, val_len = construct_dataset(index_frame, birds=SCORED_BIRDS, fraction=0.20, seed=SEED)\n\nmix_ds, _, _ , _ = construct_dataset(index_frame, birds=SCORED_BIRDS, fraction=0.20, seed=SEED, second_shuffle=True)\nmix_ds = mix_ds.cache().repeat()\n","metadata":{"execution":{"iopub.status.busy":"2022-05-24T07:10:36.754132Z","iopub.execute_input":"2022-05-24T07:10:36.754433Z","iopub.status.idle":"2022-05-24T07:10:37.730267Z","shell.execute_reply.started":"2022-05-24T07:10:36.754401Z","shell.execute_reply":"2022-05-24T07:10:37.729449Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\nNet parameters\n\nIncluded is the compile for BCE + Sigmoid if you wish to use that instead.\n\"\"\"\ndef weighted_cce(y_true, y_pred):\n    \"\"\"\n    loss function. categorical crossentropy, but weighted by 1/K.\n    Multi-label training with softmax + CCE. tests by facebook (see. https://arxiv.org/pdf/1805.00932.pdf) show that this offers a\n    moderate improvement to training, at the expense of inference ability. (you don't know number of labels when inferring,\n    so you need to guess or have another model)\n    \"\"\"\n    y_scale = (1/tf.math.reduce_sum(y_true, axis=-1))[:,tf.newaxis] #shape (batch, 1)\n    y_new = tf.math.multiply(y_true, y_scale)\n    loss = tf.keras.losses.categorical_crossentropy(y_new, y_pred)\n    \n    return loss\n\nEPOCHS = 100\nBATCH_SIZE = 32 #16\nSTEPS_PER_EPOCH = train_len // BATCH_SIZE\n\nmodel = res_net50(input_size=INPUT_SIZE, n_output=len(SCORED_BIRDS), final_act='softmax')\n\noptimizer = tf.keras.optimizers.Adam(learning_rate=1e-4, epsilon=1e-1)\n\n#model.compile(loss='binary_crossentropy', optimizer=optimizer, metrics=['binary_accuracy', tfa.metrics.FBetaScore(num_classes=len(SCORED_BIRDS), average='macro')])\nmodel.compile(loss=weighted_cce, optimizer=optimizer, metrics=[tfa.metrics.FBetaScore(num_classes=len(SCORED_BIRDS), average='macro')])\nmodel.summary()","metadata":{"execution":{"iopub.status.busy":"2022-05-24T07:10:55.916470Z","iopub.execute_input":"2022-05-24T07:10:55.916790Z","iopub.status.idle":"2022-05-24T07:10:57.853785Z","shell.execute_reply.started":"2022-05-24T07:10:55.916755Z","shell.execute_reply":"2022-05-24T07:10:57.852766Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\nCallbacks for training\n\"\"\"\nearlystopping = callbacks.EarlyStopping(monitor='val_loss', verbose=1, min_delta=1e-4, patience=5)\n\nplateau = callbacks.ReduceLROnPlateau(factor=0.5, patience=3, min_lr=1e-6, verbose=1)\n\ncheckpoint = callbacks.ModelCheckpoint(f'./checkpoint.hdf5', monitor='val_loss', verbose=1, save_best_only=True)\n\ncsv_logger = callbacks.CSVLogger(f'./log.out', separator=',')\n\ncallbacks = [earlystopping, plateau, checkpoint, csv_logger]\n","metadata":{"execution":{"iopub.status.busy":"2022-05-24T07:14:10.265157Z","iopub.execute_input":"2022-05-24T07:14:10.265449Z","iopub.status.idle":"2022-05-24T07:14:10.272631Z","shell.execute_reply.started":"2022-05-24T07:14:10.265420Z","shell.execute_reply":"2022-05-24T07:14:10.271700Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\nAugmenting dataset\nRemember to normalize + add single channel at the end.\nBy experience in Kaggle, TRAIN_SHUFFLE, VAL_SHUFFLE must be low, otherwise set it to train_len and val_len respectively\n\nONLY CACHE if you are able to fill your shuffle buffer with all the elements. otherwise remove it or live with the warnings\n\"\"\"\nTRAIN_SHUFFLE = 1 #train_len\nVAL_SHUFFLE = 1 #val_len\n\ntrain_dataset = train_ds.cache().shuffle(TRAIN_SHUFFLE) \\\n                                .map(time_roll) \\\n                                .map(lambda x1, y1: random_effect(x1,y1,white_noise, 0.1)) \\\n                                .map(lambda x2, y2: random_effect(x2, y2, pink_noise, 0.3)) \\\n                                .map(lambda x3, y3: random_mix_wrapper(x3, y3, mix_ds, 0.4)) \\\n                                .map(lambda x4, y4: tf.py_function(convert_mel, [x4, y4], [tf.float32, tf.float32])) \\\n                                .map(normalize_audio).map(add_single_channel).batch(BATCH_SIZE).repeat()\n\nval_dataset = val_ds.map(lambda x4, y4: tf.py_function(convert_mel, [x4, y4], [tf.float32, tf.float32])) \\\n                    .map(normalize_audio) \\\n                    .map(add_single_channel) \\\n                    .cache().shuffle(VAL_SHUFFLE).batch(BATCH_SIZE)","metadata":{"execution":{"iopub.status.busy":"2022-05-24T07:27:43.467000Z","iopub.execute_input":"2022-05-24T07:27:43.468819Z","iopub.status.idle":"2022-05-24T07:27:43.565736Z","shell.execute_reply.started":"2022-05-24T07:27:43.468758Z","shell.execute_reply":"2022-05-24T07:27:43.564778Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\nTraining the model. Tested and works, but as stated, you'll want to do this on an external computer or live with the thousand warnings.\n\"\"\"\n#os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3' #Suppress warnings if you want\n\nmodel_history = model.fit(train_dataset,\n                          epochs=EPOCHS,\n                          steps_per_epoch=STEPS_PER_EPOCH,\n                          validation_data=val_dataset,\n                          callbacks=callbacks)","metadata":{"execution":{"iopub.status.busy":"2022-05-24T07:43:51.726378Z","iopub.execute_input":"2022-05-24T07:43:51.726813Z","iopub.status.idle":"2022-05-24T07:43:51.734493Z","shell.execute_reply.started":"2022-05-24T07:43:51.726762Z","shell.execute_reply":"2022-05-24T07:43:51.733900Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\nSave model as hdf5\n\"\"\"\nmodel.save(f'model.hdf5')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Plot model training history","metadata":{}},{"cell_type":"code","source":"\"\"\"\nSimple plots. You can query more via model_history.history['(metric)']\n\"\"\"\n\nloss = model_history.history['loss']\nval_loss = model_history.history['val_loss']\n\nfig = plt.figure()\nax = fig.add_subplot(111)\n\nax.plot(epochs, loss, 'r', label='Training loss')\nax.plot(epochs, val_loss, 'bo', label='Validation loss')\n\nax.set_xlabel('Epoch')\nax.set_ylabel('Loss Value')\n\nax.set_ylim([0,np.max(loss[0],val_loss[0])])\nax.legend()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\nLoad model if you have.\n\nEnsure you have compile=False unless you have the custom loss function defined elsewhere.\n\"\"\"\n#model = tf.keras.models.load_model('./model.hdf5', compile=False)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Testing pipeline + Examples\n\nSome examples to visualise outputs from the Dataset generator","metadata":{}},{"cell_type":"code","source":"\"\"\"\nTake one element from val_ds (NOT val_dataset, which is the batched transformed images used in validation), apply transformations and\nplot the result. Make sure you run the appropriate section/s that defines the augmentation and the dataset.\n\"\"\"\nfor e in val_ds.take(1).map(lambda x1, y1: random_effect(x1, y1, white_noise, 0.5)) \\\n                       .map(lambda x2,y2: random_effect(x2, y2, pink_noise, 0.5)) \\\n                       .map(lambda x3, y3: random_mix_wrapper(x3, y3, mix_ds, 0.5)) \\\n                       .map(lambda x4, y4: tf.py_function(convert_mel, [x4, y4], [tf.float32,tf.float32])) \\\n                       .map(normalize_audio):\n    print(f'Label vector: {e[1]}')\n    \n    fig, ax = plt.subplots()\n    img = librosa.display.specshow(e[0].numpy().T, sr=SR, y_axis='mel', x_axis='time', ax=ax, vmin=-1, vmax=1, cmap='coolwarm')\n    ax.set_title('Mel spectrogram')\n    fig.colorbar(img, ax=ax)\n    \nfor e in val_ds.take(1).map(lambda x4, y4: tf.py_function(convert_mel, [x4, y4], [tf.float32,tf.float32])) \\\n                       .map(normalize_audio):\n    print(f'Label vector: {e[1]}')\n    \n    fig, ax = plt.subplots()\n    img = librosa.display.specshow(e[0].numpy().T, sr=SR, y_axis='mel', x_axis='time', ax=ax, vmin=-1, vmax=1, cmap='coolwarm')\n    ax.set_title('Mel spectrogram')\n    fig.colorbar(img, ax=ax)","metadata":{"execution":{"iopub.status.busy":"2022-05-24T09:28:26.904127Z","iopub.execute_input":"2022-05-24T09:28:26.904651Z","iopub.status.idle":"2022-05-24T09:28:30.381820Z","shell.execute_reply.started":"2022-05-24T09:28:26.904617Z","shell.execute_reply":"2022-05-24T09:28:30.380783Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}