{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":70203,"databundleVersionId":8068726,"sourceType":"competition"},{"sourceId":10879416,"sourceType":"datasetVersion","datasetId":6759777},{"sourceId":10896405,"sourceType":"datasetVersion","datasetId":6771629},{"sourceId":10899844,"sourceType":"datasetVersion","datasetId":6774134}],"dockerImageVersionId":30918,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Audio classification\nThis notebook covers audio classification through melspectograms, image classification techniques through transfer learning, and finetuning. As well as learning rate schedulers, dataset preparation, preprocessing, batching and shuffling\n","metadata":{}},{"cell_type":"markdown","source":"First we load the audio, with the default sample rate, and convert it to melspectograms (from amplitude-time) to (frequency-time) since frequency based numerics give us better info about the audio. The parameters used for the conversion to melsepctograms are famously used for this dataset, yielding better results.Then we convert it to logarithmic scale, (loudness is a logarithmic parameter).\n\nWe convert it to an RGB like 3-channel array, so that it can be compatible with the model. We resize to ensure uniformity. Preprocess function converts tensorflow based variables (given as input for training, under graph execution) into numpy accessible quentities, by using py_function wrapper","metadata":{}},{"cell_type":"code","source":"import librosa\nimport librosa.display\nimport numpy as np\nimport matplotlib.pyplot as plt\nfrom PIL import Image\nimport cv2\n\ndef audio_to_melspectrogram_image(audio_path, sr=22050, n_fft=2048, hop_length=512, n_mels=128, f_min=20, f_max=16000, duration=5, img_size=256):\n    audio_path = audio_path.numpy().decode(\"utf-8\")\n    # Load the first 'duration' seconds of audio\n    y, sr = librosa.load(audio_path, sr = sr)\n    \n    # Compute mel spectrogram\n    mel_spec = librosa.feature.melspectrogram(y=y, sr=sr, n_fft=2048, hop_length=512, \n    n_mels=128, fmax=sr // 2)\n    \n    # Convert to log scale (dB)\n    mel_spec_db = librosa.power_to_db(mel_spec, ref=np.max)\n    mel_spec_norm = 255 * (mel_spec_db - mel_spec_db.min()) / (mel_spec_db.max() - mel_spec_db.min())\n    mel_spec_norm = mel_spec_norm.astype(np.float32)\n    mel_image = Image.fromarray(mel_spec_norm)\n    mel_image = mel_image.resize((img_size, img_size), Image.LANCZOS)\n    \n    # Convert to 3-channel image\n    mel_image = np.stack([mel_image] * 3, axis=-1)\n    return mel_image\n\ndef preprocess(file_path):\n    features = tf.py_function(\n            func=audio_to_melspectrogram_image, \n            inp=[file_path], \n            Tout=tf.float32\n        )\n    \n    return features\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2025-03-02T10:09:33.489688Z","iopub.execute_input":"2025-03-02T10:09:33.489964Z","iopub.status.idle":"2025-03-02T10:09:33.532133Z","shell.execute_reply.started":"2025-03-02T10:09:33.489944Z","shell.execute_reply":"2025-03-02T10:09:33.531342Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"audio.csv consists of the audios that last more than 5 seconds, audios that give us a better quality. According to the various discussions forums, it was better to clip to the audios by a certain time, hence assuring uniformity in the duration of the audio, and also capturing most important aspect of the data.","metadata":{}},{"cell_type":"code","source":"import os\nimport pandas as pd\ndf =pd.read_csv('/kaggle/input/audio-files/audio.csv')\nbird_classes = os.listdir('/kaggle/input/birdclef-2024/train_audio')\nlabels = []\nfiles = []\nfor i in df['audio']:\n    files.append(i)\n    bc = i.split('/')[-2]\n    labels.append(bird_classes.index(bc))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-02T10:09:36.313600Z","iopub.execute_input":"2025-03-02T10:09:36.313899Z","iopub.status.idle":"2025-03-02T10:09:36.417468Z","shell.execute_reply.started":"2025-03-02T10:09:36.313875Z","shell.execute_reply":"2025-03-02T10:09:36.416835Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Getting training and testing data, shuffling labels, features simultaneausly by providing a seed","metadata":{}},{"cell_type":"code","source":"import random\nrandom.seed(123)\nrandom.shuffle(files)\nrandom.seed(123)\nrandom.shuffle(labels)\ntrain_sample = int(len(files)*0.9)\ntraining_files = files[:train_sample]\ntraining_labels = labels[:train_sample]\n\ntesting_files = files[train_sample:]\ntesting_labels = labels[train_sample:]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-02T10:09:38.837603Z","iopub.execute_input":"2025-03-02T10:09:38.837891Z","iopub.status.idle":"2025-03-02T10:09:38.865007Z","shell.execute_reply.started":"2025-03-02T10:09:38.837870Z","shell.execute_reply":"2025-03-02T10:09:38.864296Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"We use tensorflow's graph based execution, to map the training data to its preprocessing function, that has been wrapped around the py_function to allow compatibility with numpy. We create for labels and audio datasets and zip them together, then batch them up. Same being done with test data","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\ntraining_dataset_files = tf.data.Dataset.from_tensor_slices(training_files)\ntraining_dataset_files = training_dataset_files.map(preprocess, num_parallel_calls=tf.data.AUTOTUNE)\ntraining_dataset_labels = tf.data.Dataset.from_tensor_slices(training_labels)\ntraining_data = tf.data.Dataset.zip((training_dataset_files, training_dataset_labels))\ntraining_data = training_data.map(lambda audio, label: (tf.ensure_shape(audio, (256,256,3)), \n                                          tf.ensure_shape(label, ())))\n                                          \ntraining_data = training_data.batch(64)\n\ntesting_dataset_files = tf.data.Dataset.from_tensor_slices(testing_files)\ntesting_dataset_files = testing_dataset_files.map(preprocess, num_parallel_calls=tf.data.AUTOTUNE)\ntesting_dataset_labels = tf.data.Dataset.from_tensor_slices(testing_labels)\ntesting_data = tf.data.Dataset.zip((testing_dataset_files, testing_dataset_labels))\ntesting_data = testing_data.map(lambda audio, label: (tf.ensure_shape(audio, (256,256,3)), \n                                          tf.ensure_shape(label, ())))\n                                          \ntesting_data = testing_data.batch(64)\n\n\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-02T10:09:40.975357Z","iopub.execute_input":"2025-03-02T10:09:40.975633Z","iopub.status.idle":"2025-03-02T10:09:41.159486Z","shell.execute_reply.started":"2025-03-02T10:09:40.975613Z","shell.execute_reply":"2025-03-02T10:09:41.158838Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"This class applies cosine annealing learning rate scheduler that ensures linear increase (for a certain set of warmup epochs) in learning rate followed by a cosine decay","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\nimport numpy as np\n\nclass CosineAnnealingWithWarmup(tf.keras.callbacks.Callback):\n    def __init__(self, total_epochs, warmup_epochs=5, peak_lr=1e-4):\n        super().__init__()\n        self.total_epochs = total_epochs\n        self.warmup_epochs = warmup_epochs\n        self.peak_lr = peak_lr\n        self.current_epoch = 0\n\n    def on_epoch_begin(self, epoch, logs=None):\n        self.current_epoch = epoch + 1  # Keras epoch starts from 0\n        new_lr = self.compute_lr()\n        self.model.optimizer.learning_rate.assign(new_lr)\n        print(f\"Epoch {self.current_epoch}: Learning Rate = {new_lr:.6f}\")\n\n    def compute_lr(self):\n        if self.current_epoch <= self.warmup_epochs:\n            # Linear Warmup: Increase LR linearly to peak_lr\n            return (self.peak_lr / self.warmup_epochs) * self.current_epoch\n        else:\n            # Cosine Annealing\n            progress = (self.current_epoch - self.warmup_epochs) / (self.total_epochs - self.warmup_epochs)\n            return 0.5 * self.peak_lr * (1 + np.cos(np.pi * progress))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-01T22:12:32.479956Z","iopub.execute_input":"2025-03-01T22:12:32.480298Z","iopub.status.idle":"2025-03-01T22:12:32.548356Z","shell.execute_reply.started":"2025-03-01T22:12:32.480272Z","shell.execute_reply":"2025-03-01T22:12:32.547574Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Model checkpointing for each epoch","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.callbacks import ModelCheckpoint\n\n# Define the checkpoint callback\ncheckpoint_callback = ModelCheckpoint(\n    filepath=\"model_checkpoint_epoch_{epoch:02d}.keras\",  # Save model after each epoch\n    save_weights_only=False,  # Set to True if you only want to save weights\n    save_best_only=False,  # Set to True to save only the best model based on validation loss\n    verbose=1\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-01T22:12:35.071852Z","iopub.execute_input":"2025-03-01T22:12:35.072159Z","iopub.status.idle":"2025-03-01T22:12:35.079449Z","shell.execute_reply.started":"2025-03-01T22:12:35.072134Z","shell.execute_reply":"2025-03-01T22:12:35.078767Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Building the model through trasfer learning with efficientnetB0, applying flip and cutout augmentations followed by our very own output layer. They are all strung up through the functional API keras. We use early stopping callback asw. Here only 16 epochs was set, but it is better if 30-40 epochs are used for this model. With 16 epochs it reached an accuracy of around 18%. \n\nThis pretty good considering the fact that there is around 184 classes to choose from. So these predictions are no way near random.","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\nimport keras_cv \nfrom tensorflow.keras import layers\nfrom tensorflow.keras.applications.efficientnet_v2 import EfficientNetV2B0, preprocess_input\n\n\n# Ensure bird_classes is defined\nnum_classes = len(bird_classes)  \n\n# Load base model\nbase_model = EfficientNetV2B0(include_top=False)\nbase_model.trainable = False  # Freeze for feature extraction\n\n# Define Model\ninput_shape = (256, 256, 3)\ninputs = layers.Input(shape=input_shape, name=\"input_layer\")\nx = tf.keras.layers.RandomFlip(mode=\"horizontal\")(inputs)  # Horizontal Flip\nx = keras_cv.layers.RandomCutout(height_factor=0.2, width_factor=0.2)(x)\nx = base_model(x, training=False)  # Keep batchnorm frozen\nx = layers.GlobalAveragePooling2D(name=\"global_average_pooling_layer\")(x)\noutputs = layers.Dense(num_classes, activation=\"softmax\", name=\"output_layer\")(x)\nmodel = tf.keras.Model(inputs, outputs)\n\noptimizer = tf.keras.optimizers.Adam(learning_rate=1e-6)\nmodel.compile(optimizer=optimizer, loss='sparse_categorical_crossentropy', metrics=['accuracy'])\n\ntotal_epochs = 16\nwarmup_epochs = 5\npeak_lr = 1e-4\ncosine_warmup_callback = CosineAnnealingWithWarmup(total_epochs, warmup_epochs, peak_lr)\n\n# Early stopping callback (Accuracy monitoring)\nearly_stopping = tf.keras.callbacks.EarlyStopping(\n    monitor=\"val_accuracy\", patience=7, mode=\"max\", restore_best_weights=True\n)\n\n'''\nmodel.fit(\n    training_data,\n    validation_data=testing_data,\n    epochs=total_epochs,\n    batch_size=64,\n    callbacks=[cosine_warmup_callback, early_stopping,checkpoint_callback]\n)\n'''","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-01T22:13:07.023542Z","iopub.execute_input":"2025-03-01T22:13:07.023832Z","iopub.status.idle":"2025-03-02T05:47:45.911558Z","shell.execute_reply.started":"2025-03-01T22:13:07.023811Z","shell.execute_reply":"2025-03-02T05:47:45.910617Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Here we apply finetuning tehcniques to unfreeze the top 10 layers of the model by changing their trainable property to True. This means more trainable weights, hence higher efficiency which obviously seems to be the case, as within 5 epochs the val_accuracy went up to 36%\n\nThis is the code to make the last 10 layers of the base efficientnet trainable:","metadata":{}},{"cell_type":"code","source":"'''\nbase_model.trainable = True\nfor layer in model_2_base_model.layers[:-10]:\n  layer.trainable = False\n'''","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Here i just loaded one of my already pretrained (fientuned model) for further training. IF u want to unfreeze the layers the execute the code above, then ignore the one below","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\nmodel = tf.keras.models.load_model(\n    '/kaggle/input/best-model/model_checkpoint_epochft_02.keras'\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-02T10:06:45.319694Z","iopub.execute_input":"2025-03-02T10:06:45.319965Z","iopub.status.idle":"2025-03-02T10:07:15.406306Z","shell.execute_reply.started":"2025-03-02T10:06:45.319944Z","shell.execute_reply":"2025-03-02T10:07:15.405644Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"from tensorflow.keras.callbacks import ModelCheckpoint\ncheckpoint_callback = ModelCheckpoint(\n    filepath=\"model_checkpoint_epochft_{epoch:02d}.keras\",  # Save model after each epoch\n    save_weights_only=False,  # Set to True if you only want to save weights\n    save_best_only=False,  # Set to True to save only the best model based on validation loss\n    verbose=1\n)\nmodel.compile(loss=\"sparse_categorical_crossentropy\",\n                optimizer=tf.keras.optimizers.Adam(learning_rate=0.0001), # lr is 10x lower than before for fine-tuning\n                metrics=[\"accuracy\"])\n\n\n'''\nmodel.fit(\n    training_data,\n    validation_data=testing_data,\n    epochs=5,\n    batch_size=64,\n    callbacks=[checkpoint_callback]\n)\n'''","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-02T10:09:47.681728Z","iopub.execute_input":"2025-03-02T10:09:47.682011Z","iopub.status.idle":"2025-03-02T12:31:41.732869Z","shell.execute_reply.started":"2025-03-02T10:09:47.681990Z","shell.execute_reply":"2025-03-02T12:31:41.732135Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"fin_model = tf.keras.models.load_model(\n    '/kaggle/input/best-weight-36/model_checkpoint_epochft_05.keras'\n)\nfor i,o in testing_data.take(1):\n    for audio,label in zip(i,o):\n        res = model.predict(np.expand_dims(audio, axis=0))\n        p_index = np.argmax(res)\n        print('predicted result: ', bird_classes[p_index])\n        print('actual result: ', bird_classes[label])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-02T12:38:28.673184Z","iopub.execute_input":"2025-03-02T12:38:28.673943Z","iopub.status.idle":"2025-03-02T12:38:43.445932Z","shell.execute_reply.started":"2025-03-02T12:38:28.673912Z","shell.execute_reply":"2025-03-02T12:38:43.445242Z"}},"outputs":[],"execution_count":null}]}