{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# BirdCLEF 2023 🐦\n> Identify bird calls in soundscapes\n\n","metadata":{}},{"cell_type":"markdown","source":"# Methodology 🎯\n* This notebook will demonstrate **Bird Call Identification** with `TensorFlow`. \n* This notebook will also show how to use `tf.data` for **Audio Processing** task.\n* This notebook also demonstrates how to extract **Spectrogram** features from **Raw Audio** using TensorFlow. Specifically how to read `.ogg` file in TensorFlow.\n* This notebook will also show how to set up **Augmentation Pipeline** for audio data. It will also implement some cool augmentations such as **CutMix** and **MixUp** for **Audio** data.\n* This notebook also shows how we can use **Filter Stride Reduction (FSR)** to improve model's performence.\n* This notebook is compatible with both **GPU**, **TPU**, and newly launched **TPU-VM** Device is automatically selected so you won't have to do anything to allocate device.","metadata":{}},{"cell_type":"markdown","source":"# Notebooks 📓\n\n* EffNet + FSR + CutMixUp\n * Train: [BirdCLEF23: EffNet + FSR + CutMixUp [Train]](https://www.kaggle.com/awsaf49/birdclef23-effnet-fsr-cutmixup-train/edit)\n * Infer: [BirdCLEF23: EffNet + FSR + CutMixUp [Infer]](https://www.kaggle.com/awsaf49/birdclef23-effnet-fsr-cutmixup-infer/edit)\n","metadata":{}},{"cell_type":"markdown","source":"# Update 🔄\n* `v10`:\n * Filter data for cross-validation\n * Upsample data for minority class\n* `v13`:\n * Train record duration is reduced to $10$ sec to reduce memory overhead.\n * Spectrogram shape is reduced to $128 \\times 256$","metadata":{}},{"cell_type":"markdown","source":"# Install Libraries 🛠","metadata":{"papermill":{"duration":0.062037,"end_time":"2022-03-08T03:15:20.082763","exception":false,"start_time":"2022-03-08T03:15:20.020726","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## For `TPU VM v3-8`\n\nUsually these libraries come pre-installed for other accelerators but fo tpu-vm we need to install them manually. If you want to use rather remote-TPU or GPU or CPU, then comment out the following cell.\n","metadata":{}},{"cell_type":"code","source":"# tensorflow for tpu-vm\n!pip install -q /lib/wheels/tensorflow-2.9.1-cp38-cp38-linux_x86_64.whl\n\n# tensorflow utilities\n!pip install -q tensorflow-addons==0.18.0\n!pip install -q tensorflow-probability==0.17.0\n!pip install -q tensorflow-io==0.26.0\n\n# other utilies\n!pip install -q opencv-python-headless librosa wandb scikit-learn","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-03-13T19:28:11.610975Z","iopub.execute_input":"2023-03-13T19:28:11.611766Z","iopub.status.idle":"2023-03-13T19:29:31.006681Z","shell.execute_reply.started":"2023-03-13T19:28:11.611730Z","shell.execute_reply":"2023-03-13T19:29:31.005668Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## EfficientNet with Filter Stride Reduction (FSR)\nBy default efficient don't come with this feature thus code need to be modified. In following repo I have modified the code to reduce the stride $2 \\times$ in stem block. You can check the [source code](https://github.com/awsaf49/efficientnet-spec) for more details.","metadata":{}},{"cell_type":"code","source":"# efficientnet with filter stride reduction (FSR)\n!pip install -qU git+https://github.com/awsaf49/efficientnet-spec","metadata":{"papermill":{"duration":31.227648,"end_time":"2022-03-08T03:17:54.356004","exception":false,"start_time":"2022-03-08T03:17:23.128356","status":"completed"},"tags":[],"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-03-13T19:29:31.008547Z","iopub.execute_input":"2023-03-13T19:29:31.008826Z","iopub.status.idle":"2023-03-13T19:29:49.938148Z","shell.execute_reply.started":"2023-03-13T19:29:31.008798Z","shell.execute_reply":"2023-03-13T19:29:49.937219Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Import Libraries 📚","metadata":{"papermill":{"duration":0.065343,"end_time":"2022-03-08T03:18:11.885586","exception":false,"start_time":"2022-03-08T03:18:11.820243","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import os\nimport pandas as pd\npd.options.mode.chained_assignment = None # avoids assignment warning\nimport numpy as np\nimport random\nfrom glob import glob\nfrom tqdm import tqdm\ntqdm.pandas() # enable progress bars in pandas operations\nimport gc\n\nimport librosa\nimport sklearn\nimport json\n\n# Import for visualization\nimport matplotlib as mpl\nimport matplotlib.pyplot as plt\nimport librosa.display as lid\nimport IPython.display as ipd\n\n# Import tensorflow\nimport tensorflow as tf\n# Set logging level to avoid unnecessary messages\ntf.get_logger().setLevel('ERROR')\n# Set autograph verbosity to avoid unnecessary messages\ntf.autograph.set_verbosity(0)\n# Enable xla for speed up\ntf.config.optimizer.set_jit(True)\n\n# Import required tensorflow modules\nimport tensorflow_io as tfio\nimport tensorflow_addons as tfa\nimport tensorflow_probability as tfp\nimport tensorflow.keras.backend as K\n\n# Import KaggleDatasets for accessing Kaggle datasets\nfrom kaggle_datasets import KaggleDatasets","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","papermill":{"duration":2.632068,"end_time":"2022-03-08T03:18:14.585094","exception":false,"start_time":"2022-03-08T03:18:11.953026","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-03-13T19:29:49.939505Z","iopub.execute_input":"2023-03-13T19:29:49.939784Z","iopub.status.idle":"2023-03-13T19:30:06.244397Z","shell.execute_reply.started":"2023-03-13T19:29:49.939755Z","shell.execute_reply":"2023-03-13T19:30:06.243414Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Library Version","metadata":{"papermill":{"duration":0.065649,"end_time":"2022-03-08T03:18:14.717311","exception":false,"start_time":"2022-03-08T03:18:14.651662","status":"completed"},"tags":[]}},{"cell_type":"code","source":"print('np:', np.__version__)\nprint('pd:', pd.__version__)\nprint('sklearn:', sklearn.__version__)\nprint('librosa:', librosa.__version__)\nprint('tf:', tf.__version__)\nprint('tfp:', tfp.__version__)\nprint('tfa:', tfa.__version__)\nprint('tfio:', tfio.__version__)","metadata":{"papermill":{"duration":0.155095,"end_time":"2022-03-08T03:18:14.939054","exception":false,"start_time":"2022-03-08T03:18:14.783959","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-03-13T19:30:06.247838Z","iopub.execute_input":"2023-03-13T19:30:06.248884Z","iopub.status.idle":"2023-03-13T19:30:06.256299Z","shell.execute_reply.started":"2023-03-13T19:30:06.248846Z","shell.execute_reply":"2023-03-13T19:30:06.255281Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Configuration ⚙️","metadata":{"papermill":{"duration":0.066353,"end_time":"2022-03-08T03:18:18.099835","exception":false,"start_time":"2022-03-08T03:18:18.033482","status":"completed"},"tags":[]}},{"cell_type":"code","source":"class CFG:\n # Debugging\n debug = False\n \n # Plot training history\n training_plot = True\n \n # Weights and Biases logging\n wandb = True\n competition = 'birdclef-2023' \n _wandb_kernel = 'awsaf49'\n \n # Experiment name and comment\n exp_name = 'baseline-v2'\n comment = 'EfficientNetB0|FSR|t=10s|128x384|up_thr=50|cv_filter'\n \n # Notebook link\n notebook_link = 'https://www.kaggle.com/awsaf49/birdclef23-effnet-fsr-cutmixup-train/edit'\n \n # Verbosity level\n verbose = 0\n \n # Device and random seed\n device = 'TPU-VM'\n seed = 42\n \n # Input image size and batch size\n img_size = [128, 384]\n batch_size = 32\n upsample_thr = 50 # min sample of each class (upsample)\n cv_filter = True # always keeps low sample data in train\n \n # Inference batch size, test time augmentation, and drop remainder\n infer_bs = 2\n tta = 1\n drop_remainder = True\n \n # Number of epochs, model name, and number of folds\n epochs = 25\n model_name = 'EfficientNetB0'\n fsr = True # reduce stride of stem block\n num_fold = 5\n \n # Selected folds for training and evaluation\n selected_folds = [0]\n\n # Pretraining, neck features, and final activation function\n pretrain = 'imagenet'\n neck_features = 0\n final_act = 'softmax'\n \n # Learning rate, optimizer, and scheduler\n lr = 1e-3\n scheduler = 'cos'\n optimizer = 'Adam' # AdamW, Adam\n \n # Loss function and label smoothing\n loss = 'CCE' # BCE, CCE\n label_smoothing = 0.05 # label smoothing\n \n # Audio duration, sample rate, and length\n duration = 10 # second\n sample_rate = 32000\n audio_len = duration*sample_rate\n \n # STFT parameters\n nfft = 2028\n window = 2048\n hop_length = audio_len // (img_size[1] - 1)\n fmin = 20\n fmax = 16000\n normalize = True\n \n # Data augmentation parameters\n augment=True\n \n # Spec augment\n spec_augment_prob = 0.80\n \n mixup_prob = 0.65\n mixup_alpha = 0.5\n \n cutmix_prob = 0.0\n cutmix_alpha = 0.5\n \n mask_prob = 0.65\n freq_mask = 20\n time_mask = 30\n\n\n # Audio Augmentation Settings\n audio_augment_prob = 0.5\n \n timeshift_prob = 0.0\n \n gn_prob = 0.35\n\n # Data Preprocessing Settings\n class_names = sorted(os.listdir('/kaggle/input/birdclef-2023/train_audio/'))\n num_classes = len(class_names)\n class_labels = list(range(num_classes))\n label2name = dict(zip(class_labels, class_names))\n name2label = {v:k for k,v in label2name.items()}\n\n # Training Settings\n target_col = ['target']\n tab_cols = ['filename']\n monitor = 'auc'","metadata":{"papermill":{"duration":0.156464,"end_time":"2022-03-08T03:18:18.322809","exception":false,"start_time":"2022-03-08T03:18:18.166345","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-03-13T19:30:06.257817Z","iopub.execute_input":"2023-03-13T19:30:06.258213Z","iopub.status.idle":"2023-03-13T19:30:06.304978Z","shell.execute_reply.started":"2023-03-13T19:30:06.258173Z","shell.execute_reply":"2023-03-13T19:30:06.304035Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Reproducibility ♻️\nSets value for random seed to produce similar result in each run.","metadata":{"papermill":{"duration":0.070351,"end_time":"2022-03-08T03:18:18.46058","exception":false,"start_time":"2022-03-08T03:18:18.390229","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def seeding(SEED):\n np.random.seed(SEED)\n random.seed(SEED)\n os.environ['PYTHONHASHSEED'] = str(SEED)\n# os.environ['TF_CUDNN_DETERMINISTIC'] = str(SEED)\n tf.random.set_seed(SEED)\n print('seeding done!!!')\nseeding(CFG.seed)","metadata":{"papermill":{"duration":0.153451,"end_time":"2022-03-08T03:18:18.685056","exception":false,"start_time":"2022-03-08T03:18:18.531605","status":"completed"},"tags":[],"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-13T19:30:06.306112Z","iopub.execute_input":"2023-03-13T19:30:06.306494Z","iopub.status.idle":"2023-03-13T19:30:06.312518Z","shell.execute_reply.started":"2023-03-13T19:30:06.306454Z","shell.execute_reply":"2023-03-13T19:30:06.311743Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# WandB 🪄\n\"Weights\n\nTo track model's training I'll be using **Weights & Biases** tool. Weights & Biases (W&B) is MLOps platform for tracking our experiemnts. We can use it to Build better models faster with experiment tracking, dataset versioning, and model management. Specifically for this notebook, we can do error analysis to check in which audio files models are struggling as we can also log **audio** file in WandB.","metadata":{}},{"cell_type":"code","source":"# Import wandb library for logging and tracking experiments\nimport wandb\n\n# Try to get the API key from Kaggle secrets\ntry:\n from kaggle_secrets import UserSecretsClient\n user_secrets = UserSecretsClient()\n api_key = user_secrets.get_secret(\"WANDB\")\n # Login to wandb with the API key\n wandb.login(key=api_key)\n # Set anonymous mode to None\n anonymous = None\nexcept:\n # If Kaggle secrets are not available, set anonymous mode to 'must'\n anonymous = 'must'\n # Login to wandb anonymously and relogin if needed\n wandb.login(anonymous=anonymous, relogin=True)","metadata":{"execution":{"iopub.status.busy":"2023-03-13T19:30:06.313710Z","iopub.execute_input":"2023-03-13T19:30:06.314051Z","iopub.status.idle":"2023-03-13T19:30:07.773713Z","shell.execute_reply.started":"2023-03-13T19:30:06.314021Z","shell.execute_reply":"2023-03-13T19:30:07.772591Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Set Up Device 📱\nFollowing codes automatically detects hardware(tpu or tpu-vm or gpu). ","metadata":{"papermill":{"duration":0.065779,"end_time":"2022-03-08T03:18:18.817867","exception":false,"start_time":"2022-03-08T03:18:18.752088","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def get_device():\n \"Detect and intializes GPU/TPU automatically\"\n # Check TPU category\n tpu = 'local' if CFG.device=='TPU-VM' else None\n try:\n # Connect to TPU\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver.connect(tpu=tpu) \n # Set TPU strategy\n strategy = tf.distribute.TPUStrategy(tpu)\n print(f'> Running on {CFG.device} ', tpu.master(), end=' | ')\n print('Num of TPUs: ', strategy.num_replicas_in_sync)\n device=CFG.device\n except:\n # If TPU is not available, detect GPUs\n gpus = tf.config.list_logical_devices('GPU')\n ngpu = len(gpus)\n # Check number of GPUs\n if ngpu:\n # Set GPU strategy\n strategy = tf.distribute.MirroredStrategy(gpus) # single-GPU or multi-GPU\n # Print GPU details\n print(\"> Running on GPU\", end=' | ')\n print(\"Num of GPUs: \", ngpu)\n device='GPU'\n else:\n # If no GPUs are available, use CPU\n print(\"> Running on CPU\")\n strategy = tf.distribute.get_strategy()\n device='CPU'\n return strategy, device, tpu","metadata":{"_kg_hide-input":true,"papermill":{"duration":7.941725,"end_time":"2022-03-08T03:18:26.826553","exception":false,"start_time":"2022-03-08T03:18:18.884828","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-03-13T19:30:07.775417Z","iopub.execute_input":"2023-03-13T19:30:07.775863Z","iopub.status.idle":"2023-03-13T19:30:07.784863Z","shell.execute_reply.started":"2023-03-13T19:30:07.775830Z","shell.execute_reply":"2023-03-13T19:30:07.783921Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Initialize GPU/TPU/TPU-VM\nstrategy, CFG.device, tpu = get_device()\nCFG.replicas = strategy.num_replicas_in_sync","metadata":{"execution":{"iopub.status.busy":"2023-03-13T19:30:07.786114Z","iopub.execute_input":"2023-03-13T19:30:07.786372Z","iopub.status.idle":"2023-03-13T19:30:17.844860Z","shell.execute_reply.started":"2023-03-13T19:30:07.786348Z","shell.execute_reply":"2023-03-13T19:30:17.843369Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Dataset Path 📁\n`TPU v3-8` aka `Remote-TPU` requires GCS path for training. Thankfully kaggle provides us GCS path for each dataset. Following codes automatically uses GCS path if remote TPU is used as device.","metadata":{}},{"cell_type":"code","source":"BASE_PATH = '/kaggle/input/birdclef-2023'\n\nif CFG.device==\"TPU\":\n from kaggle_datasets import KaggleDatasets\n GCS_PATH = KaggleDatasets().get_gcs_path(BASE_PATH.split('/')[-1])\nelse:\n GCS_PATH = BASE_PATH","metadata":{"execution":{"iopub.status.busy":"2023-03-13T19:30:17.849661Z","iopub.execute_input":"2023-03-13T19:30:17.849962Z","iopub.status.idle":"2023-03-13T19:30:17.855767Z","shell.execute_reply.started":"2023-03-13T19:30:17.849935Z","shell.execute_reply":"2023-03-13T19:30:17.854587Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Meta Data 📖\n* **train_audio/** - folder contains training data of short recordings of individual bird calls uploaded by users of xenocanto.org. These files have been downsampled to $32,000$ Hz where applicable to match the test set audio and converted to the `.ogg` format. The training data should have nearly all relevant files; It can be expected there will be no benefit to looking for more on xenocanto.org.\n* **test_soundscapes/** - directory contains $~200$ recordings to be used for scoring when a notebook is submitted. Without submission only $1$ recording is accessible. All recordings are $10$ minutes long and in `.ogg` audio format.\n* **train_metadata.csv** - file contains a wide range of metadata the training data. The most directly relevant fields are:\n * `primary_label` - a code for the bird species. You can review detailed information about the bird codes by appending the code to https://ebird.org/species/, such as https://ebird.org/species/amecro for the American Crow.\n * `seconday_label` - if any other bird call exists on the recording. This feature is carefully used in top solution before.\n * `latitude & longitude`: coordinates for where the recording was taken. Some bird species may have local call 'dialects,' so you may want to seek geographic diversity in your training data.\n * `author` - The user who provided the recording.\n * `filename`: the name of the associated audio file.\n* **sample_submission.csv** - is the valid sample submission.\n * `row_id`: A slug of [soundscape_id]_[end_time] for the prediction.\n * `[bird_id]`: There are $264$ bird ID columns. The probability of the presence of each bird for each row needs to be predicted.\n* **eBird_Taxonomy_v2021.csv** - data on the relationships between different species.","metadata":{"papermill":{"duration":0.067107,"end_time":"2022-03-08T03:18:26.962626","exception":false,"start_time":"2022-03-08T03:18:26.895519","status":"completed"},"tags":[]}},{"cell_type":"code","source":"df = pd.read_csv(f'{BASE_PATH}/train_metadata.csv')\ndf['filepath'] = GCS_PATH + '/train_audio/' + df.filename\ndf['target'] = df.primary_label.map(CFG.name2label)\ndf.head(2)","metadata":{"papermill":{"duration":0.241649,"end_time":"2022-03-08T03:18:27.408813","exception":false,"start_time":"2022-03-08T03:18:27.167164","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-03-13T19:30:17.857240Z","iopub.execute_input":"2023-03-13T19:30:17.857651Z","iopub.status.idle":"2023-03-13T19:30:18.008902Z","shell.execute_reply.started":"2023-03-13T19:30:17.857611Z","shell.execute_reply":"2023-03-13T19:30:18.007632Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tf.io.gfile.exists(df.filepath.iloc[0])","metadata":{"papermill":{"duration":0.244976,"end_time":"2022-03-08T03:18:33.994955","exception":false,"start_time":"2022-03-08T03:18:33.749979","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-03-13T19:30:18.010384Z","iopub.execute_input":"2023-03-13T19:30:18.011146Z","iopub.status.idle":"2023-03-13T19:30:18.019311Z","shell.execute_reply.started":"2023-03-13T19:30:18.011101Z","shell.execute_reply":"2023-03-13T19:30:18.018202Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# EDA 🎨","metadata":{}},{"cell_type":"markdown","source":"## Utility","metadata":{}},{"cell_type":"code","source":"import cv2\n\ndef load_audio(filepath):\n audio, sr = librosa.load(filepath)\n return audio, sr\n\ndef show_image(filename):\n img = cv2.imread(f'/kaggle/input/birdclef23-image-dataset/{filename}.jpg')\n img = img[...,::-1] # bgr => rgb\n plt.figure(figsize=(6, 6))\n plt.imshow(img)\n plt.axis('OFF')\n plt.show()\n return img\n\ndef get_spectrogram(audio):\n spec = librosa.feature.melspectrogram(y=audio, sr=CFG.sample_rate, \n n_mels=CFG.img_size[0],\n n_fft=CFG.nfft,\n hop_length=CFG.hop_length,\n fmax=CFG.fmax,\n fmin=CFG.fmin,\n )\n spec = librosa.power_to_db(spec, ref=np.max)\n return spec\n\ndef display_audio(row):\n # Caption for viz\n caption = f'Id: {row.filename} | Name: {row.common_name} | Sci.Name: {row.scientific_name} | Rating: {row.rating}'\n # Read audio file\n audio, sr = load_audio(row.filepath)\n # Keep fixed length audio\n audio = audio[:CFG.audio_len]\n # Spectrogram from audio\n spec = get_spectrogram(audio)\n # Display audio\n print(\"# Audio:\")\n display(ipd.Audio(audio, rate=CFG.sample_rate))\n print(\"# Image:\")\n show_image(row.common_name)\n print('# Visualization:')\n fig, ax = plt.subplots(2, 1, figsize=(12, 2*3), sharex=True, tight_layout=True)\n fig.suptitle(caption)\n # Waveplot\n lid.waveshow(audio,\n sr=CFG.sample_rate,\n ax=ax[0])\n # Specplot\n lid.specshow(spec, \n sr = CFG.sample_rate, \n hop_length = CFG.hop_length,\n n_fft=CFG.nfft,\n fmin=CFG.fmin,\n fmax=CFG.fmax,\n x_axis = 'time', \n y_axis = 'mel',\n cmap = 'coolwarm',\n ax=ax[1])\n ax[0].set_xlabel('');\n fig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-13T19:30:18.021037Z","iopub.execute_input":"2023-03-13T19:30:18.021631Z","iopub.status.idle":"2023-03-13T19:30:18.827958Z","shell.execute_reply.started":"2023-03-13T19:30:18.021586Z","shell.execute_reply":"2023-03-13T19:30:18.827096Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Top 3 Categories","metadata":{}},{"cell_type":"code","source":"stat = df.primary_label.value_counts().index.tolist()\nclass_names = stat[:3] + stat[-3:] # popular + not popular","metadata":{"execution":{"iopub.status.busy":"2023-03-13T19:30:18.829232Z","iopub.execute_input":"2023-03-13T19:30:18.829720Z","iopub.status.idle":"2023-03-13T19:30:18.836878Z","shell.execute_reply.started":"2023-03-13T19:30:18.829681Z","shell.execute_reply":"2023-03-13T19:30:18.835814Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Popular Categories","metadata":{}},{"cell_type":"markdown","source":"### Barn Swallow","metadata":{}},{"cell_type":"code","source":"class_name = class_names[0]\nprint(f'# Category: {class_name}')\nclass_df = df.query(\"primary_label==@class_name\")\nprint(f'# Num Samples: {len(class_df)}')\nrow = class_df.sample(1).squeeze()\n\n# Display audio\ndisplay_audio(row)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-13T19:30:18.838215Z","iopub.execute_input":"2023-03-13T19:30:18.838665Z","iopub.status.idle":"2023-03-13T19:30:31.779208Z","shell.execute_reply.started":"2023-03-13T19:30:18.838628Z","shell.execute_reply":"2023-03-13T19:30:31.778030Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Willow Warbler","metadata":{}},{"cell_type":"code","source":"class_name = class_names[1]\nprint(f'# Category: {class_name}')\nclass_df = df.query(\"primary_label==@class_name\")\nprint(f'# Num Samples: {len(class_df)}')\nrow = class_df.sample(1).squeeze()\n\n# Display audio\ndisplay_audio(row)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-13T19:30:31.781255Z","iopub.execute_input":"2023-03-13T19:30:31.782028Z","iopub.status.idle":"2023-03-13T19:30:33.385777Z","shell.execute_reply.started":"2023-03-13T19:30:31.781993Z","shell.execute_reply":"2023-03-13T19:30:33.384250Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Thursh Nightingale","metadata":{}},{"cell_type":"code","source":"class_name = class_names[2]\nprint(f'# Category: {class_name}')\nclass_df = df.query(\"primary_label==@class_name\")\nprint(f'# Num Samples: {len(class_df)}')\nrow = class_df.sample(1).squeeze()\n\n# Display audio\ndisplay_audio(row)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-13T19:30:33.387812Z","iopub.execute_input":"2023-03-13T19:30:33.388160Z","iopub.status.idle":"2023-03-13T19:30:34.903756Z","shell.execute_reply.started":"2023-03-13T19:30:33.388128Z","shell.execute_reply":"2023-03-13T19:30:34.902603Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## UpPopular Categories","metadata":{}},{"cell_type":"markdown","source":"### White-headed Sawwing","metadata":{}},{"cell_type":"code","source":"class_name = class_names[-3]\nprint(f'# Category: {class_name}')\nclass_df = df.query(\"primary_label==@class_name\")\nprint(f'# Num Samples: {len(class_df)}')\nrow = class_df.sample(1).squeeze()\n\n# Display audio\ndisplay_audio(row)","metadata":{"execution":{"iopub.status.busy":"2023-03-13T19:30:34.905092Z","iopub.execute_input":"2023-03-13T19:30:34.905415Z","iopub.status.idle":"2023-03-13T19:30:36.691915Z","shell.execute_reply.started":"2023-03-13T19:30:34.905387Z","shell.execute_reply":"2023-03-13T19:30:36.690903Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### African Pygmy Kingfisher","metadata":{}},{"cell_type":"code","source":"class_name = class_names[-2]\nprint(f'# Category: {class_name}')\nclass_df = df.query(\"primary_label==@class_name\")\nprint(f'# Num Samples: {len(class_df)}')\nrow = class_df.sample(1).squeeze()\n\n# Display audio\ndisplay_audio(row)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-13T19:30:36.693169Z","iopub.execute_input":"2023-03-13T19:30:36.693481Z","iopub.status.idle":"2023-03-13T19:30:38.285837Z","shell.execute_reply.started":"2023-03-13T19:30:36.693440Z","shell.execute_reply":"2023-03-13T19:30:38.284769Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Crested Francolin\n","metadata":{}},{"cell_type":"code","source":"class_name = class_names[-1]\nprint(f'# Category: {class_name}')\nclass_df = df.query(\"primary_label==@class_name\")\nprint(f'# Num Samples: {len(class_df)}')\nrow = class_df.sample(1).squeeze()\n\n# Display audio\ndisplay_audio(row)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-13T19:30:38.287235Z","iopub.execute_input":"2023-03-13T19:30:38.287558Z","iopub.status.idle":"2023-03-13T19:30:40.138002Z","shell.execute_reply.started":"2023-03-13T19:30:38.287528Z","shell.execute_reply":"2023-03-13T19:30:40.136803Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Split 🔪\nFollowing code will split the data into folds using target stratification.\n> **Note:** Some classess have too few samples thus not each fold contains all the classes. ","metadata":{"papermill":{"duration":0.09524,"end_time":"2022-03-08T03:18:34.861029","exception":false,"start_time":"2022-03-08T03:18:34.765789","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Import required packages\nfrom sklearn.model_selection import StratifiedKFold\n\n# Initialize the StratifiedKFold object with 5 splits and shuffle the data\nskf = StratifiedKFold(n_splits=5, shuffle=True, random_state=CFG.seed)\n\n# Reset the index of the dataframe\ndf = df.reset_index(drop=True)\n\n# Create a new column in the dataframe to store the fold number for each row\ndf[\"fold\"] = -1\n\n# Iterate over the folds and assign the corresponding fold number to each row in the dataframe\nfor fold, (train_idx, val_idx) in enumerate(skf.split(df, df['primary_label'])):\n df.loc[val_idx, 'fold'] = fold","metadata":{"papermill":{"duration":0.386301,"end_time":"2022-03-08T03:18:35.325064","exception":false,"start_time":"2022-03-08T03:18:34.938763","status":"completed"},"tags":[],"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-13T19:30:40.139554Z","iopub.execute_input":"2023-03-13T19:30:40.139857Z","iopub.status.idle":"2023-03-13T19:30:40.234051Z","shell.execute_reply.started":"2023-03-13T19:30:40.139831Z","shell.execute_reply":"2023-03-13T19:30:40.232873Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.groupby(['fold','primary_label']).size()","metadata":{"papermill":{"duration":0.173215,"end_time":"2022-03-08T03:18:36.347671","exception":false,"start_time":"2022-03-08T03:18:36.174456","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-03-13T19:30:40.235328Z","iopub.execute_input":"2023-03-13T19:30:40.235720Z","iopub.status.idle":"2023-03-13T19:30:40.249363Z","shell.execute_reply.started":"2023-03-13T19:30:40.235693Z","shell.execute_reply":"2023-03-13T19:30:40.248321Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Filter & Upsample Data ⬆️\n\n* **Filter**: As there is even only one sample for some classes we need to make sure they are in the train data using filtering. We can do this by always keeping them in the train data and do cross-validtion on the rest of the data.\n\n* **Upsample**: Even in the filtered data there are some minority classes with very few samples. To amend the class imbalance we can try upsampling those classes. Following function will simply upsample the train data for minory class which has very few samples. This can potentially mitigate the classic \"Long Tail\" problem.\n","metadata":{}},{"cell_type":"code","source":"def filter_data(df, thr=5):\n # Count the number of samples for each class\n counts = df.primary_label.value_counts()\n\n # Condition that selects classes with less than `thr` samples\n cond = df.primary_label.isin(counts[counts **Caution:** Even though we are training audio as an image (spectrogram), we can't use typical computer-vision augmentations such as HorizontalFlip, Rotation, Shear, etc as it may contaminate the information contained within the image (spectrogram). For example, if we rotate a spectrogram then it doesn't make sense anymore as we can't relate this rotated spectrogram to the actual audio. ","metadata":{"papermill":{"duration":0.151237,"end_time":"2022-03-08T03:18:47.959873","exception":false,"start_time":"2022-03-08T03:18:47.808636","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## Used Augmentations\nTwo types of Augmentations are used here, \n1. **AudioAug**: Augmentations that will be applied directly to **audio** data. Example: Gaussian Noise, Random CropPad.\n2. **SpecAug**: Augmentations that will be applied to **spectrogram** data. Example: TimeFreqMask, CutMixUp.\n\nHere are some example images before and after these augmentations,\n## AudioAug:\n\n* Random Noise\n\"aug\"\n\n* Random TimeShift\n\"aug-timeshift\"\n\n* Random CropOrPad\n\"aug-crop-or-pad\"\n\n## SpecAug:\n\n* Random TimeMask\n\"spec-time-mask\"\n\n* Random FreqMask\n\"spec-freq-mask\"\n\n* CutMix\n\"spec-cutmix\"\n\n* MixUp\n\"spec-mixup\"","metadata":{}},{"cell_type":"markdown","source":"## Utility","metadata":{}},{"cell_type":"code","source":"# Generates random integer\ndef random_int(shape=[], minval=0, maxval=1):\n return tf.random.uniform(shape=shape, minval=minval, maxval=maxval, dtype=tf.int32)\n\n\n# Generats random float\ndef random_float(shape=[], minval=0.0, maxval=1.0):\n rnd = tf.random.uniform(shape=shape, minval=minval, maxval=maxval, dtype=tf.float32)\n return rnd","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-13T19:30:43.086908Z","iopub.execute_input":"2023-03-13T19:30:43.087220Z","iopub.status.idle":"2023-03-13T19:30:43.093412Z","shell.execute_reply.started":"2023-03-13T19:30:43.087191Z","shell.execute_reply":"2023-03-13T19:30:43.092571Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## AudioAug\nApplies augmentation directly to audio","metadata":{}},{"cell_type":"code","source":"# Import required packages\nimport tensorflow as tf\n\n# Define a function to crop or pad audio data to a target length\n@tf.function\ndef CropOrPad(audio, target_len, pad_mode='constant'):\n # Get the length of the input audio\n audio_len = tf.shape(audio)[0]\n # If the length of the input audio is smaller than the target length, randomly pad the audio\n if audio_len < target_len:\n # Calculate the offset between the input audio and the target length\n diff_len = (target_len - audio_len)\n # Select a random location for padding\n pad1 = random_int([], minval=0, maxval=diff_len)\n # Calculate the second padding value\n pad2 = diff_len - pad1\n pad_len = [pad1, pad2]\n # Apply padding to the audio data\n audio = tf.pad(audio, paddings=[pad_len], mode=pad_mode)\n # If the length of the input audio is larger than the target length, crop the audio\n elif audio_len > target_len:\n # Calculate the difference in length between the input audio and the target length\n diff_len = (audio_len - target_len)\n # Select a random location for cropping\n idx = tf.random.uniform([], 0, diff_len, dtype=tf.int32)\n # Crop the audio data\n audio = audio[idx: (idx + target_len)]\n # Reshape the audio data to the target length\n audio = tf.reshape(audio, [target_len])\n # Return the cropped or padded audio data\n return audio\n\n\n# Randomly shift audio -> any sound at time may get shifted to time\n@tf.function\ndef TimeShift(audio, prob=0.5):\n # Randomly apply time shift with probability `prob`\n if random_float() < prob:\n # Calculate random shift value\n shift = random_int(shape=[], minval=0, maxval=tf.shape(audio)[0])\n # Randomly set the shift to be negative with 50% probability\n if random_float() < 0.5:\n shift = -shift\n # Roll the audio signal by the shift value\n audio = tf.roll(audio, shift, axis=0)\n return audio\n\n# Apply random noise to audio data\n@tf.function\ndef GaussianNoise(audio, std=[0.0025, 0.025], prob=0.5):\n # Select a random value of standard deviation for Gaussian noise within the given range\n std = random_float([], std[0], std[1])\n # Randomly apply Gaussian noise with probability `prob`\n if random_float() < prob:\n # Add random Gaussian noise to the audio signal\n GN = tf.keras.layers.GaussianNoise(stddev=std)\n audio = GN(audio, training=True) # training=False don't apply noise to data\n return audio\n\n# Applies augmentation to Audio Signal\ndef AudioAug(audio):\n # Apply time shift and Gaussian noise to the audio signal\n audio = TimeShift(audio, prob=CFG.timeshift_prob)\n audio = GaussianNoise(audio, prob=CFG.gn_prob)\n return audio\n\n# Standardize the audio\n@tf.function\ndef Normalize(data, min_max=True):\n # Compute the mean and standard deviation of the data\n MEAN = tf.math.reduce_mean(data)\n STD = tf.math.reduce_std(data)\n # Standardize the data\n data = tf.math.divide_no_nan(data - MEAN, STD)\n # Normalize to [0, 1]\n if min_max:\n MIN = tf.math.reduce_min(data)\n MAX = tf.math.reduce_max(data)\n data = tf.math.divide_no_nan(data - MIN, MAX - MIN)\n return data","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-13T19:30:43.094607Z","iopub.execute_input":"2023-03-13T19:30:43.094876Z","iopub.status.idle":"2023-03-13T19:30:43.113110Z","shell.execute_reply.started":"2023-03-13T19:30:43.094852Z","shell.execute_reply":"2023-03-13T19:30:43.112222Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## SpecAug\nApplies augmentation to spectrogram instead of audio","metadata":{}},{"cell_type":"code","source":"# Convert spectrogram (H,W) to image (H,W,3)\n@tf.function\ndef Spec2Img(spec, num_channels=3):\n # If the original image has 1 channel, convert it to a 3 channel image by repeating the same image across channel axis\n if num_channels > 1:\n img = tf.tile(spec[..., tf.newaxis], [1, 1, num_channels])\n else:\n img = spec[..., tf.newaxis]\n return img\n\n# Convert img (H,W,3) to image (H,W)\n@tf.function\ndef Img2Spec(img):\n # Extract the first channel of the image\n return img[..., 0]\n\n\n# Randomly mask data in time and freq axis\n@tf.function\ndef TimeFreqMask(spec, time_mask, freq_mask, prob=0.5):\n if random_float() < prob:\n # Apply frequency masking to the spectrogram\n spec = tfio.audio.freq_mask(spec, param=freq_mask)\n # Apply time masking to the spectrogram\n spec = tfio.audio.time_mask(spec, param=time_mask)\n return spec\n\n\n# Applies augmentation to Spectrogram\ndef SpecAug(spec):\n # Convert the spectrogram to a 2D matrix and transpose it to get the shape [time, mel]\n spec = tf.transpose(Img2Spec(spec), perm=[1, 0])\n # Apply time and frequency masking to the spectrogram\n spec = TimeFreqMask(spec, time_mask=CFG.time_mask, freq_mask=CFG.freq_mask, prob=0.5)\n # Transpose the spectrogram back to the original shape [mel, time] and convert it to an image\n spec = tf.transpose(spec, perm=[1, 0])\n spec = Spec2Img(spec)\n return spec\n\n# Compute MixUp Augmentation for Spectrogram\ndef MixUp(alpha=0.2, prob=0.5):\n \"\"\"Apply Spectrogram-MixUp augmentaiton. Apply Mixup to one batch and its shifted version\"\"\"\n @tf.function\n def apply(specs, labels, alpha=alpha, prob=prob):\n if random_float() > prob:\n return specs, labels\n\n spec_shape = tf.shape(specs)\n label_shape = tf.shape(labels)\n\n # Select lambda from beta distribution\n beta = tfp.distributions.Beta(alpha, alpha)\n lam = beta.sample(1)[0]\n \n # It's faster to roll the batch by one instead of shuffling it to create image pairs\n specs = lam * specs + (1 - lam) * tf.roll(specs, shift=1, axis=0) # mixup = [1, 2, 3]*lam + [3, 1, 2]*(1 - lam)\n labels = lam * labels + (1 - lam) * tf.roll(labels, shift=1, axis=0)\n\n specs = tf.reshape(specs, spec_shape)\n labels = tf.reshape(labels, label_shape)\n return specs, labels\n return apply\n\n\ndef CutMix(alpha, prob=0.5):\n \"\"\"Apply Spectrogram-CutMix augmentaiton which only cuts patch across time axis unlike \n typical Computer-Vision CutMix. Applies CutMix to one batch and its shifted version.\n \"\"\"\n @tf.function\n def apply(specs, labels, alpha=alpha, prob=prob):\n if random_float() > prob:\n return specs, labels\n spec_shape = tf.shape(specs)\n label_shape = tf.shape(labels)\n W = tf.cast(spec_shape[1], tf.int32) # [batch, time, freq, channel]\n\n # Lambda from beta distribution\n beta = tfp.distributions.Beta(alpha, alpha)\n lam = beta.sample(1)[0]\n \n # It's faster to roll the batch by one instead of shuffling it to create image pairs\n specs_rolled = tf.roll(specs, shift=1, axis=0) # specs->[1, 2, 3], specs_rolled->[3, 1, 2]\n labels_rolled = tf.roll(labels, shift=1, axis=0)\n\n # Select random patch size\n r_x = random_int([], minval=0, maxval=W)\n r = 0.5 * tf.math.sqrt(1.0 - lam)\n r_w_half = tf.cast(r * tf.cast(W, tf.float32), tf.int32)\n\n # Select random location in time axis\n x1 = tf.cast(tf.clip_by_value(r_x - r_w_half, 0, W), tf.int32)\n x2 = tf.cast(tf.clip_by_value(r_x + r_w_half, 0, W), tf.int32)\n\n # outer-pad patch -> [0, 0, x, x, 0, 0]\n patch1 = specs[:, x1:x2, :, :] # [batch, time, freq, channel]\n patch1 = tf.pad(\n patch1, [[0, 0], [x1, W - x2], [0, 0], [0, 0]]) # outer-pad\n\n # inner-pad-patch -> [y, y, 0, 0, y, y]\n patch2 = specs_rolled[:, x1:x2, :, :] # [batch, mel, time, channel]\n patch2 = tf.pad(\n patch2, [[0, 0], [x1, W - x2], [0, 0], [0, 0]]) # outer-pad\n patch2 = specs_rolled - patch2 # inner-pad-patch = img - outer-pad-patch\n \n # patch1 -> [0, 0, x, x, 0, 0], patch2 -> [y, y, 0, 0, y, y]\n # cutmix = (patch1 + patch2) -> [y, y, x, x, y, y]\n specs = patch1 + patch2 # cutmix img\n\n # Compute lambda = [1 - (patch_area/image_area)]\n lam = tf.cast((1.0 - (x2 - x1) / (W)),tf.float32) # no H term as (y1 - y2) = H\n labels = lam * labels + (1.0 - lam) * labels_rolled # cutmix label\n\n specs = tf.reshape(specs, spec_shape)\n labels = tf.reshape(labels, label_shape)\n\n return specs, labels\n return apply","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-13T19:30:43.118188Z","iopub.execute_input":"2023-03-13T19:30:43.118511Z","iopub.status.idle":"2023-03-13T19:30:43.144639Z","shell.execute_reply.started":"2023-03-13T19:30:43.118486Z","shell.execute_reply":"2023-03-13T19:30:43.143751Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Feature Extraction 💡\n* We'll feed **Mel-Spectrogram** feature to our model as input. We can simply consider **Mel-Spectrogram** as **2D image** which `x-axis `represents `time` and `y-axis` represents `frequency` information. Hence, we can consider this problem as an **Image Classification** Problem.\n* First, **Spectrogram** feature is extracted from audio sample using **Short Time Fourier Transform (STFT)** which is a variant of **Fast Fourier Transform (FFT)**. \n* The fast Fourier transform is a powerful tool that allows us to analyze the frequency content of a signal.\n* For non-periodic signals such as Music and Speech, compute several spectrums by performing FFT on several windowed segments of the signal hence it is called **Short-time Fourier transform**. It is computed on overlapping windowed segments of the signal, and we get what is called the spectrogram.\n* STFT suffers from famous Time-Frequency localization trade-off, which is\n * `Narrow-Window` results in `Good time resolution` but `Poor frequency resolution`\n * `Wide-Window` results in `Poor time resolution` but `Good frequency resolution`\n* Second, **Mel-Spectrogram** is computed which is a Spectrogram in **Mel-scale (logarithmic transformation of a signal's frequency)**. Mel spectrograms are better suited for applications that need to model human hearing perception such as **Speech and Music**.\n\n
","metadata":{}},{"cell_type":"code","source":"# Compute Spectrogram from audio\n@tf.function\ndef Audio2Spec(audio, spec_shape=[128, 384], sr=32000, nfft=2048, window=2048, fmin=500, fmax=14000):\n \"\"\"\n Computes a Mel-scaled spectrogram from audio using TensorFlow and TensorFlow-IO.\n \"\"\"\n # Get the desired height and width of the spectrogram\n spec_height = spec_shape[0]\n spec_width = spec_shape[1]\n \n # Get the length of the audio and calculate the hop length for the STFT\n audio_len = tf.shape(audio)[0]\n hop_length = tf.cast((audio_len // (spec_width - 1)), tf.int32) # sample rate * duration / spec width - 1 == 627\n \n # Compute the spectrogram and the Mel-scaled spectrogram using TensorFlow-IO\n spec = tfio.audio.spectrogram(audio, nfft=nfft, window=window, stride=hop_length)\n mel_spec = tfio.audio.melscale(spec, rate=sr, mels=spec_height, fmin=fmin, fmax=fmax)\n \n # Convert the Mel-scaled spectrogram to decibels and transpose it to keep it (mel, time)\n db_mel_spec = tfio.audio.dbscale(mel_spec, top_db=80)\n db_mel_spec = tf.transpose(db_mel_spec, perm=[1, 0])\n \n # If the spectrogram is larger than the desired shape, crop it\n if tf.shape(db_mel_spec)[1] > spec_width:\n db_mel_spec = db_mel_spec[:, :spec_width]\n \n # Reshape the spectrogram to the desired shape and return it\n db_mel_spec = tf.reshape(db_mel_spec, spec_shape)\n return db_mel_spec\n","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2023-03-13T19:30:43.145638Z","iopub.execute_input":"2023-03-13T19:30:43.145897Z","iopub.status.idle":"2023-03-13T19:30:43.160931Z","shell.execute_reply.started":"2023-03-13T19:30:43.145873Z","shell.execute_reply":"2023-03-13T19:30:43.159992Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Loader 🍚","metadata":{"papermill":{"duration":0.152812,"end_time":"2022-03-08T03:18:48.676686","exception":false,"start_time":"2022-03-08T03:18:48.523874","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## Decoders\nFollowing code will decode the raw audio from `.ogg` file and also decode spectrogram from `audio` file.","metadata":{}},{"cell_type":"code","source":"# Decodes Audio\ndef audio_decoder(with_labels=True, dim=CFG.audio_len, CFG=CFG):\n def get_audio(filepath):\n file_bytes = tf.io.read_file(filepath)\n audio = tfio.audio.decode_vorbis(file_bytes) # decode .ogg file for .wave replace `decode_wav`\n audio = tf.cast(audio, tf.float32)\n audio = tf.squeeze(audio, axis=-1)\n if CFG.normalize:\n audio = Normalize(audio)\n return audio\n\n def get_target(target): \n target = tf.reshape(target, [1])\n target = tf.cast(tf.one_hot(target, len(CFG.class_labels)), tf.float32) \n target = tf.reshape(target, [len(CFG.class_labels)])\n return target\n\n def decode(path):\n audio = get_audio(path)\n audio = CropOrPad(audio, dim) # crop or pad audio to keep a fixed length\n audio = tf.reshape(audio, [dim])\n return audio\n \n def decode_with_labels(path, label):\n label = get_target(label)\n return decode(path), label\n \n return decode_with_labels if with_labels else decode\n\n# Decodes Spectrogram\ndef spec_decoder(with_labels=True, dim=CFG.img_size, CFG=CFG):\n def decode(audio):\n # Compute Spectrogram\n spec = Audio2Spec(audio, spec_shape=dim, sr=CFG.sample_rate, \n nfft=CFG.nfft, window=CFG.window, fmin=CFG.fmin,fmax=CFG.fmax)\n # Spectrogram (H, W) to Image (H, W, C)\n spec = Spec2Img(spec, num_channels=3) \n spec = tf.reshape(spec, [*dim, 3])\n return spec\n \n def decode_with_labels(path, label):\n return decode(path), label\n \n return decode_with_labels if with_labels else decode","metadata":{"papermill":{"duration":0.251237,"end_time":"2022-03-08T03:18:49.079346","exception":false,"start_time":"2022-03-08T03:18:48.828109","status":"completed"},"tags":[],"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-13T19:30:43.162048Z","iopub.execute_input":"2023-03-13T19:30:43.162353Z","iopub.status.idle":"2023-03-13T19:30:43.178828Z","shell.execute_reply.started":"2023-03-13T19:30:43.162328Z","shell.execute_reply":"2023-03-13T19:30:43.177917Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Augmenters\nFollowing code will apply augmentations to audio and spectrogram.","metadata":{"papermill":{"duration":0.150182,"end_time":"2022-03-08T03:18:49.38214","exception":false,"start_time":"2022-03-08T03:18:49.231958","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Applies augmentation to audio\ndef audio_augmenter(with_labels=True, dim=CFG.audio_len, CFG=CFG):\n def augment(audio, dim=dim):\n if random_float() <= CFG.audio_augment_prob:\n audio = AudioAug(audio)\n audio = tf.reshape(audio, [dim])\n return audio\n \n def augment_with_labels(audio, label): \n return augment(audio), label\n \n return augment_with_labels if with_labels else augment\n\n# Applies augentation to spectrogram\ndef spec_augmenter(with_labels=True, dim=CFG.img_size, CFG=CFG):\n def augment(spec, dim=dim): \n if random_float() <= CFG.spec_augment_prob:\n spec = SpecAug(spec)\n spec = tf.reshape(spec, [*dim, 3])\n return spec\n \n def augment_with_labels(spec, label): \n return augment(spec), label\n \n return augment_with_labels if with_labels else augment","metadata":{"papermill":{"duration":0.250484,"end_time":"2022-03-08T03:18:49.79513","exception":false,"start_time":"2022-03-08T03:18:49.544646","status":"completed"},"tags":[],"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-13T19:30:43.179865Z","iopub.execute_input":"2023-03-13T19:30:43.180125Z","iopub.status.idle":"2023-03-13T19:30:43.194612Z","shell.execute_reply.started":"2023-03-13T19:30:43.180101Z","shell.execute_reply":"2023-03-13T19:30:43.193765Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data Pipeline\nFollowing code builds the complete pipeline of the data flow. It uses `tf.data.Dataset` for data processing. Here are some cool features of `tf.data`,\n* We can build complex input pipelines from simple, reusable pieces using`tf.data` API . For example, the pipeline for an audio model might aggregate data from files in a distributed file system, apply random transformation/augmentation to each audio, and merge randomly selected audios into a batch for training.\n* Moreover `tf.data` API provides a `tf.data.Dataset` feature that represents a sequence of components where each component comprises one or more pieces. For instance, in an audio pipeline, a component might be a single training example, with a pair of tensor pieces representing the audio and its label.\n\nCheck out this [doc](https://www.tensorflow.org/guide/data) if you want to learn more about `tf.data`.","metadata":{"papermill":{"duration":0.152217,"end_time":"2022-03-08T03:18:50.097623","exception":false,"start_time":"2022-03-08T03:18:49.945406","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def build_dataset(paths, labels=None, batch_size=32, target_size=[128, 256], \n audio_decode_fn=None, audio_augment_fn=None, \n spec_decode_fn=None, spec_augment_fn=None,\n cache=True, cache_dir=\"\",drop_remainder=False,\n augment=True, repeat=True, shuffle=1024):\n \"\"\"\n Creates a TensorFlow dataset from the given paths and labels.\n \n Args:\n paths (list): A list of file paths to the audio files.\n labels (list): A list of corresponding labels for the audio files.\n batch_size (int): Batch size for the created dataset.\n target_size (list): A list of target image size for the spectrograms.\n audio_decode_fn (function): A function to decode the audio file.\n audio_augment_fn (function): A function to augment the audio file.\n spec_decode_fn (function): A function to decode the spectrogram.\n spec_augment_fn (function): A function to augment the spectrogram.\n cache (bool): Whether to cache the dataset or not.\n cache_dir (str): Directory path to cache the dataset.\n drop_remainder (bool): Whether to drop the last batch if it is smaller than batch_size.\n augment (bool): Whether to augment the dataset or not.\n repeat (bool): Whether to repeat the dataset or not.\n shuffle (int): Number of elements from the dataset to buffer for shuffling.\n \n Returns:\n ds (tf.data.Dataset): A TensorFlow dataset.\n \"\"\"\n # Create cache directory if cache is enabled\n if cache_dir != \"\" and cache is True:\n os.makedirs(cache_dir, exist_ok=True)\n # Set default audio decode function if not provided\n if audio_decode_fn is None:\n audio_decode_fn = audio_decoder(\n labels is not None, dim=CFG.audio_len, CFG=CFG)\n # Set default audio augmentation function if not provided\n if audio_augment_fn is None:\n audio_augment_fn = audio_augmenter(\n labels is not None, dim=CFG.audio_len, CFG=CFG)\n # Set default spectrogram decode function if not provided\n if spec_decode_fn is None:\n spec_decode_fn = spec_decoder(\n labels is not None, dim=CFG.img_size, CFG=CFG)\n # Set default spectrogram augmentation function if not provided\n if spec_augment_fn is None:\n spec_augment_fn = spec_augmenter(\n labels is not None, dim=CFG.img_size, CFG=CFG)\n # Set TensorFlow AUTOTUNE option\n AUTO = tf.data.experimental.AUTOTUNE\n \n # Create slices based on whether labels are provided\n slices = paths if labels is None else (paths, labels)\n # Create TensorFlow dataset from slices\n ds = tf.data.Dataset.from_tensor_slices(slices)\n # Map audio decode function to dataset\n ds = ds.map(audio_decode_fn, num_parallel_calls=AUTO)\n # Cache dataset in memory if cache is enabled\n ds = ds.cache(cache_dir) if cache else ds\n # Repeat dataset indefinitely if repeat is enabled\n ds = ds.repeat() if repeat else ds\n # Create TensorFlow dataset options\n opt = tf.data.Options()\n # Shuffle dataset if shuffle is enabled\n if shuffle: \n ds = ds.shuffle(shuffle, seed=CFG.seed)\n opt.experimental_deterministic = False\n if CFG.device=='GPU':\n # If the device is a GPU, turn off auto-sharding to avoid performance issues\n opt.experimental_distribute.auto_shard_policy = tf.data.experimental.AutoShardPolicy.OFF\n # Set the options for the dataset\n ds = ds.with_options(opt)\n # Apply audio augmentation to the dataset if augment is True\n ds = ds.map(audio_augment_fn, num_parallel_calls=AUTO) if augment else ds\n # Decode the spectrograms for the dataset\n ds = ds.map(spec_decode_fn, num_parallel_calls=AUTO)\n # Apply spectrogram augmentation to the dataset if augment is True\n ds = ds.map(spec_augment_fn, num_parallel_calls=AUTO) if augment else ds\n # Batch the dataset with the specified batch size\n ds = ds.batch(batch_size, drop_remainder=drop_remainder)\n # MixUp: apply MixUp regularization to the dataset if specified\n if CFG.mixup_prob and augment and labels is not None:\n ds = ds.map(MixUp(alpha=CFG.mixup_alpha,prob=CFG.mixup_prob),num_parallel_calls=AUTO)\n # CutMix: apply CutMix regularization to the dataset if specified\n if CFG.cutmix_prob and augment and labels is not None:\n ds = ds.map(CutMix(alpha=CFG.cutmix_alpha,prob=CFG.cutmix_prob),num_parallel_calls=AUTO)\n # Prefetch the next batch of data to improve performance\n ds = ds.prefetch(AUTO)\n return ds","metadata":{"papermill":{"duration":0.240881,"end_time":"2022-03-08T03:18:50.489717","exception":false,"start_time":"2022-03-08T03:18:50.248836","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-03-13T19:30:43.195801Z","iopub.execute_input":"2023-03-13T19:30:43.196347Z","iopub.status.idle":"2023-03-13T19:30:43.215095Z","shell.execute_reply.started":"2023-03-13T19:30:43.196298Z","shell.execute_reply":"2023-03-13T19:30:43.214298Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Visualization 🔭\nTo ensure our pipeline is generating **spectrogram** and its associate **label** correctly, we'll check some samples from a batch.","metadata":{}},{"cell_type":"code","source":"def plot_batch(batch, row=3, col=3):\n \"\"\"Plot one batch data\"\"\"\n if isinstance(batch, tuple) or isinstance(batch, list):\n imgs, tars = batch\n else:\n imgs = batch\n tars = None\n plt.figure(figsize=(col*5, row*3))\n for idx in range(row*col):\n ax = plt.subplot(row, col, idx+1)\n lid.specshow(imgs[idx][...,0].numpy(), \n sr = CFG.sample_rate, \n hop_length = CFG.hop_length,\n fmin=CFG.fmin,\n fmax=CFG.fmax,\n x_axis = 'time', \n y_axis = 'mel',\n cmap = 'coolwarm')\n if tars is not None:\n label = tars[idx].numpy().argmax()\n name = CFG.label2name[label]\n plt.title(name)\n plt.tight_layout()\n plt.show()\n \n \ndef plot_history(history):\n \"\"\"Plot trainign history, credit: @cdeotte\"\"\"\n epochs = len(history.history['auc'])\n plt.figure(figsize=(15,5))\n plt.plot(np.arange(epochs),history.history['auc'],'-o',label='Train AUC',color='#ff7f0e')\n plt.plot(np.arange(epochs),history.history['val_auc'],'-o',label='Val AUC',color='#1f77b4')\n x = np.argmax( history.history['val_auc'] ); y = np.max( history.history['val_auc'] )\n xdist = plt.xlim()[1] - plt.xlim()[0]; ydist = plt.ylim()[1] - plt.ylim()[0]\n plt.scatter(x,y,s=200,color='#1f77b4'); plt.text(x-0.03*xdist,y-0.13*ydist,'max auc\\n%.2f'%y,size=14)\n plt.ylabel('AUC (PR)',size=14); plt.xlabel('Epoch',size=14)\n plt.legend(loc=2)\n plt2 = plt.gca().twinx()\n plt2.plot(np.arange(epochs),history.history['loss'],'-o',label='Train Loss',color='#2ca02c')\n plt2.plot(np.arange(epochs),history.history['val_loss'],'-o',label='Val Loss',color='#d62728')\n x = np.argmin( history.history['val_loss'] ); y = np.min( history.history['val_loss'] )\n ydist = plt.ylim()[1] - plt.ylim()[0]\n plt.scatter(x,y,s=200,color='#d62728'); plt.text(x-0.03*xdist,y+0.05*ydist,'min loss',size=14)\n plt.ylabel('Loss',size=14)\n plt.title('Fold %i - Training Plot'%(fold+1),size=18)\n plt.legend(loc=3)\n plt.show() ","metadata":{"papermill":{"duration":0.328513,"end_time":"2022-03-08T03:19:59.512224","exception":false,"start_time":"2022-03-08T03:19:59.183711","status":"completed"},"tags":[],"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-13T19:30:43.216218Z","iopub.execute_input":"2023-03-13T19:30:43.216542Z","iopub.status.idle":"2023-03-13T19:30:43.235835Z","shell.execute_reply.started":"2023-03-13T19:30:43.216514Z","shell.execute_reply":"2023-03-13T19:30:43.235016Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## W/O Augmentation","metadata":{}},{"cell_type":"code","source":"ds = build_dataset(df.filepath.tolist(), df.target.tolist(), augment=False, cache=False)\nds = ds.take(100)\nimgs, labels = next(iter(ds))\nplot_batch((imgs, labels))","metadata":{"papermill":{"duration":3.299334,"end_time":"2022-03-08T03:20:02.987986","exception":false,"start_time":"2022-03-08T03:19:59.688652","status":"completed"},"tags":[],"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-13T19:30:43.236930Z","iopub.execute_input":"2023-03-13T19:30:43.237224Z","iopub.status.idle":"2023-03-13T19:30:51.879117Z","shell.execute_reply.started":"2023-03-13T19:30:43.237197Z","shell.execute_reply":"2023-03-13T19:30:51.877966Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## W/ Augmentation","metadata":{}},{"cell_type":"code","source":"ds = build_dataset(df.filepath.tolist(), df.target.tolist(), augment=True, cache=False)\nds = ds.take(100)\nimgs, labels = next(iter(ds))\nplot_batch((imgs, labels))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-13T19:30:51.880411Z","iopub.execute_input":"2023-03-13T19:30:51.880711Z","iopub.status.idle":"2023-03-13T19:31:00.284711Z","shell.execute_reply.started":"2023-03-13T19:30:51.880685Z","shell.execute_reply":"2023-03-13T19:31:00.283656Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Loss, Metric & Optmizer 📉\nThis notebook will use the `Categorical Cross Entropy (CCE)` loss for optimization, accompanied by `AUC (PR Curve)` and `Accuracy` as performance metrics. The model's efficacy will be evaluated using the padded `cmAP` metric, which is a variant of the `macro-averaged average precision` score that is implemented in scikit-learn. You can learn more about `cmAP` metric from [here](https://www.kaggle.com/competitions/birdclef-2023/overview/evaluation). Finally, it will use `Adam` optmizer for training the model.\n\n$$\nCCE = -\\frac{1}{N} \\sum_{i=1}^{N} \\sum_{j=1}^{C} y_{i,j} \\log(p_{i,j})\n$$\n\n> **Note:** To account for cases where there are zero true positive labels for certain species, and to mitigate the impact of species with very few positive labels, this metric adds five rows of true positives to both the ground_truth and the prediction prior to scoring. This padding technique leads to even the baseline submission having a relatively strong score.\n\n> **Note**: This notebook will use `Precision Recall (PR)` curve for AUC metric instead of typical `ROC` curve.","metadata":{"papermill":{"duration":0.184301,"end_time":"2022-03-08T03:20:04.031695","exception":false,"start_time":"2022-03-08T03:20:03.847394","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import sklearn.metrics\n\ndef get_metrics():\n# acc = tf.keras.metrics.BinaryAccuracy(name='acc')\n auc = tf.keras.metrics.AUC(curve='PR', name='auc', multi_label=False) # auc on prcision-recall curve\n acc = tf.keras.metrics.CategoricalAccuracy(name='acc')\n return [acc, auc]\n\ndef padded_cmap(y_true, y_pred, padding_factor=5):\n num_classes = y_true.shape[1]\n pad_rows = np.array([[1]*num_classes]*padding_factor)\n y_true = np.concatenate([y_true, pad_rows])\n y_pred = np.concatenate([y_pred, pad_rows])\n score = sklearn.metrics.average_precision_score(y_true, y_pred, average='macro',)\n return score\n\ndef get_loss():\n if CFG.loss==\"CCE\":\n loss = tf.keras.losses.CategoricalCrossentropy(label_smoothing=CFG.label_smoothing)\n elif CFG.loss==\"BCE\":\n loss = tf.keras.losses.BinaryCrossentropy(label_smoothing=CFG.label_smoothing)\n else:\n raise ValueError(\"Loss not found\")\n return loss\n \ndef get_optimizer():\n if CFG.optimizer == \"Adam\":\n opt = tf.keras.optimizers.Adam(learning_rate=CFG.lr)\n else:\n raise ValueError(\"Optmizer not found\")\n return opt","metadata":{"papermill":{"duration":0.28125,"end_time":"2022-03-08T03:20:04.498883","exception":false,"start_time":"2022-03-08T03:20:04.217633","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-03-13T19:40:50.047247Z","iopub.execute_input":"2023-03-13T19:40:50.047884Z","iopub.status.idle":"2023-03-13T19:40:50.058932Z","shell.execute_reply.started":"2023-03-13T19:40:50.047851Z","shell.execute_reply":"2023-03-13T19:40:50.057845Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Modeling 🤖\n* The following model employs **Filter Stride Reduction (FSR)**, which reduces the stride of the stem block by a factor of 2. This results in a larger feature map and less information loss.\n\n* As the train data labels are weak, but the test data labels are strong, we trained the model for a longer duration (20 seconds), but it will be inferred on test data with a shorter time duration (5 seconds). To accommodate this requirement, the model's input is defined as (None, None), which allows it to accept **Dynamic Input**.","metadata":{"papermill":{"duration":0.182769,"end_time":"2022-03-08T03:20:04.861966","exception":false,"start_time":"2022-03-08T03:20:04.679197","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import efficientnet.tfkeras as efn\n\ndef build_model(CFG, compile_model=True):\n \"\"\"\n Builds and returns a model based on the specified configuration.\n \"\"\"\n # Set the image input shape\n DIM = (None, None)\n # Load the specified EfficientNet model\n base = getattr(efn, CFG.model_name)(input_shape=(*DIM, 3),\n include_top=False,\n weights=CFG.pretrain,\n fsr=CFG.fsr,\n )\n # Create an input layer for the model\n inp = tf.keras.layers.Input(shape=(*DIM, 3))\n # Pass the input through the base model\n out = base(inp)\n # Add a global average pooling layer\n out = tf.keras.layers.GlobalAveragePooling2D()(out)\n # Add an optional bottleneck layer\n if CFG.neck_features:\n out = tf.keras.layers.Dense(CFG.num_features, activation='selu')(out)\n # Add a final dense layer for classification\n out = tf.keras.layers.Dense(len(CFG.class_names), activation=CFG.final_act)(out)\n # Create the TensorFlow model with the input and output layers\n model = tf.keras.Model(inputs=inp, outputs=out)\n if compile_model:\n # Set the optimizer\n opt = get_optimizer()\n # Set the loss function\n loss = get_loss()\n # Set the evaluation metrics\n metrics = get_metrics()\n # Compile the model with the specified optimizer, loss function, and metrics\n model.compile(optimizer=opt,\n loss=loss,\n metrics=metrics)\n return model","metadata":{"papermill":{"duration":1.239321,"end_time":"2022-03-08T03:20:06.281118","exception":false,"start_time":"2022-03-08T03:20:05.041797","status":"completed"},"tags":[],"_kg_hide-input":false,"execution":{"iopub.status.busy":"2023-03-13T19:40:53.885259Z","iopub.execute_input":"2023-03-13T19:40:53.886073Z","iopub.status.idle":"2023-03-13T19:40:53.997229Z","shell.execute_reply.started":"2023-03-13T19:40:53.886034Z","shell.execute_reply":"2023-03-13T19:40:53.996187Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = build_model(CFG)\nmodel.summary()","metadata":{"papermill":{"duration":37.756883,"end_time":"2022-03-08T03:20:44.226871","exception":false,"start_time":"2022-03-08T03:20:06.469988","status":"completed"},"tags":[],"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-13T19:40:54.297598Z","iopub.execute_input":"2023-03-13T19:40:54.297894Z","iopub.status.idle":"2023-03-13T19:40:58.171605Z","shell.execute_reply.started":"2023-03-13T19:40:54.297868Z","shell.execute_reply":"2023-03-13T19:40:58.170336Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Check Model O/P","metadata":{}},{"cell_type":"code","source":"out = model(imgs, training=False)\nprint(out.shape)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-13T19:41:03.686555Z","iopub.execute_input":"2023-03-13T19:41:03.686914Z","iopub.status.idle":"2023-03-13T19:41:04.492187Z","shell.execute_reply.started":"2023-03-13T19:41:03.686886Z","shell.execute_reply":"2023-03-13T19:41:04.491026Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# LR Schedule ⚓\n* Learning Rate scheduler for transfer learning. \n* The learning rate starts from `lr_start`, then decreases to a`lr_min` using different methods namely,\n * **step**: Reduce lr step wise like stair.\n * **cos**: Follow Cosine graph to reduce lr.\n * **exp**: Reduce lr exponentially.","metadata":{"papermill":{"duration":0.182502,"end_time":"2022-03-08T03:20:44.594886","exception":false,"start_time":"2022-03-08T03:20:44.412384","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import math\n\ndef get_lr_callback(batch_size=8, mode='cos', epochs=CFG.epochs, plot=False):\n \"\"\"\n Returns a learning rate scheduler callback for a given batch size, mode, and number of epochs.\n \"\"\"\n # Define the learning rate schedule.\n lr_start = 0.000005\n lr_max = 0.00000140 * batch_size\n lr_min = 0.000001\n lr_ramp_ep = 5\n lr_sus_ep = 0\n lr_decay = 0.8\n \n # Function to update the lr\n def lrfn(epoch):\n if epoch < lr_ramp_ep:\n lr = (lr_max - lr_start) / lr_ramp_ep * epoch + lr_start\n\n elif epoch < lr_ramp_ep + lr_sus_ep:\n lr = lr_max\n\n elif CFG.scheduler == 'exp':\n lr = (lr_max - lr_min) * lr_decay**(epoch - \\\n lr_ramp_ep - lr_sus_ep) + lr_min\n\n elif CFG.scheduler == 'step':\n lr = lr_max * lr_decay**((epoch - lr_ramp_ep - lr_sus_ep) // 2)\n\n elif CFG.scheduler == 'cos':\n decay_total_epochs = epochs - lr_ramp_ep - lr_sus_ep + 3\n decay_epoch_index = epoch - lr_ramp_ep - lr_sus_ep\n phase = math.pi * decay_epoch_index / decay_total_epochs\n cosine_decay = 0.5 * (1 + math.cos(phase))\n lr = (lr_max - lr_min) * cosine_decay + lr_min\n return lr\n \n # Plot the lr curve\n if plot:\n plt.figure(figsize=(10,5))\n plt.plot(np.arange(epochs), [lrfn(epoch) for epoch in np.arange(epochs)], marker='o')\n plt.xlabel('epoch'); plt.ylabel('learnig rate')\n plt.title('Learning Rate Scheduler')\n plt.show()\n \n # Crate lr-callback to update lr during training\n lr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=False)\n return lr_callback","metadata":{"_kg_hide-input":true,"papermill":{"duration":0.510014,"end_time":"2022-03-08T03:20:45.290695","exception":false,"start_time":"2022-03-08T03:20:44.780681","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-03-13T19:41:06.820029Z","iopub.execute_input":"2023-03-13T19:41:06.820986Z","iopub.status.idle":"2023-03-13T19:41:06.833425Z","shell.execute_reply.started":"2023-03-13T19:41:06.820948Z","shell.execute_reply":"2023-03-13T19:41:06.832469Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"_=get_lr_callback(CFG.batch_size*CFG.replicas, plot=True)","metadata":{"execution":{"iopub.status.busy":"2023-03-13T19:41:07.199566Z","iopub.execute_input":"2023-03-13T19:41:07.199841Z","iopub.status.idle":"2023-03-13T19:41:07.762336Z","shell.execute_reply.started":"2023-03-13T19:41:07.199816Z","shell.execute_reply":"2023-03-13T19:41:07.761296Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Wandb Logger 🕵️","metadata":{"papermill":{"duration":0.182406,"end_time":"2022-03-08T03:20:45.658265","exception":false,"start_time":"2022-03-08T03:20:45.475859","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import yaml\n\ndef wandb_init(fold):\n \"\"\"\n Initializes the W&B run by creating a config file and initializing a W&B run.\n \"\"\"\n # Create a dictionary of configuration parameters\n config = {k:v for k,v in dict(vars(CFG)).items() if '__' not in k}\n config.update({\"fold\":int(fold)}) # int is to convert numpy.int -> int\n # Dump the configuration dictionary to a YAML file\n yaml.dump(config, open(f'/kaggle/working/config fold-{fold}.yaml', 'w'),)\n # Load the configuration dictionary from the YAML file\n config = yaml.load(open(f'/kaggle/working/config fold-{fold}.yaml', 'r'), Loader=yaml.FullLoader)\n # Initialize a W&B run with the given configuration parameters\n run = wandb.init(project=\"birdclef-2023-public\",\n name=f\"fold-{fold}|dim-{CFG.img_size[1]}x{CFG.img_size[0]}|model-{CFG.model_name}\",\n config=config,\n group=CFG.comment,\n save_code=True,)\n return run\n\n \ndef log_wandb(valid_df):\n \"\"\"Log and save validation results with missclassified examples as audio in W&B\"\"\"\n # Query only the rows with miss predictions\n save_df = valid_df.query(\"miss==True\")\n # Map the predicted and target labels to their corresponding names\n save_df.loc[:, 'pred_name'] = save_df.pred.map(CFG.label2name)\n save_df.loc[:, 'target_name'] = save_df.target.map(CFG.label2name)\n # Trim the dataframe for debugging purposes\n if CFG.debug:\n save_df = save_df.iloc[:CFG.replicas*CFG.batch_size*CFG.infer_bs]\n # Get the columns to be included in the wandb table\n noimg_cols = [*CFG.tab_cols, 'target', 'pred', 'target_name','pred_name']\n # Retain only the necessary columns\n save_df = save_df.loc[:, noimg_cols]\n\n data = []\n # Load audio files for each miss prediction\n for idx, row in tqdm(save_df.iterrows(), total=len(save_df), desc='wandb ', position=0, leave=True):\n filepath = '/kaggle/input/birdclef-2023/train_audio/'+row.filename\n audio, sr = librosa.load(filepath, sr=None)\n # Add the audio file to the data list along with the other relevant information\n data+=[[*row.tolist(), wandb.Audio(audio, caption=row.filename, sample_rate=sr)]]\n # Create a wandb table with the audio files and other relevant information\n wandb_table = wandb.Table(data=data, columns=[*noimg_cols, 'audio'])\n # Log the scores and wandb table to wandb\n wandb.log({'best': scores,\n 'table': wandb_table,\n })","metadata":{"_kg_hide-input":true,"papermill":{"duration":0.288344,"end_time":"2022-03-08T03:20:47.977099","exception":false,"start_time":"2022-03-08T03:20:47.688755","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-03-13T19:41:21.889250Z","iopub.execute_input":"2023-03-13T19:41:21.890202Z","iopub.status.idle":"2023-03-13T19:41:21.904886Z","shell.execute_reply.started":"2023-03-13T19:41:21.890160Z","shell.execute_reply":"2023-03-13T19:41:21.903926Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training 🚅\nOur model will be trained on the `train` data and `valid` data will be used to save `checkpoint`.","metadata":{"papermill":{"duration":0.182448,"end_time":"2022-03-08T03:20:48.340679","exception":false,"start_time":"2022-03-08T03:20:48.158231","status":"completed"},"tags":[]}},{"cell_type":"code","source":"oof_pred = []; oof_true = []; oof_val = []; oof_ids = []; oof_folds = [] \n\nfor fold in range(CFG.num_fold):\n # Check if the fold is selected\n if fold not in CFG.selected_folds:\n continue\n \n # Initialize Weights and Biases\n if CFG.wandb:\n run = wandb_init(fold)\n WandbCallback = wandb.keras.WandbCallback(save_model=False)\n \n # Compute batch size and number of samples to drop\n infer_bs = (CFG.batch_size*CFG.infer_bs)\n drop_remainder = CFG.drop_remainder\n \n # Split dataset with cv filter\n if CFG.cv_filter:\n train_df = df.query(\"fold!=@fold | ~cv\").reset_index(drop=True)\n valid_df = df.query(\"fold==@fold & cv\").reset_index(drop=True)\n else:\n train_df = df.query(\"fold!=@fold\").reset_index(drop=True)\n valid_df = df.query(\"fold==@fold\").reset_index(drop=True)\n \n # Upsample train data\n train_df = upsample_data(train_df, thr=CFG.upsample_thr)\n\n # Get file paths and labels\n train_paths = train_df.filepath.values; train_labels = train_df.target.values\n valid_paths = valid_df.filepath.values; valid_labels = valid_df.target.values\n\n # Shuffle the file paths and labels\n index = np.arange(len(train_paths))\n np.random.shuffle(index)\n train_paths = train_paths[index]\n train_labels = train_labels[index]\n \n # For debugging\n if CFG.debug:\n min_samples = CFG.batch_size*CFG.replicas*2\n train_paths = train_paths[:min_samples]; train_labels = train_labels[:min_samples]\n valid_paths = valid_paths[:min_samples]; valid_labels = valid_labels[:min_samples]\n\n # Compute the number of training and validation samples\n num_train = len(train_paths); num_valid = len(valid_paths)\n \n # Log the number of training and validation samples if Weights and Biases is being used\n if CFG.wandb:\n wandb.log({'num_train':num_train,\n 'num_valid':num_valid})\n \n # Build the training and validation datasets\n cache=True\n train_ds = build_dataset(train_paths, train_labels, \n batch_size=CFG.batch_size*CFG.replicas, cache=cache, shuffle=True,\n drop_remainder=drop_remainder)\n valid_ds = build_dataset(valid_paths, valid_labels,\n batch_size=CFG.batch_size*CFG.replicas, cache=cache, shuffle=False,\n augment=False, repeat=False, drop_remainder=drop_remainder)\n \n # Print information about the fold and training\n print('#'*25); print('#### FOLD',fold+1)\n print('#### Image Size: (%i, %i) | Model: %s | Batch Size: %i | Scheduler: %s'%\n (*CFG.img_size, CFG.model_name, CFG.batch_size*CFG.replicas, CFG.scheduler))\n print('#### Num Train: {:,} | Num Valid: {:,}'.format(len(train_paths), len(valid_paths)))\n \n # Clear the session and build the model\n K.clear_session()\n with strategy.scope():\n model = build_model(CFG)\n \n print('#'*25) \n \n # Callbacks\n sv = tf.keras.callbacks.ModelCheckpoint(\n 'fold-%i.h5'%fold, monitor='val_auc', verbose=0, save_best_only=True,\n save_weights_only=False, mode='max', save_freq='epoch')\n callbacks = [sv, get_lr_callback(CFG.batch_size*CFG.replicas)]\n if CFG.wandb:\n # wandb callback for logging metrics and media\n callbacks.append(WandbCallback)\n\n # Training\n print('# Training')\n history = model.fit(\n train_ds, \n epochs=2 if CFG.debug else CFG.epochs, \n callbacks=callbacks, \n steps_per_epoch=len(train_paths)/CFG.batch_size//CFG.replicas,\n validation_data=valid_ds, \n verbose=CFG.verbose,\n )\n \n # Load best checkpoint\n print('# Loading best model')\n model.load_weights('fold-%i.h5'%fold)\n \n # Predict on the validation data for oof result\n print('# Infering OOF')\n valid_ds = build_dataset(valid_paths, labels=None, augment=CFG.tta>1, repeat=True, cache=False, \n shuffle=False, batch_size=infer_bs*CFG.replicas, \n drop_remainder=drop_remainder)\n ct_valid = len(valid_paths); STEPS = CFG.tta * ct_valid/infer_bs/CFG.replicas\n pred = model.predict(valid_ds,steps=STEPS,verbose=CFG.verbose)[:CFG.tta*ct_valid,] \n pred = np.mean(pred.reshape((CFG.tta,ct_valid,-1)),axis=0)\n oof_pred.append(pred) \n\n # Get ids and targets\n oof_true.append(valid_df[CFG.target_col].values[:ct_valid])\n oof_folds.append(np.ones_like(oof_true[-1],dtype='int8')*fold )\n oof_ids.append(valid_paths)\n \n # Save valid data prediction\n y_true = oof_true[-1].reshape(-1).astype('float32')\n y_pred = oof_pred[-1].argmax(axis=-1)\n valid_df.loc[:num_valid - 1, 'pred'] = y_pred\n valid_df.loc[:num_valid - 1, 'miss'] = y_true != y_pred\n valid_df.loc[:num_valid - 1, CFG.class_names] = oof_pred[-1].tolist()\n valid_df.to_csv(f'fold_{fold:02d}.csv',index=False)\n \n # Log the metrics\n scores = {}\n cmAP = padded_cmap(tf.keras.utils.to_categorical(y_true), oof_pred[-1])\n best_epoch = np.argmax(history.history['val_'+CFG.monitor], axis=-1) + 1\n best_score = history.history['val_'+CFG.monitor][best_epoch - 1]\n scores.update({'auc': best_score,\n 'epoch': best_epoch,\n 'cmAP': cmAP,})\n oof_val.append(best_score)\n print('\\n>>> FOLD %i OOF AUC = %.3f | Padded_cmAP = %.3f' % (fold, oof_val[-1], cmAP))\n \n # Show training plot\n if CFG.training_plot:\n plot_history(history)\n \n # Log metrics, media to wandb\n if CFG.wandb:\n print('# WandB')\n log_wandb(valid_df)\n wandb.run.finish()\n display(ipd.IFrame(run.url, width=1080, height=720))","metadata":{"_kg_hide-input":true,"_kg_hide-output":false,"papermill":{"duration":3436.548849,"end_time":"2022-03-08T04:18:05.540592","exception":true,"start_time":"2022-03-08T03:20:48.991743","status":"failed"},"tags":[],"execution":{"iopub.status.busy":"2023-03-12T19:42:10.461526Z","iopub.execute_input":"2023-03-12T19:42:10.462498Z","iopub.status.idle":"2023-03-12T20:19:02.651687Z","shell.execute_reply.started":"2023-03-12T19:42:10.462460Z","shell.execute_reply":"2023-03-12T20:19:02.650392Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training Log\n### [Click Here ➡️](https://wandb.ai/awsaf49/birdclef-2023-public) to check training log in **WandB** dashboard.\n\n","metadata":{}},{"cell_type":"markdown","source":"# Performance 🎭\nIn the following section we'll look into our overall score of `cmAP`","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"def get_id(row):\n row['filename'] = row['filepath'].split('/',5)[-1]\n return row","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-10T18:16:26.506695Z","iopub.status.idle":"2023-03-10T18:16:26.507163Z","shell.execute_reply.started":"2023-03-10T18:16:26.506913Z","shell.execute_reply":"2023-03-10T18:16:26.506933Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# OOF Data\ny_pred = np.concatenate(oof_pred); y_true = np.concatenate(oof_true);\nids = np.concatenate(oof_ids); folds = np.concatenate(oof_folds)\n\n# Overall cmAP\ncmAP = padded_cmap(tf.keras.utils.to_categorical(y_true), y_pred)\n\n# Overall AUC in PR curve\nm = tf.keras.metrics.AUC(curve='PR')\nm.update_state(tf.keras.utils.to_categorical(y_true), y_pred)\nauc = m.result().numpy()\n\nprint('>>> Overall cmAP: ', cmAP)\nprint('>>> Overall AUC(PR): ', auc)","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-10T18:16:26.509232Z","iopub.status.idle":"2023-03-10T18:16:26.509614Z","shell.execute_reply.started":"2023-03-10T18:16:26.509433Z","shell.execute_reply":"2023-03-10T18:16:26.509451Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Save OOF","metadata":{}},{"cell_type":"code","source":"# Save OOF data to disk\ncolumns = ['filepath', 'fold', 'true', 'pred', *CFG.class_names]\ndf_oof = pd.DataFrame(np.concatenate([ids[:,None], folds, y_true,\n np.argmax(y_pred,axis=1)[:,None], y_pred], axis=1), columns=columns)\ndf_oof['class_name'] = df_oof.true.map(CFG.label2name)\ndf_oof['miss'] = df_oof.true!=df_oof.pred\ntqdm.pandas(desc='id ')\ndf_oof = df_oof.progress_apply(get_id,axis=1)\ndf_oof.to_csv('oof.csv',index=False)\ndisplay(df_oof.head(2))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-10T18:16:26.511244Z","iopub.status.idle":"2023-03-10T18:16:26.511633Z","shell.execute_reply.started":"2023-03-10T18:16:26.511436Z","shell.execute_reply":"2023-03-10T18:16:26.511454Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Missed Cases","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"print('Miss Total:')\ndisplay(df_oof.query(\"miss==True\").shape[0])\n\nprint()\nprint('Miss Distribution Top10:')\ndisplay(df_oof.query(\"miss==True\").class_name.value_counts()[:10])","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-10T18:16:26.512946Z","iopub.status.idle":"2023-03-10T18:16:26.513351Z","shell.execute_reply.started":"2023-03-10T18:16:26.513139Z","shell.execute_reply":"2023-03-10T18:16:26.513160Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Reference ✍️\n* [Fake Speech Detection: Conformer [TF]](https://www.kaggle.com/code/awsaf49/fake-speech-detection-conformer-tf/) by @awsaf49\n* [RANZCR: EfficientNet TPU Training](https://www.kaggle.com/code/xhlulu/ranzcr-efficientnet-tpu-training) by @xhlulu\n* [Triple Stratified KFold with TFRecords](https://www.kaggle.com/code/cdeotte/triple-stratified-kfold-with-tfrecords) by @cdeotte","metadata":{}},{"cell_type":"markdown","source":"# Remove Files ✂️","metadata":{}},{"cell_type":"code","source":"import shutil\ntry:\n !rm -rf ./wandb\nexcept:\n pass","metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-10T18:16:26.514550Z","iopub.status.idle":"2023-03-10T18:16:26.514922Z","shell.execute_reply.started":"2023-03-10T18:16:26.514710Z","shell.execute_reply":"2023-03-10T18:16:26.514726Z"},"trusted":true},"execution_count":null,"outputs":[]}]}