{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# What this notebook does?\n\n* by applying log-compression and PCEN to the audio files, compare the result of processed audio data\n\nI use librosa 0.9.1 implementation[2] for PCEN.\n\n# Introduction\n\nWhen converting audio file to mel-frequency spectrogram, we usually use log compression to handle dynamic range of the power spectrum. It is simple and easy to use, but Wang et. al. developed more advanced method: **per-channel energy normalization (PCEN)**[1]. It's named as \"normalization\", it actually perform more complicated things. Briefly saying, it enhances signals changing frequency with time (chirp[3]), and suppressing signals with static frequency over time. It is also considered as a way of noise-suppression. In some application of sound event detection (SED) or audio tagging, it is reported to perform better compared to the simple log-compression[4].\n\n# Reference\n\n* [1] [Wang, Y., Getreuer, P., Hughes, T., Lyon, R. F., & Saurous, R. A. (2017, March). Trainable frontend for robust and far-field keyword spotting. In Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE International Conference on (pp. 5670-5674). IEEE.](https://www.semanticscholar.org/paper/Trainable-frontend-for-robust-and-far-field-keyword-Wang-Getreuer/4f23f2d194ddb615a31d6f75da793dcc89962b22)\n* [2] https://librosa.org/doc/0.9.1/generated/librosa.pcen.html?highlight=pcen#librosa.pcen\n* [3] https://en.m.wikipedia.org/wiki/Chirp\n* [4] [Lostanlen, V., Salamon, J., McFee, B., Cartwright, M., Farnsworth, A., Kelling, S., and Bello, J. P. Per-Channel Energy Normalization: Why and How. IEEE Signal Processing Letters, 26(1), 39-43.](https://www.semanticscholar.org/paper/Per-Channel-Energy-Normalization%3A-Why-and-How-Lostanlen-Salamon/ff26a25c0c48666847965f4e9a0691ab6e5a7c75)","metadata":{}},{"cell_type":"code","source":"!pip install nb-black > /dev/null","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-03-30T03:25:32.728383Z","iopub.execute_input":"2022-03-30T03:25:32.728863Z","iopub.status.idle":"2022-03-30T03:25:44.276647Z","shell.execute_reply.started":"2022-03-30T03:25:32.728762Z","shell.execute_reply":"2022-03-30T03:25:44.275791Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\nimport warnings\n\nwarnings.filterwarnings(\"ignore\")\n\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport matplotlib.gridspec as gridspec\nimport seaborn as sns\nimport soundfile as sf\n\nimport librosa\nimport librosa.display\nimport IPython.display as ipd\n\nplt.style.use(\"ggplot\")\n\n%load_ext lab_black\n%load_ext autoreload\n%autoreload 2","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-30T03:25:46.798826Z","iopub.execute_input":"2022-03-30T03:25:46.799137Z","iopub.status.idle":"2022-03-30T03:25:46.856143Z","shell.execute_reply.started":"2022-03-30T03:25:46.7991Z","shell.execute_reply":"2022-03-30T03:25:46.855278Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"librosa.__version__","metadata":{"execution":{"iopub.status.busy":"2022-03-30T03:25:46.857664Z","iopub.execute_input":"2022-03-30T03:25:46.857862Z","iopub.status.idle":"2022-03-30T03:25:46.89941Z","shell.execute_reply.started":"2022-03-30T03:25:46.857838Z","shell.execute_reply":"2022-03-30T03:25:46.89852Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Helper Functions","metadata":{}},{"cell_type":"code","source":"def create_mel_spectrogram(audio_file, **spec_params):\n    sr, hop_length, n_fft, n_mels, fmin, fmax = [\n        spec_params[k] for k in [\"sr\", \"hop_length\", \"n_fft\", \"n_mels\", \"fmin\", \"fmax\"]\n    ]\n    audio, _ = librosa.core.load(audio_file, sr=sr, mono=True)\n    melspec = librosa.feature.melspectrogram(\n        audio,\n        sr=sr,\n        n_fft=n_fft,\n        hop_length=hop_length,\n        n_mels=n_mels,\n        fmin=fmin,\n        fmax=fmax,\n        power=1,\n    )\n    return melspec\n\n\ndef pcen_bird(melspec, **spec_params):\n    \"\"\"\n    parameters are taken from [1]:\n        - [1] Lostanlen, et. al. Per-Channel Energy Normalization: Why and How. IEEE Signal Processing Letters, 26(1), 39-43.\n    \"\"\"\n    sr, hop_length = [spec_params[k] for k in [\"sr\", \"hop_length\"]]\n    return librosa.pcen(\n        melspec * (2 ** 31),\n        time_constant=0.06,\n        eps=1e-6,\n        gain=0.8,\n        power=0.25,\n        bias=10,\n        sr=sr,\n        hop_length=hop_length,\n    )\n\n\ndef mel2audio(melspec, **spec_params):\n    n_fft, sr, hop_length = [spec_params[k] for k in [\"n_fft\", \"sr\", \"hop_length\"]]\n    return librosa.feature.inverse.mel_to_audio(\n        melspec, sr=sr, n_fft=n_fft, hop_length=hop_length, power=1\n    )\n\n\ndef get_fullpath(filename, audio_path=\"../input/birdclef-2022/train_audio\"):\n    return f\"{audio_path}/{filename}\"\n\n\ndef play_audio(audio_file):\n    display(ipd.Audio(audio_file))\n\n\ndef gen_spec_and_audio(audio_name):\n    out_file = audio_name.replace(\"/\", \"_\")[:-4] + \".wav\"\n    plot_spectrograms(audio_name)\n    print(\"source audio:\")\n    play_audio(get_fullpath(audio_name))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-30T03:28:13.81856Z","iopub.execute_input":"2022-03-30T03:28:13.819295Z","iopub.status.idle":"2022-03-30T03:28:13.88863Z","shell.execute_reply.started":"2022-03-30T03:28:13.819224Z","shell.execute_reply":"2022-03-30T03:28:13.887627Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_spectrograms(\n    audio_name,\n    spec_params=dict(\n        sr=32_000, hop_length=320, n_fft=800, n_mels=128, fmin=20, fmax=14_000\n    ),\n):\n    sr, hop_length, fmin, fmax, n_mels = [\n        spec_params[k] for k in [\"sr\", \"hop_length\", \"fmin\", \"fmax\", \"n_mels\"]\n    ]\n    print(f\"parameters: {spec_params}\")\n    audio_file = get_fullpath(audio_name)\n    if not os.path.isfile(audio_file):\n        raise FileNotFoundError\n    melspec = create_mel_spectrogram(audio_file, **spec_params)\n    log_melspec = librosa.amplitude_to_db(melspec, ref=np.max)\n    pcen_melspec = pcen_bird(melspec, **spec_params)\n\n    # plot\n    plt.tight_layout()\n    fig = plt.figure(figsize=(24, 8))\n    gs0 = gridspec.GridSpec(1, 5, figure=fig)\n    gs00 = gridspec.GridSpecFromSubplotSpec(2, 1, subplot_spec=gs0[0:3])\n    ax1 = fig.add_subplot(gs00[0])\n    ax2 = fig.add_subplot(gs00[1])\n    gs01 = gridspec.GridSpecFromSubplotSpec(2, 1, subplot_spec=gs0[3])\n    ax3 = fig.add_subplot(gs01[0])\n    ax4 = fig.add_subplot(gs01[1])\n    gs02 = gridspec.GridSpecFromSubplotSpec(2, 1, subplot_spec=gs0[4])\n    ax5 = fig.add_subplot(gs02[0])\n    ax6 = fig.add_subplot(gs02[1])\n\n    # ax1\n    img = librosa.display.specshow(\n        log_melspec,\n        sr=sr,\n        hop_length=hop_length,\n        fmin=fmin,\n        fmax=fmax,\n        x_axis=\"time\",\n        y_axis=\"mel\",\n        ax=ax1,\n    )\n    ax1.set(title=\"Spectrogram after log-compression\", xlabel=None)\n    ax1.label_outer()\n    fig.colorbar(img, ax=ax1, format=\"%+2.0f dB\")\n\n    # ax2\n    img_pcen = librosa.display.specshow(\n        pcen_melspec,\n        sr=sr,\n        hop_length=hop_length,\n        fmin=fmin,\n        fmax=fmax,\n        x_axis=\"time\",\n        y_axis=\"mel\",\n        ax=ax2,\n    )\n    ax2.set(title=\"Spectrogram after PCEN\")\n    fig.colorbar(img_pcen, ax=ax2)\n\n    # ax3\n    img = img.get_array()\n    ax3.set(title=\"Amplitude dist. log-comp\")\n    sns.histplot(img, ax=ax3)\n\n    # ax4\n    img_pcen = img_pcen.get_array()\n    ax4.set(title=\"Amplitude dist. PCEN\")\n    sns.histplot(img_pcen, ax=ax4)\n\n    # ax5\n    img = img.reshape((n_mels, -1))\n    corr = np.corrcoef(img)\n    assert corr.shape == (n_mels, n_mels)\n    sns.heatmap(corr, ax=ax5)\n    ax5.set(title=\"Corr. log-comp.\")\n    ax5.axis(\"off\")\n\n    # ax6\n    img_pcen = img_pcen.reshape((n_mels, -1))\n    corr_pcen = np.corrcoef(img_pcen)\n    assert corr_pcen.shape == (n_mels, n_mels)\n    sns.heatmap(corr_pcen, ax=ax6)\n    ax6.set(title=\"Corr. PCEN\")\n    ax6.axis(\"off\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-30T03:28:49.577542Z","iopub.execute_input":"2022-03-30T03:28:49.577838Z","iopub.status.idle":"2022-03-30T03:28:49.672223Z","shell.execute_reply.started":"2022-03-30T03:28:49.577803Z","shell.execute_reply":"2022-03-30T03:28:49.671417Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Visualize scored bird call/song\n\n## Discussion\n\n* Left top & bottom pictures are spectrograms after log-compression and PCEN respectively. We can generally see **PCEN enhances signals changing frequency with time**, whereas other signals are suppressed.\n* Middle top & bottom picture: distribution of amplitude in the mel-spectrogram after apllying log-commpression and PCEN respectively. We can see PCEN generally make signals **Gaussianized** than log-compression.\n* Ritht top & bottom pictures are self-correlations between frequency channels. It shows PCEN are more close to identity matrix, which means that **each channels are disentangled** (we usually saids it's become **\"whitened\"**).","metadata":{}},{"cell_type":"code","source":"gen_spec_and_audio(\"akiapo/XC306424.ogg\")","metadata":{"execution":{"iopub.status.busy":"2022-03-30T03:28:54.344191Z","iopub.execute_input":"2022-03-30T03:28:54.344488Z","iopub.status.idle":"2022-03-30T03:28:59.414633Z","shell.execute_reply.started":"2022-03-30T03:28:54.344459Z","shell.execute_reply":"2022-03-30T03:28:59.413699Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gen_spec_and_audio(\"aniani/XC210206.ogg\")","metadata":{"execution":{"iopub.status.busy":"2022-03-30T03:29:11.986525Z","iopub.execute_input":"2022-03-30T03:29:11.987455Z","iopub.status.idle":"2022-03-30T03:29:16.525492Z","shell.execute_reply.started":"2022-03-30T03:29:11.987411Z","shell.execute_reply":"2022-03-30T03:29:16.524692Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gen_spec_and_audio(\"apapan/XC27331.ogg\")","metadata":{"execution":{"iopub.status.busy":"2022-03-25T13:33:50.630698Z","iopub.execute_input":"2022-03-25T13:33:50.630985Z","iopub.status.idle":"2022-03-25T13:33:56.117557Z","shell.execute_reply.started":"2022-03-25T13:33:50.630951Z","shell.execute_reply":"2022-03-25T13:33:56.116936Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gen_spec_and_audio(\"barpet/XC441955.ogg\")","metadata":{"execution":{"iopub.status.busy":"2022-03-25T13:02:28.668487Z","iopub.execute_input":"2022-03-25T13:02:28.668807Z","iopub.status.idle":"2022-03-25T13:02:31.124696Z","shell.execute_reply.started":"2022-03-25T13:02:28.66877Z","shell.execute_reply":"2022-03-25T13:02:31.123981Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gen_spec_and_audio(\"crehon/XC122341.ogg\")","metadata":{"execution":{"iopub.status.busy":"2022-03-25T13:02:31.125717Z","iopub.execute_input":"2022-03-25T13:02:31.126357Z","iopub.status.idle":"2022-03-25T13:02:35.989563Z","shell.execute_reply.started":"2022-03-25T13:02:31.126316Z","shell.execute_reply":"2022-03-25T13:02:35.988572Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gen_spec_and_audio(\"elepai/XC27344.ogg\")","metadata":{"execution":{"iopub.status.busy":"2022-03-25T13:02:48.756587Z","iopub.execute_input":"2022-03-25T13:02:48.756887Z","iopub.status.idle":"2022-03-25T13:02:53.332429Z","shell.execute_reply.started":"2022-03-25T13:02:48.756856Z","shell.execute_reply":"2022-03-25T13:02:53.331231Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gen_spec_and_audio(\"elepai/XC27352.ogg\")","metadata":{"execution":{"iopub.status.busy":"2022-03-25T13:02:53.334536Z","iopub.execute_input":"2022-03-25T13:02:53.335104Z","iopub.status.idle":"2022-03-25T13:02:57.071236Z","shell.execute_reply.started":"2022-03-25T13:02:53.33505Z","shell.execute_reply":"2022-03-25T13:02:57.070229Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gen_spec_and_audio(\"ercfra/XC252696.ogg\")","metadata":{"execution":{"iopub.status.busy":"2022-03-25T13:02:57.072573Z","iopub.execute_input":"2022-03-25T13:02:57.07294Z","iopub.status.idle":"2022-03-25T13:02:59.470038Z","shell.execute_reply.started":"2022-03-25T13:02:57.072897Z","shell.execute_reply":"2022-03-25T13:02:59.469325Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gen_spec_and_audio(\"ercfra/XC410276.ogg\")","metadata":{"execution":{"iopub.status.busy":"2022-03-25T13:02:59.471346Z","iopub.execute_input":"2022-03-25T13:02:59.472295Z","iopub.status.idle":"2022-03-25T13:03:03.099227Z","shell.execute_reply.started":"2022-03-25T13:02:59.472253Z","shell.execute_reply":"2022-03-25T13:03:03.098135Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gen_spec_and_audio(\"hawama/XC27350.ogg\")","metadata":{"execution":{"iopub.status.busy":"2022-03-25T13:04:15.100763Z","iopub.execute_input":"2022-03-25T13:04:15.101051Z","iopub.status.idle":"2022-03-25T13:04:18.336396Z","shell.execute_reply.started":"2022-03-25T13:04:15.101021Z","shell.execute_reply":"2022-03-25T13:04:18.335295Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gen_spec_and_audio(\"hawgoo/XC210217.ogg\")","metadata":{"execution":{"iopub.status.busy":"2022-03-25T13:04:18.338377Z","iopub.execute_input":"2022-03-25T13:04:18.33936Z","iopub.status.idle":"2022-03-25T13:04:21.666407Z","shell.execute_reply.started":"2022-03-25T13:04:18.339317Z","shell.execute_reply":"2022-03-25T13:04:21.665717Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gen_spec_and_audio(\"hawgoo/XC314509.ogg\")","metadata":{"execution":{"iopub.status.busy":"2022-03-25T13:04:21.667423Z","iopub.execute_input":"2022-03-25T13:04:21.668116Z","iopub.status.idle":"2022-03-25T13:04:25.330602Z","shell.execute_reply.started":"2022-03-25T13:04:21.668059Z","shell.execute_reply":"2022-03-25T13:04:25.329685Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gen_spec_and_audio(\"hawhaw/XC648121.ogg\")","metadata":{"execution":{"iopub.status.busy":"2022-03-25T13:04:25.332464Z","iopub.execute_input":"2022-03-25T13:04:25.333185Z","iopub.status.idle":"2022-03-25T13:04:28.281339Z","shell.execute_reply.started":"2022-03-25T13:04:25.333144Z","shell.execute_reply":"2022-03-25T13:04:28.279706Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gen_spec_and_audio(\"hawpet1/XC328230.ogg\")","metadata":{"execution":{"iopub.status.busy":"2022-03-25T13:04:28.282549Z","iopub.execute_input":"2022-03-25T13:04:28.282859Z","iopub.status.idle":"2022-03-25T13:04:31.604543Z","shell.execute_reply.started":"2022-03-25T13:04:28.28282Z","shell.execute_reply":"2022-03-25T13:04:31.603513Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gen_spec_and_audio(\"houfin/XC423359.ogg\")","metadata":{"execution":{"iopub.status.busy":"2022-03-25T13:04:31.605817Z","iopub.execute_input":"2022-03-25T13:04:31.60606Z","iopub.status.idle":"2022-03-25T13:04:35.693044Z","shell.execute_reply.started":"2022-03-25T13:04:31.606033Z","shell.execute_reply":"2022-03-25T13:04:35.691994Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gen_spec_and_audio(\"iiwi/XC326576.ogg\")","metadata":{"execution":{"iopub.status.busy":"2022-03-25T13:05:05.108931Z","iopub.execute_input":"2022-03-25T13:05:05.109223Z","iopub.status.idle":"2022-03-25T13:05:08.137409Z","shell.execute_reply.started":"2022-03-25T13:05:05.109192Z","shell.execute_reply":"2022-03-25T13:05:08.13651Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gen_spec_and_audio(\"jabwar/XC236038.ogg\")","metadata":{"execution":{"iopub.status.busy":"2022-03-25T13:05:08.139301Z","iopub.execute_input":"2022-03-25T13:05:08.139722Z","iopub.status.idle":"2022-03-25T13:05:13.051655Z","shell.execute_reply.started":"2022-03-25T13:05:08.139677Z","shell.execute_reply":"2022-03-25T13:05:13.050597Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gen_spec_and_audio(\"omao/XC385074.ogg\")","metadata":{"execution":{"iopub.status.busy":"2022-03-25T13:05:13.052888Z","iopub.execute_input":"2022-03-25T13:05:13.053103Z","iopub.status.idle":"2022-03-25T13:05:15.389995Z","shell.execute_reply.started":"2022-03-25T13:05:13.053076Z","shell.execute_reply":"2022-03-25T13:05:15.388962Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gen_spec_and_audio(\"puaioh/XC144894.ogg\")","metadata":{"execution":{"iopub.status.busy":"2022-03-25T13:05:15.391942Z","iopub.execute_input":"2022-03-25T13:05:15.392178Z","iopub.status.idle":"2022-03-25T13:05:17.89942Z","shell.execute_reply.started":"2022-03-25T13:05:15.392151Z","shell.execute_reply":"2022-03-25T13:05:17.898158Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gen_spec_and_audio(\"puaioh/XC144893.ogg\")","metadata":{"execution":{"iopub.status.busy":"2022-03-25T13:05:17.901108Z","iopub.execute_input":"2022-03-25T13:05:17.901452Z","iopub.status.idle":"2022-03-25T13:05:21.537442Z","shell.execute_reply.started":"2022-03-25T13:05:17.901404Z","shell.execute_reply":"2022-03-25T13:05:21.536735Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gen_spec_and_audio(\"skylar/XC595571.ogg\")","metadata":{"execution":{"iopub.status.busy":"2022-03-25T13:05:21.539022Z","iopub.execute_input":"2022-03-25T13:05:21.539517Z","iopub.status.idle":"2022-03-25T13:05:25.074487Z","shell.execute_reply.started":"2022-03-25T13:05:21.539481Z","shell.execute_reply":"2022-03-25T13:05:25.073529Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gen_spec_and_audio(\"warwhe1/XC503528.ogg\")","metadata":{"execution":{"iopub.status.busy":"2022-03-25T13:05:25.075988Z","iopub.execute_input":"2022-03-25T13:05:25.076293Z","iopub.status.idle":"2022-03-25T13:05:28.11442Z","shell.execute_reply.started":"2022-03-25T13:05:25.076251Z","shell.execute_reply":"2022-03-25T13:05:28.113475Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gen_spec_and_audio(\"yefcan/XC513862.ogg\")","metadata":{"execution":{"iopub.status.busy":"2022-03-25T13:05:28.115843Z","iopub.execute_input":"2022-03-25T13:05:28.116239Z","iopub.status.idle":"2022-03-25T13:05:31.890302Z","shell.execute_reply.started":"2022-03-25T13:05:28.116128Z","shell.execute_reply":"2022-03-25T13:05:31.889397Z"},"trusted":true},"execution_count":null,"outputs":[]}]}