{"cells":[{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install colorednoise > /dev/null","execution_count":null,"outputs":[]},{"metadata":{"id":"TXNF7lu4xbqu"},"cell_type":"markdown","source":"## About\n\nこのノートブックでは音データに対するData Augmentationを紹介します。画像同様、音のデータもData Augmentationは汎化性能を担保する上で大きな役割を果たしています。\n\nIn this notebook, I will introduce some basic Data Augmentation methods for audio. Similar in computer vision, Data Augmentation is quite effective for audio as well to make generalized model.\n\nMaybe you'll find some odd translation since I translated this from Japanese and in some part I just used the result of deepL..."},{"metadata":{"id":"zFtI2ZH7xVTC","trusted":true},"cell_type":"code","source":"import librosa\nimport librosa.display\n\nimport matplotlib.pyplot as plt\nimport numpy as np\n\nfrom pathlib import Path\nfrom IPython.display import Audio","execution_count":null,"outputs":[]},{"metadata":{"id":"IODfGVGt8kYi"},"cell_type":"markdown","source":"音のデータに適用できるData Augmentationは画像のものとは大きく異なります。音のデータには大きく分けて以下の二種類のData Augmentationが存在します。\n\n1. 元の音データ(1次元の状態)に対して適用されるData Augmentaion (Data Augmentation for waveform)\n2. スペクトログラムやメルスペクトログラムに対して適用されるData Augmentation (Data Augmentation for spectrogram)\n\n元の音データに対して適用されるData Augmentationは音の聞こえ方自体を変化させるものが多く画像のものとは大きく異なります。\n\n一方スペクトログラムに対して適用されるものは画像に対するData Augmentationに似ていますが、一つ大きく異なる点として、スペクトログラムには明確に軸があるという点を無視してData Augmentationをかけることには意味がありません。軸というのは、スペクトログラムのx軸は時間、y軸は周波数の軸であるということでここは明確に自然画像とは異なります。従って単純にFlipしたり回転をかけることは意味がないどころか有害です。"},{"metadata":{},"cell_type":"markdown","source":"Data Augmentations for audio is quite different from those for images. There are two types of augmentations:\n\n1. Augmentations for waveform\n2. Augmentations for spectrogram/melspectrogram\n\nAugmentation for waveform is applied to raw 1D signal, and changes how it sounds like, so we can check how it alters the signal by listening it.\nOn the other hand, augmentation for spectrogram is something similar to image augmentation. However, spectrogram has some big differences from natural image.\nOne of the biggest difference is that it has axis - time axis and frequency axis. Applying augmentations that ignores this axis (Flip, Rotation, etc...) is nonsense."},{"metadata":{"id":"yDUEZVvI-UdG"},"cell_type":"markdown","source":"## Data Loading\n\n\n"},{"metadata":{"id":"u0wuC5uG8jN1","executionInfo":{"status":"ok","timestamp":1605227706605,"user_tz":-540,"elapsed":1934,"user":{"displayName":"Hidehisa Arai","photoUrl":"","userId":"12684386091436935327"}},"outputId":"3f9b3dce-049f-4fbb-e317-8465e6fb61ff","trusted":true},"cell_type":"code","source":"DATA_DIR = Path(\"../input/rfcx-species-audio-detection/train\")","execution_count":null,"outputs":[]},{"metadata":{"id":"15jAUswG-tv3","executionInfo":{"status":"ok","timestamp":1605227727175,"user_tz":-540,"elapsed":19955,"user":{"displayName":"Hidehisa Arai","photoUrl":"","userId":"12684386091436935327"}},"outputId":"72c707d9-9147-42ff-d63f-d9088c10cd7c","trusted":true},"cell_type":"code","source":"flacfiles = list(DATA_DIR.glob(\"*.flac\"))\ny, sr = librosa.load(flacfiles[0], duration=10)\ny, sr","execution_count":null,"outputs":[]},{"metadata":{"id":"JST8TacX-_B8"},"cell_type":"markdown","source":"音のデータを読み込むことができました。今回はこのデータに様々なData Augmentationをかけてその結果を実際に聞いたり見たりしながら各Data Augmentationの効果を確認していきます。\n\nまずは元の音データを確認してみましょう。\n\nNow we've loaded audio data. In this notebook, we'll apply several Data Augmentations and check the result by listening or visually watching it."},{"metadata":{"id":"QBfSxk0z-7_l","executionInfo":{"status":"ok","timestamp":1605227733814,"user_tz":-540,"elapsed":2328,"user":{"displayName":"Hidehisa Arai","photoUrl":"","userId":"12684386091436935327"}},"outputId":"47828cdd-a2e7-4f55-ab8a-1e4b3376ac5f","trusted":true},"cell_type":"code","source":"Audio(y, rate=sr)","execution_count":null,"outputs":[]},{"metadata":{"id":"IypsOPUOCzm1","executionInfo":{"status":"ok","timestamp":1605227757758,"user_tz":-540,"elapsed":1616,"user":{"displayName":"Hidehisa Arai","photoUrl":"","userId":"12684386091436935327"}},"outputId":"55ef789b-4378-4709-d084-1ce879bb1fed","trusted":true},"cell_type":"code","source":"librosa.display.waveplot(y, sr=sr);","execution_count":null,"outputs":[]},{"metadata":{"id":"6KxVjQvTC4r8","executionInfo":{"status":"ok","timestamp":1605227760881,"user_tz":-540,"elapsed":2041,"user":{"displayName":"Hidehisa Arai","photoUrl":"","userId":"12684386091436935327"}},"outputId":"624a16a4-54f4-487a-ff7a-1eb7cbb2289a","trusted":true},"cell_type":"code","source":"melspec = librosa.power_to_db(librosa.feature.melspectrogram(y, sr=sr, n_mels=128))\nlibrosa.display.specshow(melspec, sr=sr, x_axis=\"time\", y_axis=\"mel\")\nplt.colorbar();","execution_count":null,"outputs":[]},{"metadata":{"id":"CdVKUPN8PyQB"},"cell_type":"markdown","source":"## 音データに対するData Augmentation (Data Augmentation for waveform)\n\nまずは、音の一次元データに対するData Augmentationを紹介します。ここでは実際に実装しながら以下のaugmentationを紹介します。\n\n1. AddGaussianNoise\n2. GaussianNoiseSNR\n3. PinkNoiseSNR\n4. PitchShift\n5. TimeStretch\n6. TimeShift\n7. VolumeControl\n\nなお、実装は[albumentations](https://github.com/albumentations-team/albumentations)を参考にしています。albumentationsのインターフェースを継承して作成をすることもできるのですが若干手間が多いため自前実装で済ませます。\n\nHere, I'll introduce some Data Augmentation methods for raw waveform.\n\n1. AddGaussianNoise\n2. GaussianNoiseSNR\n3. PinkNoiseSNR\n4. PitchShift\n5. TimeStretch\n6. TimeShift\n7. VolumeControl\n\nI imitated the implementation of [albumentations](https://github.com/albumentations-team/albumentations)."},{"metadata":{"id":"hTMz5-7tD0vV","trusted":true},"cell_type":"code","source":"class AudioTransform:\n    def __init__(self, always_apply=False, p=0.5):\n        self.always_apply = always_apply\n        self.p = p\n\n    def __call__(self, y: np.ndarray):\n        if self.always_apply:\n            return self.apply(y)\n        else:\n            if np.random.rand() < self.p:\n                return self.apply(y)\n            else:\n                return y\n\n    def apply(self, y: np.ndarray):\n        raise NotImplementedError\n\n\nclass Compose:\n    def __init__(self, transforms: list):\n        self.transforms = transforms\n\n    def __call__(self, y: np.ndarray):\n        for trns in self.transforms:\n            y = trns(y)\n        return y\n\n\nclass OneOf:\n    def __init__(self, transforms: list):\n        self.transforms = transforms\n\n    def __call__(self, y: np.ndarray):\n        n_trns = len(self.transforms)\n        trns_idx = np.random.choice(n_trns)\n        trns = self.transforms[trns_idx]\n        return trns(y)","execution_count":null,"outputs":[]},{"metadata":{"id":"qWXWLcVAVGkU"},"cell_type":"markdown","source":"### AddGaussianNoise\n\n正規分布に従うノイズ、いわゆるホワイトノイズを音に加えます。ノイズの振幅はランダムです。\n\n参考・出典: [Data Augmentation for Audio](https://medium.com/@makcedward/data-augmentation-for-audio-76912b01fdf6)の`NoiseInjection`より\n\nAdd noise that follows normal distribution (a.k.a whitenoise). The amplitude of noise is randomly decided.\n\nReference: [Data Augmentation for Audio](https://medium.com/@makcedward/data-augmentation-for-audio-76912b01fdf6)"},{"metadata":{"id":"oZIBa2KNVF70","trusted":true},"cell_type":"code","source":"class AddGaussianNoise(AudioTransform):\n    def __init__(self, always_apply=False, p=0.5, max_noise_amplitude=0.5, **kwargs):\n        super().__init__(always_apply, p)\n\n        self.noise_amplitude = (0.0, max_noise_amplitude)\n\n    def apply(self, y: np.ndarray, **params):\n        noise_amplitude = np.random.uniform(*self.noise_amplitude)\n        noise = np.random.randn(len(y))\n        augmented = (y + noise * noise_amplitude).astype(y.dtype)\n        return augmented","execution_count":null,"outputs":[]},{"metadata":{"id":"QfJw-gE-WK7M","executionInfo":{"status":"ok","timestamp":1605227776141,"user_tz":-540,"elapsed":1453,"user":{"displayName":"Hidehisa Arai","photoUrl":"","userId":"12684386091436935327"}},"outputId":"7792bd2d-5c4f-4cf9-c1b9-b0b67a729549","trusted":true},"cell_type":"code","source":"transform = AddGaussianNoise(always_apply=True, max_noise_amplitude=0.05)\ny_gaussian_added = transform(y)\nAudio(y_gaussian_added, rate=sr)","execution_count":null,"outputs":[]},{"metadata":{"id":"6EWNgDiEWbuS","executionInfo":{"status":"ok","timestamp":1605227790968,"user_tz":-540,"elapsed":847,"user":{"displayName":"Hidehisa Arai","photoUrl":"","userId":"12684386091436935327"}},"outputId":"4588bad2-73fe-4e83-ff6c-4e7c6b7b7f52","trusted":true},"cell_type":"code","source":"librosa.display.waveplot(y_gaussian_added, sr=sr);","execution_count":null,"outputs":[]},{"metadata":{"id":"lAKvyPy9WkpE","executionInfo":{"status":"ok","timestamp":1605227794793,"user_tz":-540,"elapsed":1277,"user":{"displayName":"Hidehisa Arai","photoUrl":"","userId":"12684386091436935327"}},"outputId":"1a6727e8-e881-428c-c3cd-05c2fe9d3da8","trusted":true},"cell_type":"code","source":"melspec = librosa.power_to_db(librosa.feature.melspectrogram(y_gaussian_added, sr=sr, n_mels=128))\nlibrosa.display.specshow(melspec, sr=sr, x_axis=\"time\", y_axis=\"mel\")\nplt.colorbar();","execution_count":null,"outputs":[]},{"metadata":{"id":"21hbXLmfW5DZ"},"cell_type":"markdown","source":"スペクトログラムを見るとわかりやすいのですが、全体に散らしたようにノイズが加わっていることがわかります。Gaussian Noiseを加えることでノイズが比較的少ないデータで学習して、ノイズが多い環境の音を処理しなければいけない場合などに汎化性能をあげることができます。"},{"metadata":{"id":"QTVLneYLXecQ"},"cell_type":"markdown","source":"### GaussianNoiseSNR\n\n上で紹介したData Augmentationの問題点はノイズの強さ(amplitude)を指定してしまうと、元の信号が微弱なときに雑音に覆い隠されてしまうことがありうる点です。\n\nこれを防ぐために元の音の中の信号の振幅を元に適切な雑音レベルを適応的に設定できるようにしたほうが使いやすいです。\n\n信号の大きさと雑音の大きさの比を表したものをSignal-to-Noise Ratio(SNR)と呼びます。信号の大きさ、といった場合には振幅のことをさすことが多いのですが多くの場合SNRは実際の振幅の比に対数を取ったものとして表現され、以下の式で計算されます。\n\n$$\nSNR = 20\\log_{10}\\frac{A_{signal}}{A_{noise}}\n$$\n\nこの量は大きければ大きいほど信号が強い、すなわち音が聞こえやすいことを表す量で単位はdB(デシベル)で表現されます。0dBで信号の強さと雑音の強さが釣り合っている状態で、負の場合には雑音の方が強い状態、正の場合には信号の方が強い状態です。\n\nまた、信号音の強さの推定法はいくつかあるかと思いますが、今回はクリップ内の振幅の絶対値の最大値を信号の振幅として扱います。\n\n参考・出典: [任意のSignal-to-Noise比の音声波形をPythonで作ろう！](https://engineering.linecorp.com/ja/blog/voice-waveform-arbitrary-signal-to-noise-ratio-python/)より\n\nThe problem with Data Augmentation introduced above is that if you specify an amplitude of noise, it can be masked by noise when the original signal is weak.\n\nTo prevent this, it is easier to adaptively set an appropriate noise level based on the amplitude of the signal in the original sound.\n\nThe ratio of the signal-to-noise level is called Signal-to-Noise Ratio (SNR). The signal-to-noise ratio (SNR) is expressed as the ratio of the actual amplitude to the logarithm of the signal's amplitude and is calculated by the following formula.\n\n$$\nSNR = 20\\log_{10}\\frac{A_{signal}}{A_{noise}}\n$$\n\nThe larger this amount is, the stronger the signal, or the more audible the sound is, and it is expressed in dB (decibel), where 0dB means that the strength of the signal is balanced with the strength of the noise, when it is negative, the noise is stronger, and when it is positive, the signal is stronger.\n\nThere may be several ways to estimate the strength of the signal sound, but in this case we will treat the absolute maximum of the amplitude in the clip as the amplitude of the signal.\n\nReference: [任意のSignal-to-Noise比の音声波形をPythonで作ろう！](https://engineering.linecorp.com/ja/blog/voice-waveform-arbitrary-signal-to-noise-ratio-python/) (Japanese)"},{"metadata":{"id":"W1Eve45uWpb_","trusted":true},"cell_type":"code","source":"class GaussianNoiseSNR(AudioTransform):\n    def __init__(self, always_apply=False, p=0.5, min_snr=5.0, max_snr=20.0, **kwargs):\n        super().__init__(always_apply, p)\n\n        self.min_snr = min_snr\n        self.max_snr = max_snr\n\n    def apply(self, y: np.ndarray, **params):\n        snr = np.random.uniform(self.min_snr, self.max_snr)\n        a_signal = np.sqrt(y ** 2).max()\n        a_noise = a_signal / (10 ** (snr / 20))\n\n        white_noise = np.random.randn(len(y))\n        a_white = np.sqrt(white_noise ** 2).max()\n        augmented = (y + white_noise * 1 / a_white * a_noise).astype(y.dtype)\n        return augmented","execution_count":null,"outputs":[]},{"metadata":{"id":"fBwP8BXpdHbh","executionInfo":{"status":"ok","timestamp":1605227811652,"user_tz":-540,"elapsed":1230,"user":{"displayName":"Hidehisa Arai","photoUrl":"","userId":"12684386091436935327"}},"outputId":"177759fe-6d54-4232-c643-0bb6e0c9d875","trusted":true},"cell_type":"code","source":"transform = GaussianNoiseSNR(always_apply=True, min_snr=5, max_snr=20)\ny_gaussian_snr = transform(y)\nAudio(y_gaussian_snr, rate=sr)","execution_count":null,"outputs":[]},{"metadata":{"id":"yxEG_wi3dS8g","executionInfo":{"status":"ok","timestamp":1605227834801,"user_tz":-540,"elapsed":970,"user":{"displayName":"Hidehisa Arai","photoUrl":"","userId":"12684386091436935327"}},"outputId":"b14349ed-f71a-43d1-a0d6-389a1e592263","trusted":true},"cell_type":"code","source":"librosa.display.waveplot(y_gaussian_snr, sr=sr);","execution_count":null,"outputs":[]},{"metadata":{"id":"UHdjqyWldZgb","executionInfo":{"status":"ok","timestamp":1605227836588,"user_tz":-540,"elapsed":1189,"user":{"displayName":"Hidehisa Arai","photoUrl":"","userId":"12684386091436935327"}},"outputId":"ae80421a-19c8-454a-a997-6fd55ea5ae7a","trusted":true},"cell_type":"code","source":"melspec = librosa.power_to_db(librosa.feature.melspectrogram(y_gaussian_snr, sr=sr, n_mels=128))\nlibrosa.display.specshow(melspec, sr=sr, x_axis=\"time\", y_axis=\"mel\")\nplt.colorbar();","execution_count":null,"outputs":[]},{"metadata":{"id":"VSBjpWhQdtKx"},"cell_type":"markdown","source":"### PinkNoiseSNR\n\nGaussian Noiseはいわゆる白色雑音(white noise)で全周波数帯にノイズをかけるものでした。ここで紹介するPink Noiseは低周波数帯から低周波数帯にかけて徐々にノイズの強さが減少するようなノイズのことをさします。自然界に存在するノイズはこのようなノイズであるとされます。\n\nなお、白色雑音以外のノイズはcolored noiseと呼ばれ、他にはブラウンノイズ、ブルーノイズなど様々なノイズが提案されています。\n\nPink Noiseを発生させるために`colorednoise`というライブラリを用いるのですがその名前の由来は上記のようなものになります。\n\nなお、白色雑音の時は最初に強さを直接指定するような実装を紹介しましたが、今回はいきなり強さをSNRベースで適応的に決定できる実装を紹介します。\n\n参考・出典: [Wikipedia カラードノイズ](https://ja.wikipedia.org/wiki/%E3%82%AB%E3%83%A9%E3%83%BC%E3%83%89%E3%83%8E%E3%82%A4%E3%82%BA#:~:text=%E3%82%AB%E3%83%A9%E3%83%BC%E3%83%89%E3%83%8E%E3%82%A4%E3%82%BA%EF%BC%88%E8%8B%B1%3A%20colors%20of,%E3%81%9D%E3%81%AE%E7%89%B9%E6%80%A7%E3%82%82%E5%A4%A7%E3%81%8D%E3%81%8F%E7%95%B0%E3%81%AA%E3%82%8B%E3%80%82), [`colorednoise`](https://github.com/felixpatzelt/colorednoise)\n\nGaussian Noise is a so-called white noise, which is a noise over the whole frequency range. Pink noise, which we introduce here, is noise with a gradual decrease in noise intensity from low frequency to low frequency bands. The noise in the natural world is said to be such noise.\n\nThe noise other than white noise is called \"colored noise\", and various noises such as brown noise and blue noise have been proposed.\n\nThe `colorednoise` library is used to generate pink noise, and its name comes from the above.\n\nIn the previous article, we introduced an implementation of white noise that directly specifies the intensity of the noise.\n\nReference: [Wikipedia Colors of noise](https://en.wikipedia.org/wiki/Colors_of_noise), [`colorednoise`](https://github.com/felixpatzelt/colorednoise)"},{"metadata":{"id":"SCRfJsg5dc_m","trusted":true},"cell_type":"code","source":"import colorednoise as cn\n\n\nclass PinkNoiseSNR(AudioTransform):\n    def __init__(self, always_apply=False, p=0.5, min_snr=5.0, max_snr=20.0, **kwargs):\n        super().__init__(always_apply, p)\n\n        self.min_snr = min_snr\n        self.max_snr = max_snr\n\n    def apply(self, y: np.ndarray, **params):\n        snr = np.random.uniform(self.min_snr, self.max_snr)\n        a_signal = np.sqrt(y ** 2).max()\n        a_noise = a_signal / (10 ** (snr / 20))\n\n        pink_noise = cn.powerlaw_psd_gaussian(1, len(y))\n        a_pink = np.sqrt(pink_noise ** 2).max()\n        augmented = (y + pink_noise * 1 / a_pink * a_noise).astype(y.dtype)\n        return augmented","execution_count":null,"outputs":[]},{"metadata":{"id":"1WWOrJNfgpg6","executionInfo":{"status":"ok","timestamp":1605227858728,"user_tz":-540,"elapsed":1399,"user":{"displayName":"Hidehisa Arai","photoUrl":"","userId":"12684386091436935327"}},"outputId":"6c6626a2-7c7e-4585-fda5-88e9f7af8580","trusted":true},"cell_type":"code","source":"transform = PinkNoiseSNR(always_apply=True, min_snr=5.0, max_snr=20.0)\ny_pink_noise = transform(y)\nAudio(y_pink_noise, rate=sr)","execution_count":null,"outputs":[]},{"metadata":{"id":"7XPBqAaqgzh6","executionInfo":{"status":"ok","timestamp":1605227872295,"user_tz":-540,"elapsed":1041,"user":{"displayName":"Hidehisa Arai","photoUrl":"","userId":"12684386091436935327"}},"outputId":"8487eb8e-69f3-4736-e1b7-4ad4f3c5817b","trusted":true},"cell_type":"code","source":"librosa.display.waveplot(y_pink_noise, sr=sr);","execution_count":null,"outputs":[]},{"metadata":{"id":"QRmTDA6Ig4ko","executionInfo":{"status":"ok","timestamp":1605227876079,"user_tz":-540,"elapsed":1300,"user":{"displayName":"Hidehisa Arai","photoUrl":"","userId":"12684386091436935327"}},"outputId":"6e515b92-c5ce-4958-b240-fb00d5cd61b8","trusted":true},"cell_type":"code","source":"melspec = librosa.power_to_db(librosa.feature.melspectrogram(y_pink_noise, sr=sr, n_mels=128))\nlibrosa.display.specshow(melspec, sr=sr, x_axis=\"time\", y_axis=\"mel\")\nplt.colorbar();","execution_count":null,"outputs":[]},{"metadata":{"id":"nYeqSNWn4Npc"},"cell_type":"markdown","source":"### PitchShift\n\nPitchShiftは音のピッチ(高低)に関する調整を施すData Augmentationで、効果として聞こえる音が高く/低くなります。メルスペクトログラム上では、パターンのある周波数帯が上または下にズレます。\n\nPitchShiftはリサンプリングを行うため今まで紹介したData Augmentationと比べると時間がかかるほか、ピッチを変えすぎると音割れを起こしてしまうこともあるため注意が必要です。\n\n参考・出典: [librosa.effects.pitch_shift](http://man.hubwiz.com/docset/LibROSA.docset/Contents/Resources/Documents/generated/librosa.effects.pitch_shift.html)\n\nPitchShift is a data augmentation that adjusts the pitch of the sound (high and low), making the sound heard as an effect higher/lower. On the Meru spectrogram, certain frequency bands in the pattern will be shifted up or down.\n\nPitchShift takes more time than the previously introduced Data Augmentation because of resampling, and you should be careful not to change the pitch too much because it may cause the sound to crack.\n\nReference: [librosa.effects.pitch_shift](http://man.hubwiz.com/docset/LibROSA.docset/Contents/Resources/Documents/generated/librosa.effects.pitch_shift.html)"},{"metadata":{"id":"oJl0bDSrg8gE","trusted":true},"cell_type":"code","source":"class PitchShift(AudioTransform):\n    def __init__(self, always_apply=False, p=0.5, max_steps=5, sr=32000):\n        super().__init__(always_apply, p)\n\n        self.max_steps = max_steps\n        self.sr = sr\n\n    def apply(self, y: np.ndarray, **params):\n        n_steps = np.random.randint(-self.max_steps, self.max_steps)\n        augmented = librosa.effects.pitch_shift(y, sr=self.sr, n_steps=n_steps)\n        return augmented","execution_count":null,"outputs":[]},{"metadata":{"id":"cige7FibBzfc","executionInfo":{"status":"ok","timestamp":1605227884518,"user_tz":-540,"elapsed":2622,"user":{"displayName":"Hidehisa Arai","photoUrl":"","userId":"12684386091436935327"}},"outputId":"3d26f79a-01bf-43f1-cbd7-cdcb8bc7fe9c","trusted":true},"cell_type":"code","source":"transform = PitchShift(always_apply=True, max_steps=5, sr=sr)\ny_pitch_shift = transform(y)\nAudio(y_pitch_shift, rate=sr)","execution_count":null,"outputs":[]},{"metadata":{"id":"U8P6SIdbCE_S","executionInfo":{"status":"ok","timestamp":1605227896795,"user_tz":-540,"elapsed":1015,"user":{"displayName":"Hidehisa Arai","photoUrl":"","userId":"12684386091436935327"}},"outputId":"ed7e06b9-b129-4251-b957-ba3b0bffa3a0","trusted":true},"cell_type":"code","source":"librosa.display.waveplot(y_pitch_shift, sr=sr);","execution_count":null,"outputs":[]},{"metadata":{"id":"vju6YS9KCcFd","executionInfo":{"status":"ok","timestamp":1605227900050,"user_tz":-540,"elapsed":1281,"user":{"displayName":"Hidehisa Arai","photoUrl":"","userId":"12684386091436935327"}},"outputId":"2a519e21-869a-48a4-d3e4-2dc4586d4bf2","trusted":true},"cell_type":"code","source":"melspec = librosa.power_to_db(librosa.feature.melspectrogram(y_pitch_shift, sr=sr, n_mels=128))\nlibrosa.display.specshow(melspec, sr=sr, x_axis=\"time\", y_axis=\"mel\")\nplt.colorbar();","execution_count":null,"outputs":[]},{"metadata":{"id":"Om_mo8Z0D7QA"},"cell_type":"markdown","source":"### TimeStretch\n\nTimeStretchは元の音を時間的に引き延ばしたり圧縮したりします。結果として音のスピードが速くなったり遅くなったりします。\n\nTimeStretchも時間がかかるData Augmentationです。\n\n参考・出典: [librosa.effects.time_stretch](http://man.hubwiz.com/docset/LibROSA.docset/Contents/Resources/Documents/generated/librosa.effects.time_stretch.html)\n\nTimeStretch stretches and compresses the original sound in time. As a result, the speed of the sound may be increased or decreased.\n\nTimeStretch is another time-consuming form of data augmentation.\n\nReference: [librosa.effects.time_stretch](http://man.hubwiz.com/docset/LibROSA.docset/Contents/Resources/Documents/generated/librosa.effects.time_stretch.html)"},{"metadata":{"id":"5oxJhXeACjNY","trusted":true},"cell_type":"code","source":"class TimeStretch(AudioTransform):\n    def __init__(self, always_apply=False, p=0.5, max_rate=1.2):\n        super().__init__(always_apply, p)\n\n        self.max_rate = max_rate\n\n    def apply(self, y: np.ndarray, **params):\n        rate = np.random.uniform(0, self.max_rate)\n        augmented = librosa.effects.time_stretch(y, rate)\n        return augmented","execution_count":null,"outputs":[]},{"metadata":{"id":"av8Tg4pmF2F6","executionInfo":{"status":"ok","timestamp":1605227916454,"user_tz":-540,"elapsed":1013,"user":{"displayName":"Hidehisa Arai","photoUrl":"","userId":"12684386091436935327"}},"outputId":"7075ad49-7949-4642-ac09-2e991e7d3524","trusted":true},"cell_type":"code","source":"transform = TimeStretch(always_apply=True, max_rate=2.0)\ny_time_stretch = transform(y)\nAudio(y_time_stretch, rate=sr)","execution_count":null,"outputs":[]},{"metadata":{"id":"gPKJa-g-Gc50","executionInfo":{"status":"ok","timestamp":1605227926180,"user_tz":-540,"elapsed":802,"user":{"displayName":"Hidehisa Arai","photoUrl":"","userId":"12684386091436935327"}},"outputId":"26d8a1ef-9278-4b02-e0ec-a19df5a26bbd","trusted":true},"cell_type":"code","source":"librosa.display.waveplot(y_time_stretch, sr=sr);","execution_count":null,"outputs":[]},{"metadata":{"id":"3TMV2hlMGmMW","executionInfo":{"status":"ok","timestamp":1605227929996,"user_tz":-540,"elapsed":1203,"user":{"displayName":"Hidehisa Arai","photoUrl":"","userId":"12684386091436935327"}},"outputId":"23824a0e-e479-4d49-f857-f840bb54b691","trusted":true},"cell_type":"code","source":"melspec = librosa.power_to_db(librosa.feature.melspectrogram(y_time_stretch, sr=sr, n_mels=128))\nlibrosa.display.specshow(melspec, sr=sr, x_axis=\"time\", y_axis=\"mel\")\nplt.colorbar();","execution_count":null,"outputs":[]},{"metadata":{"id":"zMfK3EeOHeSf"},"cell_type":"markdown","source":"### TimeShift\n\nTimeShiftは時間的に音イベントをずらすような操作です。ズラした結果、元の音クリップの長さからはみ出した部分の扱いに関しては、前(または後ろ)に持っていってくっつける、無視して捨ててしまう、などのやり方があります。\n\n参考・出典: [Data Augmentation for Audio](https://medium.com/@makcedward/data-augmentation-for-audio-76912b01fdf6)\n\nTimeShift is such an operation that shifts a sound event in time. As for dealing with the part of the sound clip that goes out of the original length as a result of shifting, you can bring it forward (or backward) and stick it to the front (or backward), or ignore it and throw it away.\n\nReference: [Data Augmentation for Audio](https://medium.com/@makcedward/data-augmentation-for-audio-76912b01fdf6)"},{"metadata":{"id":"wFy4tpvwGt4p","trusted":true},"cell_type":"code","source":"class TimeShift(AudioTransform):\n    def __init__(self, always_apply=False, p=0.5, max_shift_second=2, sr=32000, padding_mode=\"replace\"):\n        super().__init__(always_apply, p)\n    \n        assert padding_mode in [\"replace\", \"zero\"], \"`padding_mode` must be either 'replace' or 'zero'\"\n        self.max_shift_second = max_shift_second\n        self.sr = sr\n        self.padding_mode = padding_mode\n\n    def apply(self, y: np.ndarray, **params):\n        shift = np.random.randint(-self.sr * self.max_shift_second, self.sr * self.max_shift_second)\n        augmented = np.roll(y, shift)\n        if self.padding_mode == \"zero\":\n            if shift > 0:\n                augmented[:shift] = 0\n            else:\n                augmented[shift:] = 0\n        return augmented","execution_count":null,"outputs":[]},{"metadata":{"id":"bbfhn2iZKQ3B","executionInfo":{"status":"ok","timestamp":1605228029221,"user_tz":-540,"elapsed":1179,"user":{"displayName":"Hidehisa Arai","photoUrl":"","userId":"12684386091436935327"}},"outputId":"cbe1acb8-8d9b-4442-b72e-cd6701a63901","trusted":true},"cell_type":"code","source":"transform = TimeShift(always_apply=True, max_shift_second=4, sr=sr)\ny_time_shifted = transform(y)\nAudio(y_time_shifted, rate=sr)","execution_count":null,"outputs":[]},{"metadata":{"id":"l5pSTeM1Ka0U","executionInfo":{"status":"ok","timestamp":1605228042384,"user_tz":-540,"elapsed":931,"user":{"displayName":"Hidehisa Arai","photoUrl":"","userId":"12684386091436935327"}},"outputId":"50d61748-babe-4cd4-8b06-7f8b68278454","trusted":true},"cell_type":"code","source":"librosa.display.waveplot(y_time_shifted, sr=sr);","execution_count":null,"outputs":[]},{"metadata":{"id":"xy2WrJu2Kfr1","executionInfo":{"status":"ok","timestamp":1605228046872,"user_tz":-540,"elapsed":1235,"user":{"displayName":"Hidehisa Arai","photoUrl":"","userId":"12684386091436935327"}},"outputId":"3e08340d-3988-41eb-ee45-532842384551","trusted":true},"cell_type":"code","source":"melspec = librosa.power_to_db(librosa.feature.melspectrogram(y_time_shifted, sr=sr, n_mels=128))\nlibrosa.display.specshow(melspec, sr=sr, x_axis=\"time\", y_axis=\"mel\")\nplt.colorbar();","execution_count":null,"outputs":[]},{"metadata":{"id":"zSPbzky-K-p7"},"cell_type":"markdown","source":"### VolumeControl\n\nVolumeControlは音量を調節します。音の認識には音量そのものよりSNRが影響するという話を以前紹介したかと思いますが、音量を調節することでメルスペクトログラムにはごく僅かな変化が生じます。また、音量をサイン曲線、コサイン曲線などに合わせて調節する、などはメルスペクトログラムには大きな変化をもたらすため有用です。\n\nVolumeControl controls the volume. I think I mentioned before that the SNR has more influence on sound perception than the volume itself, but adjusting the volume causes a very small change in the mel spectrogram. Adjusting the volume according to a sine curve, cosine curve, etc. is also useful because it causes a big change in the mel spectrogram."},{"metadata":{"id":"uYceRHv7Ku9M","trusted":true},"cell_type":"code","source":"class VolumeControl(AudioTransform):\n    def __init__(self, always_apply=False, p=0.5, db_limit=10, mode=\"uniform\"):\n        super().__init__(always_apply, p)\n\n        assert mode in [\"uniform\", \"fade\", \"fade\", \"cosine\", \"sine\"], \\\n            \"`mode` must be one of 'uniform', 'fade', 'cosine', 'sine'\"\n\n        self.db_limit= db_limit\n        self.mode = mode\n\n    def apply(self, y: np.ndarray, **params):\n        db = np.random.uniform(-self.db_limit, self.db_limit)\n        if self.mode == \"uniform\":\n            db_translated = 10 ** (db / 20)\n        elif self.mode == \"fade\":\n            lin = np.arange(len(y))[::-1] / (len(y) - 1)\n            db_translated = 10 ** (db * lin / 20)\n        elif self.mode == \"cosine\":\n            cosine = np.cos(np.arange(len(y)) / len(y) * np.pi * 2)\n            db_translated = 10 ** (db * cosine / 20)\n        else:\n            sine = np.sin(np.arange(len(y)) / len(y) * np.pi * 2)\n            db_translated = 10 ** (db * sine / 20)\n        augmented = y * db_translated\n        return augmented","execution_count":null,"outputs":[]},{"metadata":{"id":"wVbLH1v5YWGn","executionInfo":{"status":"ok","timestamp":1605228055829,"user_tz":-540,"elapsed":819,"user":{"displayName":"Hidehisa Arai","photoUrl":"","userId":"12684386091436935327"}},"outputId":"8d04218b-b6c8-4f1a-b349-7a6cb67bfeea","trusted":true},"cell_type":"code","source":"transform = VolumeControl(always_apply=True, mode=\"sine\")\ny_volume_controlled = transform(y)\nAudio(y_volume_controlled, rate=sr)","execution_count":null,"outputs":[]},{"metadata":{"id":"Bd7VMuDkYvTM","executionInfo":{"status":"ok","timestamp":1605228076549,"user_tz":-540,"elapsed":803,"user":{"displayName":"Hidehisa Arai","photoUrl":"","userId":"12684386091436935327"}},"outputId":"84919a4b-b295-45eb-ea56-dfc2a70ac2af","trusted":true},"cell_type":"code","source":"librosa.display.waveplot(y_volume_controlled, sr=sr);","execution_count":null,"outputs":[]},{"metadata":{"id":"b-E_W2scZs72","executionInfo":{"status":"ok","timestamp":1605228080457,"user_tz":-540,"elapsed":1587,"user":{"displayName":"Hidehisa Arai","photoUrl":"","userId":"12684386091436935327"}},"outputId":"37c3f271-8047-41b2-be31-af564e1935e3","trusted":true},"cell_type":"code","source":"melspec = librosa.power_to_db(librosa.feature.melspectrogram(y_volume_controlled, sr=sr, n_mels=128))\nlibrosa.display.specshow(melspec, sr=sr, x_axis=\"time\", y_axis=\"mel\")\nplt.colorbar();","execution_count":null,"outputs":[]},{"metadata":{"id":"jhrLOovWd-oE"},"cell_type":"markdown","source":"### 組み合わせて使う (Combination)"},{"metadata":{"id":"tgwxarNUbd_d","executionInfo":{"status":"ok","timestamp":1605228107415,"user_tz":-540,"elapsed":1954,"user":{"displayName":"Hidehisa Arai","photoUrl":"","userId":"12684386091436935327"}},"outputId":"28e60f50-4e8e-4b9d-b12f-08f3ba21bc9b","trusted":true},"cell_type":"code","source":"transform = Compose([\n  OneOf([\n    GaussianNoiseSNR(min_snr=10),\n    PinkNoiseSNR(min_snr=10)\n  ]),\n  PitchShift(max_steps=2, sr=sr),\n  TimeStretch(),\n  TimeShift(sr=sr),\n  VolumeControl(mode=\"sine\")\n])\ny_composed = transform(y)\nAudio(y_composed, rate=sr)","execution_count":null,"outputs":[]},{"metadata":{"id":"SAjAgeK3tvTL","executionInfo":{"status":"ok","timestamp":1605228116210,"user_tz":-540,"elapsed":1051,"user":{"displayName":"Hidehisa Arai","photoUrl":"","userId":"12684386091436935327"}},"outputId":"5b692a1e-2fec-4969-ea41-1043bb72e3c6","trusted":true},"cell_type":"code","source":"librosa.display.waveplot(y_composed, sr=sr);","execution_count":null,"outputs":[]},{"metadata":{"id":"hVZ44fZ0uSUC","executionInfo":{"status":"ok","timestamp":1605228125744,"user_tz":-540,"elapsed":1426,"user":{"displayName":"Hidehisa Arai","photoUrl":"","userId":"12684386091436935327"}},"outputId":"f73465bb-e9b0-4358-e96f-0c47611aabb0","trusted":true},"cell_type":"code","source":"melspec = librosa.power_to_db(librosa.feature.melspectrogram(y_composed, sr=sr, n_mels=128))\nlibrosa.display.specshow(melspec, sr=sr, x_axis=\"time\", y_axis=\"mel\")\nplt.colorbar();","execution_count":null,"outputs":[]},{"metadata":{"id":"0M3kQvs-9Cfd"},"cell_type":"markdown","source":"## [WIP] スペクトログラム/メルスペクトログラムに対するData Augmentation(Data Augmentation for waveform)\n\nWork in progress..."},{"metadata":{},"cell_type":"markdown","source":"## More to come. Stay tuned!"},{"metadata":{"id":"Ioomr2dPuWb7","trusted":false},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}