{"cells":[{"metadata":{},"cell_type":"markdown","source":"# V3\n\n* Added Cut-out","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# About this Notebook\n\nHello everyone , welcome to birdcall detection competition , this is a tough one , especially for me because I have never worked with audio data. As we all know how augmentations tend to boost performance in computer vision tasks , audio related tasks are no different . As discussed in forum [here](https://www.kaggle.com/c/birdsong-recognition/discussion/170959#951943) and in many threads that augmentations and audio transforms will play a major role in this competition. \n\nWhile we have a great library in the form of albumentations for Computer Vision Tasks , we don't have anything for audio data , but it will surely make our life easier if we can have something like that for audio , then we can quickly test different augs right??\n\n<font color='orange'> Well since my baseline model's F1 score remains at zero both for training and validation due to some bug and I have been scratching my head from past 4 days ,I thought to make an audio-transform on top of albumenatations in the meantime to refresh my mind for anyone to use it directly just like our computer vision transforms </font>\n\n\n![](https://miro.medium.com/max/6000/0*9DPkDJoprR07mN-1)\n\n\nI have explained the use of each audio transform seperately and finally I have added an example to use the transforms as we use the albumentations transforms directly with pytorch dataset\n\n### I hope you like this and it helps you with this competition\n\nThis notebook is inspired by Alex's notebook [here](https://www.kaggle.com/shonenkov/nlp-albumentations) where he does the same thing for Text data . Many thanks to Alex for teaching me this","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport random\nimport pandas as pd\nimport numpy as np \nimport glob\n\n#visuals\nimport matplotlib.pyplot as plt\nimport cv2\nimport IPython.display as ipd\n\n#sound\nimport librosa\n\n#albumentations core\nfrom albumentations.core.transforms_interface import DualTransform, BasicTransform","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> Note that the transforms are build in such a way that you can apply them on the audio data after reading it as a raw time series numpy array , In case you want the transform classes to read the audio file and then transform you can easily add a function in the main class below to read audio data from file path and then call this function in every other transform class to read data .\n\n<font color = 'orange'>\nThis implementation is done keeping in mind that you would want to apply these transform after cropping 5 min clips from audio which seem to work best till now as per the results shared in discussion forums from people who are the top of lb righ now</font>","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"class AudioTransform(BasicTransform):\n    \"\"\" Transform for audio task. This is the main class where we override the targets and update params function for our need\"\"\"\n\n    @property\n    def targets(self):\n        return {\"data\": self.apply}\n    \n    def update_params(self, params, **kwargs):\n        if hasattr(self, \"interpolation\"):\n            params[\"interpolation\"] = self.interpolation\n        if hasattr(self, \"fill_value\"):\n            params[\"fill_value\"] = self.fill_value\n        return params","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Time Shifting Transform\n\nShift the start time of the audio by some margin","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class TimeShifting(AudioTransform):\n    \"\"\" Do time shifting of audio \"\"\"\n    def __init__(self, always_apply=False, p=0.5):\n        super(TimeShifting, self).__init__(always_apply, p)\n        \n    def apply(self,data,**params):\n        '''\n        data : ndarray of audio timeseries\n        '''        \n        start_ = int(np.random.uniform(-80000,80000))\n        if start_ >= 0:\n            audio_time_shift = np.r_[data[start_:], np.random.uniform(-0.001,0.001, start_)]\n        else:\n            audio_time_shift = np.r_[np.random.uniform(-0.001,0.001, -start_), data[:start_]]\n        \n        return audio_time_shift","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> General Usage","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"audio_path = '../input/birdsong-recognition/train_audio/aldfly/XC181484.mp3'\n\ny,sr = librosa.load(audio_path,sr=22050)\n\nprint('Audio Intially')\nipd.Audio(y, rate=sr)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"transform = TimeShifting(p=1.0)\n\nprint('audio after transform')\nipd.Audio(transform(data=y)['data'],rate=sr)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Speed Tuning\n\nIncease or decrease the speed of the audio under consideration","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class SpeedTuning(AudioTransform):\n    \"\"\" Do speed Tuning of audio \"\"\"\n    def __init__(self, always_apply=False, p=0.5,speed_rate = None):\n        '''\n        Give Rate between (0.5,1.5) for best results\n        '''\n        super(SpeedTuning, self).__init__(always_apply, p)\n        \n        if speed_rate:\n            self.speed_rate = speed_rate\n        else:\n            self.speed_rate = np.random.uniform(0.6,1.3)\n        \n    def apply(self,data,**params):\n        '''\n        data : ndarray of audio timeseries\n        '''        \n        audio_speed_tune = cv2.resize(data, (1, int(len(data) * self.speed_rate))).squeeze()\n        if len(audio_speed_tune) < len(data) :\n            pad_len = len(data) - len(audio_speed_tune)\n            audio_speed_tune = np.r_[np.random.uniform(-0.001,0.001,int(pad_len/2)),\n                                   audio_speed_tune,\n                                   np.random.uniform(-0.001,0.001,int(np.ceil(pad_len/2)))]\n        else: \n            cut_len = len(audio_speed_tune) - len(data)\n            audio_speed_tune = audio_speed_tune[int(cut_len/2):int(cut_len/2)+len(data)]\n        \n        return audio_speed_tune","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> General Usage","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"audio_path = '../input/birdsong-recognition/train_audio/ameavo/XC133080.mp3'\n\ny,sr = librosa.load(audio_path,sr=22050)\n\nprint('Audio Intially')\nipd.Audio(y, rate=sr)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"transform = SpeedTuning(p=1.0)\n\nprint('audio after transform')\nipd.Audio(transform(data=y)['data'],rate=sr)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Stretch Audio\n\nStretch the audio file under consideration","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class StretchAudio(AudioTransform):\n    \"\"\" Do stretching of audio file\"\"\"\n    def __init__(self, always_apply=False, p=0.5 , rate = None):\n        super(StretchAudio, self).__init__(always_apply, p)\n        \n        if rate:\n            self.rate = rate\n        else:\n            self.rate = np.random.uniform(0.5,1.5)\n        \n    def apply(self,data,**params):\n        '''\n        data : ndarray of audio timeseries\n        '''        \n        input_length = len(data)\n        \n        data = librosa.effects.time_stretch(data,self.rate)\n        \n        if len(data)>input_length:\n            data = data[:input_length]\n        else:\n            data = np.pad(data, (0, max(0, input_length - len(data))), \"constant\")\n\n        return data","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> General Usage","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"audio_path = '../input/birdsong-recognition/train_audio/ameavo/XC292919.mp3'\n\ny,sr = librosa.load(audio_path,sr=22050)\n\nprint('Audio Intially')\nipd.Audio(y, rate=sr)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"transform = StretchAudio(p=1.0)\n\nprint('audio after transform')\nipd.Audio(transform(data=y)['data'],rate=sr)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Pitch Shifting\n\nShift the pitch of any audio file by number of semitones","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class PitchShift(AudioTransform):\n    \"\"\" Do time shifting of audio \"\"\"\n    def __init__(self, always_apply=False, p=0.5 , n_steps=None):\n        super(PitchShift, self).__init__(always_apply, p)\n        '''\n        nsteps here is equal to number of semitones\n        '''\n        \n        self.n_steps = n_steps\n        \n    def apply(self,data,**params):\n        '''\n        data : ndarray of audio timeseries\n        '''        \n        return librosa.effects.pitch_shift(data,sr=22050,n_steps=self.n_steps)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> General Usage","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"audio_path = '../input/birdsong-recognition/train_audio/ameavo/XC292919.mp3'\n\ny,sr = librosa.load(audio_path,sr=22050)\n\nprint('Audio Intially')\nipd.Audio(y, rate=sr)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"transform = PitchShift(p=1.0,n_steps=4)\n\nprint('audio after transform')\nipd.Audio(transform(data=y)['data'],rate=sr)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Adding Noise\n\nAdd Gaussian Noise to the audio","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class AddGaussianNoise(AudioTransform):\n    \"\"\" Do time shifting of audio \"\"\"\n    def __init__(self, always_apply=False, p=0.5):\n        super(AddGaussianNoise, self).__init__(always_apply, p)\n        \n        \n    def apply(self,data,**params):\n        '''\n        data : ndarray of audio timeseries\n        ''' \n        noise = np.random.randn(len(data))\n        data_wn = data + 0.005*noise\n        return data_wn","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> General Usage","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"audio_path = '../input/birdsong-recognition/train_audio/ameavo/XC292919.mp3'\n\ny,sr = librosa.load(audio_path,sr=22050)\n\nprint('Audio Intially')\nipd.Audio(y, rate=sr)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"transform = AddGaussianNoise(p=1.0)\n\nprint('audio after transform')\nipd.Audio(transform(data=y)['data'],rate=sr)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Add Custom Noise\n\nAdd audio from any file as cutom noise to the audio under consideration . I have audio from Free audio Tagging competition to show as an example","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class AddCustomNoise(AudioTransform):\n    \"\"\"\n    This Function allows you to add noise from any custom file you want just give path to the directory where the files\n    are stored and you are good to go.\n    \"\"\"\n    def __init__(self,file_dir, always_apply=False, p=0.5 ):\n        super(AddCustomNoise, self).__init__(always_apply, p)\n        '''\n        file_dir must be of form '.../input/.../something'\n        '''\n        \n        self.noise_files = glob.glob(file_dir+'/*')\n        \n    def apply(self,data,**params):\n        '''\n        data : ndarray of audio timeseries\n        ''' \n        nf = self.noise_files[int(np.random.uniform(0,len(self.noise_files)))]\n        \n        noise,_ = librosa.load(nf)\n        \n        if len(noise)>len(data):\n            start_ = np.random.randint(len(noise)-len(data))\n            noise = noise[start_ : start_+len(data)] \n        else:\n            noise = np.pad(noise, (0, len(data)-len(noise)), \"constant\")\n            \n        data_wn= data  + noise\n\n        return data_wn","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> General Usage","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"audio_path = '../input/birdsong-recognition/train_audio/ameavo/XC292919.mp3'\n\ny,sr = librosa.load(audio_path,sr=22050)\n\nprint('Audio Intially')\nipd.Audio(y, rate=sr)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"transform = AddCustomNoise(file_dir='../input/freesound-audio-tagging/audio_train',p=1.0)\n\nprint('audio after transform')\nipd.Audio(transform(data=y)['data'],rate=sr)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Polarity Inversion","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class PolarityInversion(AudioTransform):\n    def __init__(self, always_apply=False, p=0.5 ):\n        super(PolarityInversion, self).__init__(always_apply, p)\n        \n    def apply(self,data,**params):\n        '''\n        data : ndarray of audio timeseries\n        '''\n        return -data","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> General Usage","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"audio_path = '../input/birdsong-recognition/train_audio/ameavo/XC292919.mp3'\n\ny,sr = librosa.load(audio_path,sr=22050)\n\nprint('Audio Intially')\nipd.Audio(y, rate=sr)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"transform = PolarityInversion(p=1.0)\n\nprint('audio after transform')\nipd.Audio(transform(data=y)['data'],rate=sr)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Amplitude Gain","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class Gain(AudioTransform):\n    \"\"\"\n    Multiply the audio by a random amplitude factor to reduce or increase the volume. This\n    technique can help a model become somewhat invariant to the overall gain of the input audio.\n    \"\"\"\n\n    def __init__(self, min_gain_in_db=-12, max_gain_in_db=12, always_apply=False,p=0.5):\n        super(Gain,self).__init__(always_apply,p)\n        assert min_gain_in_db <= max_gain_in_db\n        self.min_gain_in_db = min_gain_in_db\n        self.max_gain_in_db = max_gain_in_db\n\n\n    def apply(self, data, **args):\n        amplitude_ratio = 10**(random.uniform(self.min_gain_in_db, self.max_gain_in_db)/20)\n        return data * amplitude_ratio","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"audio_path = '../input/birdsong-recognition/train_audio/ameavo/XC292919.mp3'\n\ny,sr = librosa.load(audio_path,sr=22050)\n\nprint('Audio Intially')\nipd.Audio(y, rate=sr)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"transform = Gain(p=1.0,max_gain_in_db=-800,min_gain_in_db=-900)\n\nprint('audio after transform')\nipd.Audio(transform(data=y)['data'],rate=sr)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Cut-Out\n\nCut-out is famous augmentation for images where it is used to make the model generalize better , in this , a random portion of image pixels are given a value zero\n\n![](https://miro.medium.com/max/341/0*LheHpgaVwsVw2p7L)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class CutOut(AudioTransform):\n    def __init__(self, always_apply=False, p=0.5 ):\n        super(CutOut, self).__init__(always_apply, p)\n        \n    def apply(self,data,**params):\n        '''\n        data : ndarray of audio timeseries\n        '''\n        start_ = np.random.randint(0,len(data))\n        end_ = np.random.randint(start_,len(data))\n        \n        data[start_:end_] = 0\n        \n        return data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"audio_path = '../input/birdsong-recognition/train_audio/ameavo/XC292919.mp3'\n\ny,sr = librosa.load(audio_path,sr=22050)\n\nprint('Audio Intially')\nipd.Audio(y, rate=sr)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"transform = CutOut(p=1.0)\n\nprint('audio after transform')\nipd.Audio(transform(data=y)['data'],rate=sr)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Transforms coming in v4\n\n* Frequency Masking\n* Time Masking\n* Mix-Up\n* Cut-Mix","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Getting It Together\n\nSo Far we have seen how the transforms can be used separately , now lets see how to use them with dataset class of pytorch just like we do for Computer vision Tasks\n\nI will be using dataset used in Public kernels now it's for You to decide whether you want to apply transforms before clip cropping or after clip cropping","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import albumentations\n\ndef get_train_transforms():\n    return albumentations.Compose([\n        TimeShifting(p=0.9),  # here not p=1.0 because your nets should get some difficulties\n        albumentations.OneOf([\n            AddCustomNoise(file_dir='../input/freesound-audio-tagging/audio_train', p=0.8),\n            SpeedTuning(p=0.8),\n        ]),\n        AddGaussianNoise(p=0.8),\n        PitchShift(p=0.5,n_steps=4),\n        Gain(p=0.9),\n        PolarityInversion(p=0.9),\n        StretchAudio(p=0.1),\n    ])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from torch.utils.data import Dataset\n\nclass DatasetRetriever(Dataset):\n    def __init__(\n            self,\n            file_list,\n            waveform_transforms=None):\n        self.file_list = file_list  # list of list: [file_path, ebird_code]\n        self.waveform_transforms = waveform_transforms\n\n    def __len__(self):\n        return len(self.file_list)\n\n    def __getitem__(self, idx: int):\n        wav_path, ebird_code = self.file_list[idx]\n\n        y, sr = librosa.load(wav_path)\n\n        if self.waveform_transforms:\n            y = self.waveform_transforms(data=y)['data']\n        else:\n            len_y = len(y)\n            effective_length = sr * PERIOD\n            if len_y < effective_length:\n                new_y = np.zeros(effective_length, dtype=y.dtype)\n                start = np.random.randint(effective_length - len_y)\n                new_y[start:start + len_y] = y\n                y = new_y.astype(np.float32)\n            elif len_y > effective_length:\n                start = np.random.randint(len_y - effective_length)\n                y = y[start:start + effective_length].astype(np.float32)\n            else:\n                y = y.astype(np.float32)\n                \n\n        #labels = np.zeros(len(BIRD_CODE), dtype=\"f\")\n        #labels[BIRD_CODE[ebird_code]] = 1\n\n        return {\"waveform\": y}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from pathlib import Path\ntmp_list = []\nebird_d = Path('../input/birdsong-resampled-train-audio-00/aldfly')\nfor wav_f in ebird_d.iterdir():\n    tmp_list.append([ebird_d.name, wav_f.name, wav_f.as_posix()])\n            \ntrain_wav_path = pd.DataFrame(\n    tmp_list, columns=[\"ebird_code\", \"resampled_filename\", \"file_path\"])\n\ndel tmp_list\n\ntrain_file_list = train_wav_path[[\"file_path\", \"ebird_code\"]].values.tolist()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from tqdm import tqdm\ndataset = DatasetRetriever(file_list=train_file_list, waveform_transforms=get_train_transforms())\nfor albumentation_text in tqdm(dataset, total=len(dataset)):\n    pass","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Please note that this method can easily be extended by anyone to write his own augmentations , I have just shown a way.\n\nI hope you enjoyed reading it as much as I did writing it\n\n### Thank You","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}