{"cells":[{"metadata":{},"cell_type":"markdown","source":"About:\n===\n\nIn this notebook, I have discussed a way to create a **validation set (or pseudo test set)** by putting together 10-second clips from [DCASE](http://dcase.community/challenge2018/task-bird-audio-detection) to create 10-minute long clips of background noises and adding random birdcalls on top of that. You can also use some other dataset or a combination of [datasets](https://www.kaggle.com/c/birdsong-recognition/discussion/158877) to create background noises.\n\nBecause of limitation of kaggle dataset, I have downloaded the dataset from [DCASE](http://dcase.community/challenge2018/task-bird-audio-detection) on a Google Colaboratory Notebook to create 10 minutes long clips. You can take the look at the process [here](https://colab.research.google.com/drive/1TY2a7eeS1kw3RnGZorIJm_65PMvFwEih?usp=sharing) after which I uploaded the dataset on kaggle at [here](https://www.kaggle.com/gauravchopracg/preprocessed14d) and use them in this kernel to add random birdcalls of 5 seconds each.\n\nPlease take a look at the [notebook](https://colab.research.google.com/drive/1TY2a7eeS1kw3RnGZorIJm_65PMvFwEih?usp=sharing) before going through this kernel, it help you comprehend the process of creating a validation set in an easier way.\n\nLet's get started.","execution_count":null},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"import os\n\n# let's take a look at the number of 10-minute long wav files \npreprocessed_fns = os.listdir('../input/preprocessed14d')\nlen(preprocessed_fns)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# use 50% to add birdcall and remaining ones as no_call\nbackgrounds = preprocessed_fns[:47]\nlen(backgrounds)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from pydub import AudioSegment\n\ndef normalize(fn):\n    '''\n    function to read the audio\n    and set the sampling rate to 32000\n    '''\n    audio = AudioSegment.from_file(fn)\n    audio = audio.set_channels(1).set_frame_rate(32000)\n    return audio","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\n\ntrain = pd.read_csv('../input/birdsong-recognition/train.csv')\n\nnp.random.seed(0)\nmsk = np.random.randn(len(train)) < 0.7\n\n# only use 30% of the data for validation\ntrain = train[~msk] # this is validation i forget and call it train\n# less than 1 hour bird call clip to make sure we do not run out of memory\ntrain = train[train.duration <= 60]\ntrain.filename = '../input/birdsong-recognition/train_audio/' + train.ebird_code + '/' + train.filename\n# make sure we have all the classes\ntrain.ebird_code.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**We are going to place a random audio after every 5 seconds interval or sometimes two audio of length < 2.5 seconds and create a csv file for each audio to check the data distribution**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_random_audio(df=train, length=20000):\n    # list of all the classes\n    classes = df.ebird_code.unique()\n    # shuffle the classes\n    np.random.shuffle(classes)\n    # select a random class:\n    random_class = np.random.choice(classes)\n    # list of filenames from this class\n    filenames = df[df.ebird_code == random_class].filename.tolist()\n    # select a random file:\n    fn = np.random.choice(filenames)\n    # read the audio file:\n    audio = normalize(fn)\n    \n    if len(audio) <= length:\n        return fn, random_class, audio\n    else:\n        return get_random_audio(df, length)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!mkdir dataset","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nclass_check = []","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Set the random seed\nnp.random.seed(18)\nfor j in range(len(backgrounds)):\n    background = normalize('../input/preprocessed14d/'+backgrounds[j])\n    # Make background quieter\n    background = background - 20\n    # set input length of audio to be added\n    input_length = 5000\n    start = 0\n    end = start + input_length\n    file_list = []\n    class_list = []\n    seconds = []\n    for i in range(int(len(background)/input_length)):\n        # get a random audio and class to which it belongs\n        fn, random_class, audio = get_random_audio()\n        list_class = random_class\n        class_check.append(random_class)\n        list_fn = fn\n        k = np.random.randint(int(len(background)/input_length))\n        segment = int(end/1000)\n        if k == i:\n            random_class = 'no_call'\n            fn = 'preprocessed/'+backgrounds[j]\n            audio = background[start: end]\n            list_class = random_class\n            list_fn = fn\n        elif len(audio) > input_length:\n            max_offset = len(audio) - input_length\n            offset = np.random.randint(max_offset)\n            audio = audio[offset:(input_length+offset)]\n            background = background.overlay(audio, position=start)\n        elif input_length/2 > len(audio):\n            background = background.overlay(audio, position=start)\n            length = input_length - len(audio)\n            fn, random_class, audio_ = get_random_audio(length=length)\n            background = background.overlay(audio_, position=start + (input_length/2))\n            list_fn += ',' + fn\n            list_class += ',' + random_class\n        else:\n            background = background.overlay(audio, position=start)\n        \n        start = end\n        end = end + input_length\n        file_list.append(list_fn)\n        class_list.append(list_class)\n        seconds.append(segment)\n    \n    # Export new training example\n    file_handle = background.export(\"dataset/\"+backgrounds[j].split('.')[0] + \".wav\", format=\"wav\")\n    print(\"File was saved in your directory (dataset) with number of unique classes:\", len(np.unique(np.hstack(class_list))))\n    \n    pre_data = pd.DataFrame({'filename': file_list, 'classes': class_list, 'seconds':seconds})\n    pre_data.to_csv(\"dataset/\"+backgrounds[j].split('.')[0] + \".csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# sanity check\nlen(np.unique(class_check))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"new_filenames = []\nnew_classes = []\nnew_site = []\nnew_time = []","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# concatenate each dataframe to create a single csv file\nfor fn in backgrounds:\n    fn_df = pd.read_csv('dataset/'+fn.split('.')[0]+'.csv')\n    filename = fn\n    classes = fn_df.classes.str.cat(sep=' ')\n    seconds = fn_df.seconds.astype(str).str.cat(sep=' ')\n    new_filenames.append(filename)\n    new_classes.append(classes)\n    new_time.append(seconds)\n\nnew_df = pd.DataFrame({'filename':new_filenames, 'ebird_code':new_classes, 'seconds':new_time})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"new_df.head(15)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"new_df.to_csv('dataset/val_label.csv', index=None)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\n\n# 47 wav files + 47 csv files\nlen(os.listdir('dataset'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\nimport shutil\nshutil.make_archive('dataset', 'zip', 'dataset')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!rm -r dataset","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\n\n\ndef convert_bytes(num):\n    \"\"\"\n    this function will convert bytes to MB.... GB... etc\n    \"\"\"\n    for x in ['bytes', 'KB', 'MB', 'GB', 'TB']:\n        if num < 1024.0:\n            return \"%3.1f %s\" % (num, x)\n        num /= 1024.0\n\n\ndef file_size(file_path):\n    \"\"\"\n    this function will return the file size\n    \"\"\"\n    if os.path.isfile(file_path):\n        file_info = os.stat(file_path)\n        return convert_bytes(file_info.st_size)\n\n\n# Lets check the file size\nfile_path = r\"/kaggle/working/dataset.zip\"\nprint(file_size(file_path))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Thank you for making to the end. Let me know if you have any further questions.**\n\n*And, please don't forget to \"vote-up\"!*","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}