{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Background\n\nHere I provide code for concatenating the .wav files from each of the birdcall directories into a single larger .wav file.\n\nI decided to do so after reading the winning solution in the BirdCLEF 2019 competition. [Bird Species Identification in Soundscapes](http://ceur-ws.org/Vol-2380/paper_86.pdf)\n\nIn his write-up after the competition, Mario Lasseck describes a data augmentation technique in which he uses a single large file composed of background noise from various soundscapes. Similarly, I decided to use a similar scheme in which I concat a large wav file and pick 5 sec segments out rather than dealing with more complicated logic to account for wav files that are less than 5 seconds or (not evenly divisible into 5 second intervals).\n\nIt took me a few hours to figure out how to call ffmpeg from within Python, so I provide the code here in case it's of assistance to someone. Additionally, I've provided the dataset that was produced from this code. More details on the dataset at the links below:\n\n# Dataset\n[A to B](https://www.kaggle.com/smodad/birdcall-no-background-concat-a-b)\n\n[C to F](https://www.kaggle.com/smodad/birdcall-no-background-concat-cf)\n\n[G to M](https://www.kaggle.com/smodad/birdcall-no-background-concat-gm)\n\n[N to R](https://www.kaggle.com/smodad/birdcall-no-background-concat-nr)\n\n[S to Y](https://www.kaggle.com/smodad/birdcall-no-background-concat-sy)\n\n[Background noise](https://www.kaggle.com/smodad/birdcall-background-concat)\n\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nfrom pathlib import Path\nimport subprocess #module for launching process from within Python","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Creating the .txt file for ffmpeg to process\n\nIn order to automate the concat process, ffmpeg requires a .txt file with a path to each .wav file you wish to merge together. In order to create this .txt file, I create a list containing the name of each directory. Because the Cornell Birdcall Identification dataset is already sub-divided by eBirdcode (i.e., species), this makes it easy to create .wav files with the appropriate eBirdcode so that the data is more organized.\n\nEach .txt file created contains a path to each .wav file in the species folder so that ffmpeg will concatenate  every .wav file in the folder to a single large file that contains all of the sounds for that species. I did this because it makes the data augmentation easier as some of the clips are less than 5 seconds but we need to pull 5 second clips for the labeling as part of the competition.\n\nBelow I'm using absolute pathnames because ffmpeg was giving me trouble with relative paths. You may also notice the \"file\" before the pathname. This is because ffmpeg requires the following format for .txt files it uses for concatenation:\n    \n    file 'file1.wav'\n    file 'file2.wav'\n    ...\n\nNote: The ' ' are stated as not being necessary by ffmpeg but it's considered best practice to include them as some directories may have spaces.\n"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"code","source":"# os.listdir(directory) makes a list of all subdirectories in 'directory' (specified as the main argument)\n# Here, I placed all of the birdcall folders (aldfly, ... , yetvir) in the directory 'call'\ndir_list = os.listdir('call')\n\n\nfor directory in dir_list:\n    filename_list = os.listdir('./call/' + directory) # Make a list of filenames for each sub-directory in 'call'\n    file_object = open(directory + '.txt', 'w') # Create a .txt file named according to the species\n    with open('./' + directory + '.txt', 'w') as filehandle:\n        for filename in filename_list:\n            # This creates a .txt file where each filename is listed per line for ffmpeg to process\n            filehandle.write(\"file 'D:\\\\Kaggle\\\\birdcall\\\\call\\\\\"  + directory + \"\\\\\" + \"%s\" % filename + \"'\\n\")\n    file_object.close()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Calling ffmpeg using subprocess.run()\n\nHere I create a string first and then pass that string to subprocess.run() which sends the ffmpeg command to the system. When *shell=True*, subprocess.run() will use the first argument supplied as a string to pass directly to the OS. *check=True* will raise an exception if the process raises an exception on exit. Fortunately, subprocess.run() will wait for the process to end before creating a new process, so you can run it in the background without fear of memory problems.\n\nThe command line string is of the form:\n\n    ffmpeg -f concat -safe 0 -i path_to/input_file.txt -c copy path_to/concat_file.wav\n\nThe run-time on my local machine (8 cores) was around 30 minutes using the CPU.\n\nNote: Not all audio file formats work properly with ffmpeg. More information on using ffmpeg to concat audio files can be found here: https://trac.ffmpeg.org/wiki/Concatenate"},{"metadata":{"trusted":true},"cell_type":"code","source":"for directory in dir_list:\n    cmdargs = \"ffmpeg -f concat -safe 0 -i \" + directory + \".txt -c copy ./call/\" + directory + \"/\" + directory + \".wav\"\n    # subprocess.run() will allow the process to end before creating a new one\n    subprocess.run(cmdargs, check=True, shell=True)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}