{"cells":[{"metadata":{},"cell_type":"markdown","source":"If you, as me, wanted to start with competition [Cornell Birdcall Identification](https://www.kaggle.com/c/birdsong-recognition/overview) but did not find an easy way to deal with a lot of audio files, this notebook may help you.\n\nIn this competition, we are given audio files to identify birds by their calls. Training models on them requires feature extraction. One of the common methods is MFCC. There are two issues with doing this on-the-fly: \n* It takes a lot of time to process ALL given files.\n* The extracted features do not fit into RAM.\n\nThis notebook fixes both of these issues by extracting features from ALL audio files and saving the results in separate files as standard **NumPy arrays**. The files are archived in TAR file **train_features.xz** (lzma compression) which you will need to open in the reading mode. To train your models, you can load features from those files (you can do that in batches as well). Here is a minimal example showing how to load features from the archive:\n\n     import tarfile\n     from io import BytesIO\n     \n     with tarfile.open(\"../input/mfcc-features-as-numpy-files-for-training/train_features.xz\", \"r:xz\") as tar:\n          for member in tar.getmembers():\n              np_file = tar.extractfile(member)\n              features = np.load(BytesIO(np_file.read()))\n\nThe files are named as XC\\*\\*\\*.mp3.npy, so you can load features for any audio file from the training set using its name. The link between the file names and labels (bird identifiers) can be found in **train.csv**.\n\nI will make another notebook for test audio files once they become available. Please upvote if you find this useful.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Imports","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport warnings\nimport random\n\nimport librosa\nimport librosa.display\nfrom tqdm import tqdm_notebook as tqdm\n\nimport tarfile\nfrom pathlib import Path\n\nwarnings.filterwarnings('ignore')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Extract MFCC features and save them to files","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"input_dir = Path('../input/birdsong-recognition/train_audio')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"MFCC = {\n    \"sr\": 22050, # sampling rate for loading audio\n    \"n_mfcc\": 12 # number of MFCC features per frame that can fit in HDD\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def load_audio(filename):\n    try:\n        return librosa.load(filename, sr=None)\n    except Exception as e:\n        print(f\"Cannot load '{filename}': {e}\")\n        return None","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def extract_mfcc(y, sr=22050, n_mfcc=10):\n    try:\n        return librosa.feature.mfcc(y=y, \n                                    sr=sr if sr > 0 else MFCC[\"sr\"], \n                                    n_mfcc=n_mfcc)\n    except Exception as e:\n        print(f\"Cannot extract MFCC: {e}\")\n        return None","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def parse_audio(input_dir, output_file, max_per_label=10000):\n    \n    with tarfile.open(output_file, \"w:xz\") as tar:\n    \n        sub_dirs = list(input_dir.iterdir())    \n        for sub_dir in tqdm(sub_dirs):\n\n            for i, mp3 in enumerate(sub_dir.glob(\"*.mp3\")):\n\n                if i >= max_per_label:\n                    break\n\n                ysr = load_audio(mp3)\n                if ysr is None:\n                    continue\n\n                mfcc = extract_mfcc(y=ysr[0], \n                                    sr=ysr[1], \n                                    n_mfcc=MFCC['n_mfcc'])\n                if mfcc is None:\n                    continue\n                \n                filename = Path(f\"{mp3.name}.npy\")\n                np.save(filename, mfcc)            \n                tar.add(filename)\n                filename.unlink()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"output_file = Path('train_features.xz')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"parse_audio(input_dir, output_file)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub_df = pd.read_csv('../input/birdsong-recognition/sample_submission.csv')\nsub_df.to_csv('submission.csv', index = None)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}