{"cells":[{"metadata":{},"cell_type":"markdown","source":"One of the most used method for audio based classification is to convert 1D audio signal to 2D image representation and use well established computer vision techniques. Following this widely used and successful method, this kernel builds an \"interesting\" image dataset using audio dataset.\n\n\n**Idea:** Transform the audio data to get one channel image data like spectrogram. Take three such transforms and stack them together to form 3 channel image.\n\nThe three transforms that this notebook uses are:\n- STFT based spectrogram\n- Log of the spectrogram\n- MFCC based spectrogram\n\n_Sneak Peak of the dataset_\n\n![img](https://i.imgur.com/z8P7ByC.png)\n\nOverview: \n\n* Generate multiple image datasets using different values of the hyperparameters - `n_fft` and `hop_length`. \n* Since we are creating multiple dataset to experiment with, dataset version control can be useful. We will save the dataset as W&B artifacts. \n\n_Sneak Peak of our dataset version control_\n\n![img](https://i.imgur.com/1cqGL0B.png)\n\nIn a different kernel we shall consume these artifacts to train neural network based models. "},{"metadata":{},"cell_type":"markdown","source":"# Imports and Setups"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport csv\nimport numpy as np\nimport pandas as pd\nfrom pathlib import Path\nfrom PIL import Image\n\nimport librosa as lb \nimport librosa.display\nimport matplotlib.pyplot as plt\nimport IPython.display as ipd\n\nfrom skimage.transform import resize\nfrom scipy import stats\n\nimport wandb\nwandb.login()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Simple EDA\n\nLet's briefly investigate `csv` files. We have two annotation files - `train_tp.csv` and `train_fp.csv`.\n\nFor clarity or more information on `train_tp` vs `train_fp` check out this [disussion thread](https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/197866). We are discarding `train_fp.csv` for now.\n\nFor more on `submission.csv` check out this [discussion thread](https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/200757)."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"data_dir = '/kaggle/input/rfcx-species-audio-detection/'\ntrain_tp = pd.read_csv(os.path.join(data_dir, 'train_tp.csv'))\ntrain_fp = pd.read_csv(os.path.join(data_dir, 'train_fp.csv'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### A peek into the file."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_tp.head(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_fp.head(2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### General description"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_tp.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_fp.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Number of Classes"},{"metadata":{"trusted":true},"cell_type":"code","source":"# number of classes.\nprint('Classes: ', sorted(train_tp.species_id.unique()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# unique `songtype_id`\nsorted(train_tp.songtype_id.unique())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Pointers:\n\n* There are 24 species of birds and frogs together. Thus 24 classes.\n* `songtype_id` is the same species using different frequencies as annotated. \n* There are 1216 rows in `train_tp.csv` and 7781 rows in `train_fp.csv`.\n\nLet us see the number of training audio files.\n\n### Number of train and test audio records"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_folder= Path(data_dir+'train')\ntest_folder = Path(data_dir+'test')\n\ntrain_file_path = list(map(str, list(train_folder.glob('*.flac'))))\ntest_file_path = list(map(str, list(test_folder.glob('*.flac'))))\n\nprint('Number of audio files to train: {} and test: {}'.format(len(train_file_path), len(test_file_path)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The numbers don't add up and thus a closer look at the `csv` files is required. "},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Number of unique true positive annotated files: ', len(train_tp.recording_id.unique()))\nprint('Number of unique false positive annotated files: ', len(train_fp.recording_id.unique()))\n\nprint('TP + FP: ', 1132+3958)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The reason for tp+fp to not add up to the number of audion files is that there are some files annotated with tp as well as fp."},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Number of files present in both tp and fp: ', len(set(train_tp.recording_id.unique()).intersection(set(train_fp.recording_id.unique()))))\nprint('Total number of files: ', 5090-363)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Save the `csv` files as W&B Artifacts"},{"metadata":{"trusted":true},"cell_type":"code","source":"# initialize a W&B run\nrun = wandb.init(project='rainforest', job_type='load_dataset')\n\n# create an artifact to add file(s) and meaningful description.\nartifact = wandb.Artifact('csv_reference', \n                          type='dataset', \n                          description='These csv files contain contain both true positive and false positive annotations.',\n                          metadata={'type': 'csv'})\n    \nartifact.add_file(data_dir+'train_tp.csv')\nartifact.add_file(data_dir+'train_fp.csv')\n\n# Save the artifact version to W&B and mark it as the output of this run\nrun.log_artifact(artifact)\n    \nrun.join()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Audio to Image Transformations\n\nWe will quickly look at the standard audio to image transformation techniques. They will be used to generate the dataset to train our model. \n\nThe transformations are based on this [Kaggle kernel](https://www.kaggle.com/samcantor9/getting-started-with-rainforest-audio-data) by [Sam Cantor](https://www.kaggle.com/samcantor9)."},{"metadata":{"trusted":true},"cell_type":"code","source":"training_files = train_tp.recording_id.unique()\ntraining_files","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_recording_id = np.random.choice(training_files, 1)[0]\nsample_path = [path for path in train_file_path if sample_recording_id in path][0]\n\nSR = 48000\nsignal, sr = lb.load(sample_path, sr=SR)\nlb.display.waveplot(signal, sr=SR)\nplt.xlabel('Time')\nplt.ylabel('Amplitude')\nplt.show()\n\nipd.Audio(sample_path)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# which specie(s) does the sample audio file belong\ntrain_tp.loc[train_tp['recording_id'] == sample_recording_id]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 1. STFT based spectrogram"},{"metadata":{"trusted":true},"cell_type":"code","source":"n_fft = 2048 # number of samples per FFT (the duration of each slice)\nhop_length = 512 # shift\n\nstft = lb.core.stft(signal, hop_length=hop_length, n_fft=n_fft)\n\nspectrogram = np.abs(stft)\n\nlb.display.specshow(spectrogram, sr=sr, hop_length=hop_length)\nplt.xlabel('Time') \nplt.ylabel('Frequency')\nclb = plt.colorbar()\nclb.set_label('Amplitude')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2. STFT based log spectrogram\n\nSImply convert the spectrogram to log scale."},{"metadata":{"trusted":true},"cell_type":"code","source":"log_spectrogram = lb.amplitude_to_db(spectrogram)\n\nlb.display.specshow(log_spectrogram, sr=sr, hop_length=hop_length)\nplt.xlabel('Time')\nplt.ylabel('Frequency')\nclb = plt.colorbar()\nclb.set_label('Amplitude')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3. Mel Frequency Ceptral Coefficients(MFCCs)"},{"metadata":{"trusted":true},"cell_type":"code","source":"mel_spectrogram = lb.feature.melspectrogram(signal, n_fft=n_fft, hop_length=hop_length, sr=sr)\n\nlog_mel_spectrogram = lb.amplitude_to_db(mel_spectrogram)\n\nlb.display.specshow(log_mel_spectrogram, sr=sr, hop_length=hop_length)\nplt.xlabel('Time')\nplt.ylabel('MFCC')\nclb = plt.colorbar()\nclb.set_label('Volume') \nplt.show()  ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# IDEA: Stack all three transformations such that we get a standard 3 channel image."},{"metadata":{"trusted":true},"cell_type":"code","source":"spectrogram = resize(spectrogram, (224, 400))\nlog_spectrogram = resize(log_spectrogram, (224, 400))\nlog_mel_spectrogram = resize(log_mel_spectrogram, (224, 400))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"img = np.stack((spectrogram, log_spectrogram, log_mel_spectrogram), axis=-1)\nprint(img.shape)\n\nplt.imshow(img);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# normalize image\nnorm_img = stats.zscore(img)\nplt.imshow(norm_img);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Dataset Creation and W&B Artifacts\n\nLearn more about the artifacts through this easy to understand [Colab notebook](https://colab.research.google.com/github/wandb/examples/blob/master/colabs/wandb-artifacts/Pipeline_Versioning_with_W%26B_Artifacts.ipynb)."},{"metadata":{},"cell_type":"markdown","source":"### Dataset related hyperparameters\n\nI created three variants of the dataset by changing the hyperparameters, by simply running the jupyter cells you can create a new dataset and save as artifacts. You can later use the same to download the dataset and train a model. "},{"metadata":{"trusted":true},"cell_type":"code","source":"N_FFT = 1024\nHOP_LENGTH = 512\nSR = 48000 # high sr for less rounding errors this way\nLENGTH = 10 * SR #length of slice\n\nIMG_WIDTH = 400\nIMG_HEIGHT = 224\n\nSAVE_DIR = 'kaggle/working/'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Prepare dataset and save as [W&B artifacts](https://www.wandb.com/artifacts)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# initialize a W&B run\nrun = wandb.init(project='rainforest', job_type='prepare_dataset')\n\n# declare which artifact we'll be using\nartifact_csv = run.use_artifact('wandb/rainforest/csv_reference:v0')\n\n# we can eithr download the csv file from the artifact_csv or use the one which is opened. \nrecording_ids = train_tp.recording_id.values\n\n# make a directory to save the created files\nos.makedirs(SAVE_DIR+'nfft_{}_hop_{}'.format(N_FFT, HOP_LENGTH), exist_ok=True)\nprint('Dir successfully made')\n\n# create a Artifact to save in the dataset to be used later\nartifact = wandb.Artifact('spectrogram-dataset_nfft_{}_hop_{}'.format(N_FFT, HOP_LENGTH), \n                          type='dataset', \n                          description='This dataset was generated by stacking spectrogrm ,log spectrogram and MFCC based spectrogram with n_fft value of {}\\\n                                and hop length of {}.'.format(N_FFT, HOP_LENGTH),\n                          metadata={'width': 400,\n                                    'height': 224,\n                                    'channel': 3,\n                                    'data_type': 'uint8'})\n\n# actual loop to create the dataset \nfor i, recording_id in enumerate(recording_ids):\n    # load the audio \n    file_path = [path for path in train_file_path if recording_id in path][0]\n    wav, sr = librosa.load(file_path, sr=SR)\n    \n    # get features from the train_tp.csv file\n    features = train_tp.loc[train_tp['recording_id'] == recording_id].values[0]\n    t_min = features[3] * sr\n    t_max = features[5] * sr\n    \n    # Get the postition to slice the audio\n    center = np.round((t_min + t_max) / 2)\n    beginning = center - LENGTH / 2\n    if beginning < 0:\n        beginning = 0\n    ending = beginning + LENGTH\n    \n    if ending > len(wav):\n        ending = len(wav)\n        beginning = ending - LENGTH\n        \n    wav_slice = wav[int(beginning):int(ending)]\n    \n    # spectrogram\n    stft = lb.core.stft(wav_slice, hop_length=HOP_LENGTH, n_fft=N_FFT)\n    spectrogram = np.abs(stft)\n    spectrogram = resize(spectrogram, (IMG_HEIGHT, IMG_WIDTH))\n    \n    # log_spectrogram\n    log_spectrogram = lb.amplitude_to_db(spectrogram)\n    log_spectrogram = resize(log_spectrogram, (IMG_HEIGHT, IMG_WIDTH))\n    \n    # mel_spectrogram\n    mel_spectrogram = lb.feature.melspectrogram(wav_slice, n_fft=N_FFT, hop_length=HOP_LENGTH, sr=sr)\n    log_mel_spectrogram = lb.amplitude_to_db(mel_spectrogram)\n    log_mel_spectrogram = resize(log_mel_spectrogram, (IMG_HEIGHT, IMG_WIDTH))\n    \n    # generate image by stacking three transforms \n    img = np.stack((spectrogram, log_spectrogram, log_mel_spectrogram), axis=-1)\n    \n    # normalize image\n    norm_img = stats.zscore(img)\n    #scale image to 0-1\n    norm_img = norm_img - np.min(norm_img)\n    norm_img = norm_img / np.max(norm_img)\n    # scale up to 0-255 to save in bmp format\n    norm_img = np.round(norm_img*255).astype('uint8')\n    norm_img = np.asarray(norm_img)\n    \n    # convert to PIL Image and save in bmp format\n    bmp = Image.fromarray(norm_img)\n    bmp.save(SAVE_DIR + 'nfft_{}_hop_{}/'.format(N_FFT, HOP_LENGTH) + recording_id + '_' + str(features[1]) + '_' + str(center) + '.bmp')\n    \n    if i % 100 == 0:\n        print('Processed ' + str(i) + ' train examples from ' + str(len(recording_ids)))\n\n# save the directory as an artifact\nartifact.add_dir(SAVE_DIR+'nfft_{}_hop_{}'.format(N_FFT, HOP_LENGTH))\n\n# Save the artifact version to W&B\nrun.log_artifact(artifact)\n\n# let W&B know that this run is complete\nrun.join()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}