{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Introduction\n\nThe training data for the *Rainforest Connection Species Audio Detection* competition is presented in two formats.  The first data presentation is through a directory of .flac compressed audio files plus two .csv files containing the annotations for those files.  The audio files can be loaded, decoded, clipped and annotated using lookups to the .csv files.  The second data presentation is with a single directory of .tfrec files, where each .tfrec file contains many data records consisting of the audio signal and the annotation serialized into a [protocol buffer](https://developers.google.com/protocol-buffers/).  The .tfrec files can be directly loaded into `TFRecordDataset`s with annotations already attached.  While the first format may seem more straightforward to users unfamiliar with `TFRecordDataset`, use of the second format will reduce data preprocessing overhead and allow for more streamlined computation.  This notebook presents a data wrangling method to take advantage of the `TFRecordDataset` data presentation. "},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport tensorflow as tf\n\nfrom IPython.core.interactiveshell import InteractiveShell\nInteractiveShell.ast_node_interactivity = \"last_expr_or_assign\"\n\nfrom IPython.display import Markdown as md","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Mapping the Data Source\n\nFirst we must map the `.tfrec` datafiles to a `TFRecordDataset`."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# Create TFRecordDataset from training files\n\nTFREC_TRAIN_PATH = '/kaggle/input/rfcx-species-audio-detection/tfrecords/train'\n\ndatafiles  = os.listdir(TFREC_TRAIN_PATH)\nraw_dataset = tf.data.TFRecordDataset([os.path.join(TFREC_TRAIN_PATH,x) for x in datafiles])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The `raw_dataset` created from the `tfrec` datafiles contains a single `tf.Tensor` of type `tf.string` for each audio recording in the training dataset.  Each of these strings is a [serialized representation of the data record](https://developers.google.com/protocol-buffers/) that contains the recording_id, the audio_wav, and the label_info (as specified in the competition [data description](https://www.kaggle.com/c/rfcx-species-audio-detection/data)).   \n\n# Extract the Features\nWe must now extract the features from the serialized string into three distinct tf.Tensors using a dataset mapping."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# Add the feature labels to the raw dataset examples\n\nfeature_description = {\n    'recording_id': tf.io.FixedLenFeature([], tf.string, default_value=''),\n    'audio_wav': tf.io.FixedLenFeature([], tf.string, default_value=''),\n    'label_info': tf.io.FixedLenFeature([],tf.string,default_value=''),\n}\n\ndef _label_features(example_proto):\n    # Parse using the above dictionary\n    return tf.io.parse_single_example(example_proto, feature_description)\n\nparsed_dataset = raw_dataset.map(_label_features)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now with the `parsed_dataset` we can access the individuals features in a record."},{"metadata":{"trusted":true},"cell_type":"code","source":"next_example = next(iter(parsed_dataset))\nprint(next_example['label_info'])\nprint(next_example['recording_id'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Limitations of the Parsed Dataset\n\nIn the above `parsed_dataset` each record represents an audio recording, but as per the data specification, a recording could contain more than one training example.  The following code snippet lists the number of training examples examples for the first 10 audio recordings:"},{"metadata":{"trusted":true},"cell_type":"code","source":"def examples_per_record(example_proto): \n    label = example_proto['label_info']\n    labels = tf.strings.split(label,sep=\";\")\n    return (example_proto['recording_id'],len(labels))\n\nexample_counts = parsed_dataset.map(examples_per_record)\n\nprint(\"(Recording ID, Number of Examples)\")\naudio_recordings_with_multiple_examples=[]\nfor ex in example_counts.take(10).as_numpy_iterator():\n    print(ex)\n    if ex[1]>1:\n        audio_recordings_with_multiple_examples = audio_recordings_with_multiple_examples + [ex]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Note that some of the records displayed above contain multiple examples:"},{"metadata":{"trusted":true},"cell_type":"code","source":"audio_recordings_with_multiple_examples","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Remapping to a More Intuitive Dataset Interface\n\nWe would like a dataset in which each record is a training example with the audio signal clipped to the range [t_min,t_max] as specified in the `label_info`.  Also it will be convenient for the audio signal to be represented as `tf.float32` rather than a byte string.  Finally, we'd like to be able to access the features annotated in the label info individually using keys.  \n\nThe following code performs another dataset mapping to extract and properly format these features.  The tf.dataset method `flat_map` is used to flatten the one-to-many mapping of `parsed_dataset` records that contain multiple examples."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Map to a new dataset with eight features: \n#    recording_id (tf.string)\n#    species_id (tf.int32)\n#    songtype_id(tf.int32)\n#    t_min(tf.float32)\n#    f_min(tf.float32)\n#    t_max(tf.float32)\n#    f_max(tf.float32)\n#    is_tp(tf.bool)\n#    sample_rate(tf.int32)\n#    signal(tf.float32)\n\ndef decode_audio(audio_binary):\n    audio, sample_rate = tf.audio.decode_wav(audio_binary)\n    return tf.squeeze(audio, axis=-1), sample_rate\n\ndef clip_signal_to_interval(signal,sr,tmin,tmax):\n    sr = tf.cast(sr,tf.float32)\n    return signal[tf.cast(sr*tmin,tf.int32):tf.cast(sr*tmax,tf.int32)]\n\ndef parse_label(example_proto): \n    recording_id = example_proto['recording_id']\n    \n    label = example_proto['label_info']\n    labels = tf.strings.split(label,sep=\";\")\n    labels=tf.strings.regex_replace(labels,'\"','')\n    labels=tf.strings.strip(labels)\n    labels = tf.strings.split(labels,',')\n    labels=tf.strings.to_number(labels)\n    \n    (signal,sample_rate) = decode_audio(example_proto['audio_wav'])\n    # Create dataset from label_info\n    # Label info keys:\n    #     (species_id, songtype_id, t_min, f_min, t_max, f_max, is_tp)\n    dataset = tf.data.Dataset.from_tensor_slices(labels)\n    # Map to new dataset with recording_id and label_info keys\n    dataset = dataset.map(lambda x: {'recording_id':recording_id, \n                                     'species_id':tf.cast(x[0],tf.int32),\n                                     'songtype_id':tf.cast(x[1],tf.int32),\n                                     't_min':x[2],\n                                     'f_min':x[3],\n                                     't_max':x[4],\n                                     'f_max':x[5],\n                                     'is_tp':tf.cast(x[6],tf.bool),\n                                     'sample_rate':sample_rate,\n                                     'signal':clip_signal_to_interval(signal,sample_rate,x[2],x[4]),\n                                    })\n    \n    return dataset\n    \n    \n    \n\ndataset = parsed_dataset.flat_map(lambda x: parse_label(x))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here's a look at the first 5 records in the new dataset:"},{"metadata":{"trusted":true},"cell_type":"code","source":"k=0\nfor example in iter(dataset.take(5)):\n    print('\\n******',k,'*****\\n',example)\n    k+=1","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can also verify that the `parsed_dataset` records containing multiple examples were mapped to multiple records in `dataset`:"},{"metadata":{"trusted":true},"cell_type":"code","source":"for recording_id, nexamples in audio_recordings_with_multiple_examples:\n    print('***************{}**************'.format(recording_id))\n    examples = dataset.filter(lambda x: x['recording_id']==recording_id).take(nexamples)\n    for ex in examples:\n        print('{},{},{},{}'.format(recording_id,ex['species_id'],ex['songtype_id'],ex['t_min']))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It is also much easier for us to plot the waveforms, since the records now contain uncompressed audio signals with `tf.float32` datatype and we can easily index into the signal and sampling rate."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Plot the first 10 records\nimport matplotlib.pyplot as plt\nimport numpy as np\n\nnrecs = 10\n\nfig,ax_arr = plt.subplots(5,2,figsize=(24,36))\n\n\ndef plot_example(ax,example):\n    signal = example['signal'].numpy()\n    sample_rate = example['sample_rate'].numpy()\n    t = np.linspace(0,len(signal)/sample_rate,len(signal))\n    ax.plot(t,signal)\n    ax.set_xlabel('time')\n    ax.set_title('recording_id={recid},tstart={tstart}'.format(recid=example['recording_id'],tstart=example['t_min']))\n    \nds_iter = iter(dataset)\nfor subarr in ax_arr:\n    for ax in subarr:\n        plot_example(ax,next(ds_iter))\n\nplt.show()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}