{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Interactive inspection of rainforest recordings\n> A (¿hopefully?) helpful but certainly entertaining tool to inspect rainforest recordings. "},{"metadata":{"trusted":true},"cell_type":"markdown","source":"## TL;DR\n\nThis notebook mainly implements two classes, `SingleRecordingInspector` and `SongOrSpeciesRecordingInspector`. These make use of `ipywidgets` and provide an interactive tool to inspect audio files, highlighting the regions with known species/song labels. \n\nWhat they do:\n* `SingleRecordingInspector`: \n * displays one specific spectrogram of interest\n * renders labeled boxes filtered by `f_min`, `f_max` and whether or not the box label is tp or fp\n * provides audio playback of the file\n* `SongOrSpeciesRecordingInspector`: \n * displays your choice of one or more spectrograms and audio for a specific `species_id` or `songtype_id`\n * renders labeled boxes\n \nSo, `SingleRecordingInspector` helps to understand what is in a single audio file and `SongOrSpeciesRecordingInspector` helps to understand what a specific signal (by species or song type) looks like across audio files.\n\nHappy clicking! 😁"},{"metadata":{},"cell_type":"markdown","source":"## References\n\n\n- blog article about the mel spectrogram ([TDS](https://towardsdatascience.com/getting-to-know-the-mel-spectrogram-31bca3e2d9d0))\n\nSteve Brunton explainer videos: \n- Spectrogram example in python ([yt](https://www.youtube.com/watch?v=TJGlxdW7Fb4))\n- The Gabor transform ([yt](https://www.youtube.com/watch?v=EfWnEldTyPA))\n\nAnalysis notebooks:\n* https://www.kaggle.com/prokaggler/rfcx-species-audio-detection-eda-pytorch\n* https://www.kaggle.com/lakshya91/basic-audio-and-data-analysis\n* [](http://)https://www.kaggle.com/alkahapur/exploratory-data-analysis-and-modelling-using-cnn"},{"metadata":{},"cell_type":"markdown","source":"## Imports"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nimport matplotlib as mpl\nimport matplotlib.pyplot as plt\n\nimport torchaudio\nimport librosa\nimport librosa.display\n\nimport IPython.display as ipd\nfrom IPython.display import display, HTML, clear_output\n\nimport ipywidgets as widgets\nfrom ipywidgets.widgets.interaction import show_inline_matplotlib_plots\n\nfrom fastcore.all import *","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Setup"},{"metadata":{"trusted":true},"cell_type":"code","source":"base_path = Path('/kaggle/input/rfcx-species-audio-detection/')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Inspecting a single file\n> Preamble to our two `Inspector` 🧐 classes."},{"metadata":{},"cell_type":"markdown","source":"First, let's code something to parse flac files using `torchaudio`."},{"metadata":{"trusted":true},"cell_type":"code","source":"rec_id = 'c12e0a62b'\naudio, sample_rate = torchaudio.load(base_path/f'train/{rec_id}.flac')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'audio size: {audio.size()}, sample_rate {sample_rate}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Second, let's listen to the audio file."},{"metadata":{"trusted":true},"cell_type":"code","source":"ipd.display(ipd.Audio(data=audio[0], rate=sample_rate))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Third, let's collect our label data."},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_labels(base_path):\n df_train_fp = (pd.read_csv(base_path/'train_fp.csv')\n .assign(positive=True))\n df_train_tp = (pd.read_csv(base_path/'train_tp.csv')\n .assign(positive=False))\n return pd.concat((df_train_fp, df_train_tp), ignore_index=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ndf = get_labels(base_path)\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mask = df['recording_id'] == rec_id\ndf_sounds = df.loc[mask]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Fourth, let's plot the spectrogram and on top of it the labels."},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_spectrogram(audio, sample_rate, df_sounds, rec_id,\n positive_background_color='white',negative_background_color='white',\n positive_frame_color='green', negative_frame_color='red',\n positive_font_color='green', negative_font_color='red', figsize=(12,4), dpi=150):\n fig, ax = plt.subplots(nrows=1, ncols=1, figsize=figsize, dpi=dpi)\n ax.tick_params(axis='x', labelsize=10)\n ax.tick_params(axis='y', labelsize=10)\n\n tmp = librosa.stft(audio[0].numpy())\n tmp_db = librosa.amplitude_to_db(abs(tmp))\n img = librosa.display.specshow(tmp_db, sr=sample_rate, x_axis='time',\n y_axis='hz',ax=ax)\n\n boxes = {'positive': [], 'negative': []}\n frames = {'positive': [], 'negative': []}\n for i, row in df_sounds.iterrows():\n xy = (row['t_min'], row['f_min'])\n width = row['t_max'] - row['t_min']\n height = row['f_max'] - row['f_min']\n box = mpl.patches.Rectangle(xy, width, height)\n frame = mpl.patches.Rectangle(xy, width, height)\n if row['positive']:\n boxes['positive'].append(box)\n frames['positive'].append(frame)\n else:\n boxes['negative'].append(box)\n frames['negative'].append(frame)\n msg = f'species: {row[\"species_id\"]}\\nsong: {row[\"songtype_id\"]}'\n c = positive_font_color if row['positive'] else negative_font_color\n ax.annotate(msg, xy=(xy[0],xy[1]+height), color=c, fontsize=12)\n for k,c in zip(['positive','negative'],[positive_background_color, negative_background_color]):\n _boxes = mpl.collections.PatchCollection(boxes[k],facecolor='white', lw=2, alpha=.2)\n ax.add_collection(_boxes)\n for k,c in zip(['positive','negative'],[positive_frame_color, negative_frame_color]):\n _frames = mpl.collections.PatchCollection(frames[k],facecolor='None',edgecolor=c, lw=2, alpha=.7)\n ax.add_collection(_frames)\n cax = fig.colorbar(img)\n cax.ax.set_title('dB')\n ax.set_title(f'Spectrogram for recording_id {rec_id}', fontsize=20)\n return fig, ax","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nplot_spectrogram(audio, sample_rate, df_sounds, rec_id);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Lastly, let's collect all the file paths."},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_flac_files(path:Path, condition=lambda x: x.suffix == '.flac'):\n return {val.name: val for val in path.ls().filter(condition)}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nfiles = {k: get_flac_files(base_path/k) for k in ['train','test']}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## The big show using `ipywidgets`\n> Here we \"only\" re-use the above functions."},{"metadata":{},"cell_type":"markdown","source":"Both inspector (🧐) classes have the same structure. So let's create a template class. "},{"metadata":{"trusted":true},"cell_type":"code","source":"class SoundInspectorTemplate:\n 'Template for inspector GUIs'\n def __init__(self, files, df):\n self.files = files\n self.df = df\n \n def plot(self, change):\n 'Define what is plotted here'\n pass\n \n def render(self):\n 'Define widgets and their arrangement here'\n pass","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Inspecting a single recording"},{"metadata":{"trusted":true},"cell_type":"code","source":"@delegates() # fun decorator from fastcore: https://fastcore.fast.ai/meta.html#delegates\nclass SingleRecordingInspector(SoundInspectorTemplate):\n 'Inspecting a single recording'\n def __init__(self, *args, **kwargs):\n super().__init__(*args, **kwargs)\n \n def _update_file_options(self, new_set, f_min, f_max, sample_type):\n if len(new_set) == 0: return []\n if (f_min == f_max) or (f_max < f_min): return []\n if len(sample_type) == 0: return []\n files = list(self.files[new_set].keys())\n types = [True] if sample_type == 'positive' else [False] if sample_type == 'negative' else [True,False]\n if new_set == 'train':\n mask_fmin = (self.df['f_min']>=f_min) \n mask_fmax = (self.df['f_max']<=f_max) \n mask_type = (self.df['positive'].isin(types))\n rec_ids = self.df.loc[mask_fmin & mask_fmax & mask_type,'recording_id'].unique()\n files = [f'{v}.flac' for v in rec_ids if f'{v}.flac' in files]\n return files\n \n def update_file_options_set(self, new_set):\n return self._update_file_options(new_set, self.f_min.value, self.f_max.value, self.samples.value)\n \n def update_file_options_fmin(self, f_min):\n return self._update_file_options(self.set.value, f_min, self.f_max.value, self.samples.value)\n \n def update_file_options_fmax(self, f_max):\n return self._update_file_options(self.set.value, self.f_min.value, f_max, self.samples.value)\n \n def update_file_options_sample_type(self, sample_type):\n return self._update_file_options(self.set.value, self.f_min.value, self.f_max.value, sample_type) \n \n def plot(self, change):\n audio, sample_rate = torchaudio.load(self.files[self.set.value][self.file.value])\n rec_id = self.file.value.split('.')[0]\n mask = self.df['recording_id'] == rec_id\n msg = f'''\n Visualizing {self.set.value}/{self.file.value}
\n \n {self.df.loc[mask].to_html()}\n '''\n df = self.df.loc[mask]\n with self.out:\n clear_output()\n self.text.value = msg\n ipd.display(\n ipd.Audio(data=audio[0], rate=sample_rate),\n plot_spectrogram(audio, sample_rate, df, rec_id)\n );\n show_inline_matplotlib_plots() # shortest code i've found to remove matplotlib charts during update\n \n def render(self):\n self.set = widgets.Dropdown(value='train', options=['train','test'])\n self.file = widgets.Combobox(placeholder='Choose a file', value='c12e0a62b.flac',\n options=list(self.files['train'].keys()))\n self.samples = widgets.Dropdown(value='all', options=['positive', 'negative', 'all'])\n self.f_min = widgets.FloatText(value=0, placeholder='enter f_min')\n self.f_max = widgets.FloatText(value=5e4, placeholder='enter f_ax')\n self.dlink_set_files = widgets.dlink((self.set,'value'),(self.file,'options'),self.update_file_options_set)\n self.dlink_fmin_files = widgets.dlink((self.f_min,'value'),(self.file,'options'),self.update_file_options_fmin)\n self.dlink_fmax_files = widgets.dlink((self.f_max,'value'),(self.file,'options'),self.update_file_options_fmax)\n self.dlink_sample_files = widgets.dlink((self.samples,'value'),(self.file,'options'),self.update_file_options_sample_type)\n self.out = widgets.Output()\n self.text = widgets.HTML()\n self.submit = widgets.Button(description='Submit')\n self.submit.on_click(self.plot)\n \n return widgets.HBox([\n widgets.VBox([widgets.Label('source'),widgets.Label('to highlight'),\n widgets.Label('f_min (Hz)'),widgets.Label('f_max (Hz)'),widgets.Label('file')]),\n widgets.VBox([self.set,self.samples,self.f_min,self.f_max,self.file,self.submit,self.out,self.text])\n ])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**How to use:**\n\nFirst set each of the fields:\n* `source` = source of the recording (`train` or `test` set set) \n* `to high...(light)` = which labels to draw boxes for (`positive` = tp, `negative` = fp, `all` = ¯\\\\\\_(ツ)\\_/¯)\n* `f_min` = the lowest allowed frequency for a box (only `train` set)\n* `f_max` = the highest allowed frequency for a box (only `train` set)\n* `file` = the filename of the recording to render (the list will update based on the values in the other fields)\n\nHit \"Submit\"!"},{"metadata":{"trusted":true},"cell_type":"code","source":"gui = SingleRecordingInspector(files, df)\ngui.render()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Inspecting multiple recordings"},{"metadata":{"trusted":true},"cell_type":"code","source":"@delegates() # fun decorator from fastcore: https://fastcore.fast.ai/meta.html#delegates\nclass SongOrSpeciesRecordingInspector(SoundInspectorTemplate):\n 'Inspecting a single recording'\n def __init__(self, *args, **kwargs):\n super().__init__(*args, **kwargs)\n \n def plot(self, change):\n mask = self.df['recording_id'].isin(self.recordings.value)\n mask = mask & (self.df[f'{self.mode.value}_id'] == int(self.selection.value))\n df = self.df.loc[mask]\n msg = f'''\n Visualizing {self.mode.value}_id == {self.selection.value} in recordings: {\", \".join(self.recordings.value)}\n '''\n self.text.value = msg\n for i,(rec_id, _df) in enumerate(df.groupby('recording_id')):\n with self.out.children[i]: \n clear_output()\n audio, sample_rate = torchaudio.load(self.files['train'][f'{rec_id}.flac'])\n ipd.display(\n ipd.Audio(data=audio[0], rate=sample_rate),\n plot_spectrogram(audio, sample_rate, _df, rec_id)\n );\n show_inline_matplotlib_plots() # shortest code i've found to remove matplotlib charts during update\n \n def update_selection_options(self,change):\n 'updating self.selection.options based on self.mode.value'\n if len(change) == 0: return []\n col = f'{change}_id'\n return tuple(map(str,self.df[col].unique()))\n \n def _update_recording_options(self,selection,sample):\n if (len(selection) == 0) or (len(sample) == 0): return []\n col = f'{self.mode.value}_id'\n sample_options = [True] if (sample == 'positive') else [False] if (sample == 'negative') else [True,False]\n mask = (self.df[col] == int(selection)) & (self.df['positive'].isin(sample_options))\n return tuple(self.df.loc[mask,'recording_id'].unique())\n \n def update_recording_options_selection(self,change):\n 'updating self.recording.options based on self.selection.value'\n return self._update_recording_options(change,self.samples.value)\n \n def update_recording_options_samples(self,change):\n 'updating self.recording.options based on self.selection.value'\n return self._update_recording_options(self.selection.value,change)\n \n def _update_tab_titles(self):\n rec_ids = self.recordings.value\n self.out.children = [widgets.Output() for _ in rec_ids]\n for i, rec_id in enumerate(rec_ids):\n self.out.set_title(i,f'rec_id: {rec_id}')\n \n def update_tab_titles(self, change):\n if (change['owner'] != self.recordings) or (change['type'] != 'change') or isinstance(change['new'],dict) or (change['new']==change['old']) or (len(change['new']) == 0):\n return\n self._update_tab_titles()\n \n def render(self):\n self.mode = widgets.Dropdown(value='species', options=['species','songtype'])\n self.selection = widgets.Combobox(placeholder='Choose a species/song type', value='3',\n options=list(map(str,self.df['species_id'].unique())))\n self.dlink_mode_selection = widgets.dlink((self.mode,'value'),(self.selection, 'options'), self.update_selection_options)\n recs = self.df.loc[self.df['species_id']==3,'recording_id'].unique()\n self.recordings = widgets.SelectMultiple(options=recs,value=['c12e0a62b'])\n self.samples = widgets.Dropdown(value='all', options=['positive', 'negative', 'all'])\n self.dlink_selection_recordings = widgets.dlink((self.selection, 'value'),(self.recordings,'options'), self.update_recording_options_selection)\n self.dlink_samples_recordings = widgets.dlink((self.samples, 'value'),(self.recordings,'options'), self.update_recording_options_samples)\n self.out = widgets.Accordion(children=[widgets.Output() for _ in self.recordings.value])\n self._update_tab_titles()\n self.recordings.observe(self.update_tab_titles)\n self.text = widgets.HTML()\n self.submit = widgets.Button(description='Submit')\n self.submit.on_click(self.plot)\n \n return widgets.HBox([\n widgets.VBox([widgets.Label('id type'),widgets.Label('species/song id'),widgets.Label('to highlight'),\n widgets.Label('recording ids'),]),\n widgets.VBox([self.mode,\n self.selection,\n self.samples,\n self.recordings,\n self.submit,self.text,self.out]),\n ])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**How to use:**\n\nFirst enter stuff:\n* `id type` = id (`species` or `songtype`) to filter the recordings for \n* `species/...(songtype)` = which id of species or song type to filter the recordings for\n* `to high...(light)` = which labels to draw boxes for (`positive` = tp, `negative` = fp, `all` = ¯\\\\\\_(ツ)\\_/¯)\n* `recording` = `recording_id` values to select (using ctrl or shift you can select multiple!!!) (also: the list of available `recording_id` values is updated based on the other fields (sometimes this isn't instantaneous))\n\nHit \"Submit\"!\n\nNote: To not overwhelm the output for each selected recording is hidden in a designated clickable field. If you want to inspect the results don't forget to click on the respective field and 🧐."},{"metadata":{"trusted":true},"cell_type":"code","source":"gui = SongOrSpeciesRecordingInspector(files, df)\ngui.render()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"That's it! 🥳"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}