{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
\n\n

BirdCLEF '22 - Visualize Predictions

\n
CREATED BY: DARIEN SCHETTLER
\n\n
\n\n---\n\n
\n\n
\n 👏   IF YOU FORK THIS OR FIND THIS HELPFUL   👏

PLEASE UPVOTE!

This was a lot of work for me and while it may seem silly, it makes me feel appreciated when others like my work. 😅\n
\n\n---","metadata":{}},{"cell_type":"markdown","source":"
\n\n\n\n

0  IMPORTS    ⤒

\n\n**You don't need all of these... it's just easier for me to copy and paste them all over from my other notebooks**","metadata":{}},{"cell_type":"code","source":"print(\"\\n... IMPORTS STARTING ...\\n\")\n\n# print(\"\\n... PIP/APT INSTALLS AND DOWNLOADS/ZIP STARTING ...\")\n# print(\"... PIP/APT INSTALLS COMPLETE ...\\n\")\n\nprint(\"\\n\\tVERSION INFORMATION\")\n# Machine Learning and Data Science Imports\nimport tensorflow as tf; print(f\"\\t\\t– TENSORFLOW VERSION: {tf.__version__}\");\nimport tensorflow_addons as tfa; print(f\"\\t\\t– TENSORFLOW ADDONS VERSION: {tfa.__version__}\");\nimport tensorflow_io as tfio; print(f\"\\t\\t– TENSORFLOW I/O VERSION: {tfio.__version__}\");\nimport pandas as pd; pd.options.mode.chained_assignment = None; pd.set_option('display.max_columns', 50);\nimport numpy as np; print(f\"\\t\\t– NUMPY VERSION: {np.__version__}\");\nimport sklearn; print(f\"\\t\\t– SKLEARN VERSION: {sklearn.__version__}\");\nfrom sklearn.preprocessing import RobustScaler, PolynomialFeatures\nfrom pandarallel import pandarallel; pandarallel.initialize()\nfrom sklearn.model_selection import GroupKFold, StratifiedKFold\n\n# Competition Specific (AUDIO)\nimport librosa; import librosa.display\nimport soundfile as sf\nfrom soundfile import SoundFile\nimport IPython.display as ipd\n\n# Built In Imports\nfrom kaggle_datasets import KaggleDatasets\nfrom collections import Counter\nfrom datetime import datetime\nfrom glob import glob\nimport warnings\nimport requests\nimport hashlib\nimport imageio\nimport IPython\nimport sklearn\nimport urllib\nimport zipfile\nimport pickle\nimport random\nimport shutil\nimport string\nimport json\nimport math\nimport time\nimport gzip\nimport ast\nimport sys\nimport io\nimport os\nimport gc\nimport re\n\n# Visualization Imports\nfrom matplotlib.colors import ListedColormap\nfrom joblib import Parallel, delayed\nimport matplotlib.patches as patches\nimport plotly.graph_objects as go\nimport matplotlib.pyplot as plt\nfrom tqdm.notebook import tqdm; tqdm.pandas();\nimport plotly.express as px\nimport seaborn as sns\nfrom PIL import Image, ImageEnhance\nimport matplotlib; print(f\"\\t\\t– MATPLOTLIB VERSION: {matplotlib.__version__}\");\nfrom matplotlib import animation, rc; rc('animation', html='jshtml')\nimport plotly\nimport PIL\nimport cv2\n\nimport plotly.io as pio\nprint(pio.renderers)\n\ndef seed_it_all(seed=7):\n \"\"\" Attempt to be Reproducible \"\"\"\n os.environ['PYTHONHASHSEED'] = str(seed)\n random.seed(seed)\n np.random.seed(seed)\n tf.random.set_seed(seed)\n\n \nprint(\"\\n\\n... IMPORTS COMPLETE ...\\n\")","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-03-27T22:59:32.247349Z","iopub.execute_input":"2022-03-27T22:59:32.247668Z","iopub.status.idle":"2022-03-27T22:59:42.296770Z","shell.execute_reply.started":"2022-03-27T22:59:32.247596Z","shell.execute_reply":"2022-03-27T22:59:42.295069Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n

1  SETUP    ⤒

\n\n---\n\nPlease note that your setup may differ based on your particular modelling pipeline. **The setup should not impact the visualization tool as it only digests your predictions.** I took this as an opportunity to distil my pipeline and explore it hence the detailed explanation.\n\n**In other words, you could skip this cell if you already had a dataframe that looks like the `sample_submission` dataframe in terms of format but contains valid file information. The visualization tool will take either the `boolean` version of this (True v. False) or a logit version of this (0.00-1.00).**\n\n***Example 1 - Bools***\n```\nrow_id,target\nsoundscape_1000170626_akiapo_5,False\nsoundscape_1000170626_akiapo_10,False\nsoundscape_1000170626_akiapo_15,False\netc.\n```\n\n***Example 2 - Probabilities***\n```\nrow_id,target\nsoundscape_1000170626_akiapo_5,0.01\nsoundscape_1000170626_akiapo_10,0.025\nsoundscape_1000170626_akiapo_15,0.05\netc.\n```\n\n
\n\n---\n\n
\n\nThe setup will take **5-10 minutes** and is composed of the following steps:\n\n1. Add file path to dataframe\n2. Add audio information (duration) to dataframe\n3. Remove all audio clips that don't fit within a given duration length *(optional)*\n * This is to help us capture a good sampling of Hawaiian birds along with other birds\n * This also helps us avoid ending up with very-short or very-long clips of audio\n * For reference my extents are:\n * Minimum: 3 Seconds\n * Maximum: 180 seconds\n * We also take a very particular clip that has a duration of 249.048 as this is the only clip that contains a particular Hawaiian bird.\n4. Reduce the size of the dataframe considerably by sampling audio clips for each respective bird\n * Note that we give a preference for audio clips that are close to 60 seconds (as this is the length of the test audio clips\n * Note that we pad/clip audio clips to be exactly 60 seconds to mimic the hidden test data\n * Note that the **`duration_goal`** parameter is how we control the degree of reduction (i.e. 66,000 is 1/10th the size of the expected test set and is a decent starting point)\n5. Break the clips down into 7 second segments with a 2 second step (i.e. a 60 second clip will be made of 28 7-second segments) and save these segments as Mel Spectrograms to be used for inferencing\n6. Find weights that will allow us to map the 7-second segments onto the 5-second scoring segments\n7. Conduct inference and create the prediction dataframe using nocall and multilabel models.\n * NOTE: I have not made these models public yet, so this pipeline won't work if you just fork and run it. It will however, work for YOUR predictions as mentioned previously. I will also most likely make my models public in the near future.","metadata":{}},{"cell_type":"code","source":"#################################\n##### CONSTANTS AND KWARGS ####\n#################################\nDATA_DIR = \"/kaggle/input/birdclef-2022\"\nTMP_OUT_DIR = \"/tmp/data_dir\"\nSR = 32_000 # Define the Sample Rate\nWINDOW = 7 # Define the Window Size (in seconds)\nWINDOW_SR = WINDOW*SR # Define the Window Size (in samples)\nSTEP = 2 # Define the Step/Overlap Size (in seconds)\nOVERLAP = WINDOW-STEP # Define the Step/Overlap Size (in seconds)\nSTEP_SR = int(round(STEP*SR)) # Define the Step/Overlap Size (in samples)\nOVERLAP_SR = int(round(OVERLAP*SR)) # Define the Step/Overlap Size (in samples)\nDISCARD = 10 # Define the Discard Threshold Time (in seconds)(impossible for inference)\nDISCARD_SR = int(DISCARD*SR) # Define the Discard Threshold Time (in samples)\nNFFT = SR//10\nHOP_LENGTH = SR//40\nWIN_LENGTH = NFFT\nN_MELS = 128\nMEL_LEN = 281\nF_MIN = 0\nF_MAX = SR//2\nMAX_CLIP_LEN = 60\nMAX_CLIP_LEN_SR = MAX_CLIP_LEN*SR\n\nos.makedirs(TMP_OUT_DIR, exist_ok=True)\nmelspec_kwargs = dict(\n sr=SR, \n n_mels=N_MELS, \n fmin=F_MIN, \n fmax=F_MAX, \n n_fft=NFFT, \n hop_length=HOP_LENGTH,\n)\nsegment_kwargs = dict(\n sample_rate=SR, \n window_size=WINDOW_SR, \n step_size=STEP_SR,\n overlap_size=OVERLAP_SR, \n discard_size=DISCARD_SR, \n output_dir=TMP_OUT_DIR,\n _fill_value=0.0, \n _format=\"MEL\"\n)\n#################################\n\ndef get_audio_info(filepath):\n \"\"\"\n Get some properties from an audio file\n \n SOURCE: https://www.kaggle.com/kneroma/birdclef-mels-computer-public\n \"\"\"\n with SoundFile(filepath) as f:\n sr = f.samplerate\n frames = f.frames\n duration = float(frames)/sr\n return {\"frames\": frames, \"sr\": sr, \"duration\": duration}\n\ndef add_audio_to_df(row):\n \"\"\" Wrapper for pandas to add audio info to each row \"\"\"\n for k,v in get_audio_info(row[\"f_path\"]).items():\n row[k] = v\n return row\n\n\ndef add_clip_weighting(_df):\n def _return_pct_overlap(test_span, clip_span, full_clip_length=160_000):\n upper_lim = min(test_span[-1], clip_span[-1])\n lower_lim = max(test_span[0], clip_span[0])\n return max(0.0, upper_lim-lower_lim)/full_clip_length\n\n \n __df = _df.copy()\n \n # Create temporary dataframe \n _tmp_df = pd.DataFrame({\"mel_f_path\":sorted(glob(os.path.join(TMP_OUT_DIR, \"**/*.png\")))})\n _tmp_df.insert(0, \"test_id\", _tmp_df[\"mel_f_path\"].apply(lambda x: x.rsplit(\"/\", 2)[1]))\n _tmp_df.insert(1, \"time_span\", _tmp_df[\"mel_f_path\"].apply(lambda x: x.rsplit(\"/\", 1)[-1][:-4]))\n _tmp_df.insert(2, \"_id_w_timespan\", _tmp_df[\"test_id\"]+\"/\"+_tmp_df[\"time_span\"])\n for i in range(5,65,5):\n _tmp_df[f\"clip_{i}_wt\"] = _tmp_df.time_span.apply(lambda x: _return_pct_overlap(test_span=((i-5)*SR, i*SR), clip_span=(int(x[:9]), int(x[-9:]))))\n \n return _tmp_df\n\ndef get_reduced_df(df, return_indexed_df=True, duration_goal=66_000, verbose=0):\n \"\"\" \n This function willl pare the train dataframe down into a dataframe containing \n enough examples to add up to a duration of approximately 66,000 seconds of audio.\n \n The restrictions are that we attempt to evenly pull from all birds with a preference\n being given for audio clips that are as close to 60 seconds as possible.\n \n Args:\n df (pd.DataFrame): To be pared down\n return_indexed_df (bool, optional): Whether to return the \n pared dataframe ... or simply the indices to use for paring.\n duration_goal (int, optional): Sum of durations for pared dataframe\n \n Returns:\n Either a list of indices or a pared down pd.DataFrame\n \"\"\"\n def _verbose_print(print_duration=True, v_span=100):\n # Verbose printing\n if len(pare_indices)%v_span==0: \n print(f\"\\n\\n\\t{len(pare_indices):>4}-{len(pare_indices)+v_span:<4} {'(DURATION='+str(round(duration, 2))+')' if print_duration else ''}\")\n else: \n print(\".\", sep=\"\", end=\"\")\n \n pare_indices, duration = [], 0\n redux_df = df.copy()\n redux_df[\"t_delta\"] = (redux_df.duration-60).abs()\n \n _all_species = df.primary_label.unique()\n \n while True: \n # For loop for species... to be iterated over\n for _species in _all_species:\n # temporary dataframe based on species\n _tmp_df = redux_df[redux_df.primary_label==_species]\n \n # Catch for empty dataframe and verbose print\n if len(_tmp_df)==0: continue\n if verbose: _verbose_print()\n \n pare_indices.append(_tmp_df.sort_values(by=\"t_delta\").index.values[0])\n redux_df=redux_df[~redux_df.index.isin(pare_indices)] \n \n # Return catch\n duration+=60\n if duration>duration_goal: \n if return_indexed_df:\n return df.iloc[pare_indices].reset_index(drop=True)\n else:\n return pare_indices\n\ndef get_melspec(audio, sr, n_mels, fmin, fmax, n_fft, hop_length,):\n \n # Coerce Input\n if type(audio)==str: audio = sf.read(audio, dtype=\"float32\")[0]\n if len(audio.shape)>1: audio=audio[:, 0] \n \n # Convert To Mel Spectrogram\n melspec = librosa.feature.melspectrogram(y=audio, sr=sr, n_mels=n_mels, fmin=fmin, fmax=fmax, n_fft=n_fft, hop_length=hop_length)\n melspec = librosa.power_to_db(melspec).astype(np.float32)\n \n return mono_to_color(melspec)\n\ndef mono_to_color(X, eps=1e-6, mean=None, std=None):\n \"\"\" https://www.kaggle.com/kneroma/birdclef-mels-computer-public \"\"\"\n mean = mean or X.mean()\n std = std or X.std()\n X = (X - mean) / (std + eps)\n \n _min, _max = X.min(), X.max()\n\n if (_max - _min) > eps:\n V = np.clip(X, _min, _max)\n V = 255 * (V - _min) / (_max - _min)\n V = V.astype(np.uint8)\n else:\n V = np.zeros_like(X, dtype=np.uint8)\n\n return V\n \n# We will update segment_audio fn to output/save melspecs instead of ogg files\ndef segment_audio(audio_path, sample_rate, window_size, \n step_size, overlap_size, discard_size, \n output_dir, _fill_value=0.0, _format=\"MEL\", \n _mspec_kwargs=melspec_kwargs, enforce_len=True,\n enforce_len_sr=MAX_CLIP_LEN_SR):\n \"\"\"\n \n Create segments of audio from longer audio clip and save to file\n \n Usage:\n _ = train_df.f_path.progress_apply(lambda x: segment_audio(x, **segment_kwargs))\n \n Args:\n audio_path (str): Path to the audio file to be segmented (.ogg) file\n sample_rate (int): Rate at which the audio was originally sampled (default is 32_000)(Hz)\n window_size (int): Length of desired window segment measured in # of samples.\n i.e. A window segment value of 224_000 would be a 7 second segment assuming\n a sample rate of 32_000 samples per second.\n step_size (int): Length of desired step between segments measured in # of samples.\n i.e. A step size value of 149_333 would be approximately a 4.6667 second step\n assuming a sample rate of 32_000 samples per second.\n overlap_size (int): Length of overlap between consecutive segments measured in # of samples.\n i.e. An overlap value of 74_667 would be approximately a 2.33333 second segment \n assuming a sample rate of 32_000 samples per second.\n discard_size (int): Length of empty audio identifing a segment to be discarded measured in # of samples.\n i.e. A discard size value of 192_000 would indicate that segments containing more than 6\n seconds of empty audio should be discarded... assuming a sample rate of 32_000.\n output_dir (str): The path to the directory to output the segmented audio files\n _mspec_kwargs (dict): Keyword arguments for mel spectrogram creation\n _fill_value (float, optional): The 'empty' audio value to pad the original audio with \n to allow the original audio to be segmented evenly without having a single shorter/longer\n segment at the end/beginning.\n _format (str, optional):\n The format for which to save the segments\n enforce_len (bool, optional):\n Whether to limit the audio clip read to the maximum duratino (60 seconds usually)\n enforce_len_sr (int, optional):\n Only used if enforce_len==True. Number of frames of audio to limit audio file to.\n \n Returns:\n None; This function will simply save audio segments to file. No actual entities\n will be returned in the interpreter.\n \n \"\"\"\n # Prepare\n example_id = audio_path.rsplit(\"/\", 1)[-1][:-4]\n save_dir = os.path.join(output_dir, example_id)\n if not os.path.isdir(save_dir): os.makedirs(save_dir, exist_ok=True)\n \n # Process\n for i, audio_block in enumerate(sf.blocks(audio_path, window_size, overlap_size, frames=enforce_len_sr if enforce_len else -1, dtype=\"float32\", fill_value=_fill_value)):\n \n # Discard if not the first clip and less than the discard amount of the clip is padding\n if enforce_len or i==0 or np.count_nonzero(audio_block==0.0)09}_{_end:>09}.ogg\"), audio_block, samplerate=sample_rate, format=\"OGG\")\n elif _format.lower()==\"mel\":\n cv2.imwrite(os.path.join(save_dir, f\"{_start:>09}_{_end:>09}.png\"), get_melspec(audio_block, **_mspec_kwargs)) \n \ndef mp_segment_audio(_df, _backend=\"loky\", _njobs=-1, _prefer=\"processes\", _verbose=0, _enforce_len=True):\n \"\"\" Parallel file creation of mel spectrograms\n \n Args:\n _df (pd.DataFrame): The dataframe to use as base for file creation\n _backend (str, optional): Joblib backend\n NOTE: `loky` backend operates almost twice as fast as others \n so it's recommended to leave it be... for default kaggle instance\n _njobs (int, optional ): Number of concurrent workers for joblib\n _prefer (str, optional): Whether to prefer proccesses or threads in joblib\n _verbose (int, optional): Whether to let joblib display progress\n \n Returns:\n None; Simply call the function that will write files to disk and let it happen\n \"\"\"\n _ogg_file_paths = list(set(_df.ogg_f_path.tolist()))\n n_test_files = len(_ogg_file_paths)\n \n _ = Parallel(n_jobs=_njobs, backend=_backend, prefer=_prefer, verbose=_verbose)(\n delayed(segment_audio)(audio_path=_f_path, **segment_kwargs, _mspec_kwargs=melspec_kwargs, enforce_len=_enforce_len) for _f_path in _ogg_file_paths\n )\n \ndef augment_batch(img_batch, pp_fn):\n \"\"\" Simple pass through for now...\n \n Investigate this \n --> https://www.tensorflow.org/io/tutorials/audio#trim_the_noise\n \n \"\"\"\n return pp_fn(tf.cast(img_batch, tf.float32))\n\ndef get_ss_row_id_base(tensor_str):\n return tf.strings.split(tensor_str, sep=\"/\", maxsplit=-1)[-2]\n\ndef get_timespan(tensor_str):\n return tf.strings.split(tf.strings.split(tensor_str, sep=\"/\", maxsplit=-1)[-1], sep=\".\")[0]\n\ndef tf_load_image(img_path, reshape_to=(128,281,3)):\n return tf.reshape(tf.image.decode_png(tf.io.read_file(img_path), channels=3), reshape_to)\n\ndef get_ds_for_inference(_clip_df, _bs, _pp, do_plot=False):\n _mel_files = _clip_df.mel_f_path.tolist()\n n_files = len(_mel_files)\n\n # Create Filepath Dataset\n _ds = tf.data.Dataset.from_tensor_slices(_mel_files)\n\n # Read The File From Path and Get Relevant Info\n _ds = _ds.map(lambda path: (tf_load_image(path), get_ss_row_id_base(path), get_timespan(path)), num_parallel_calls=tf.data.AUTOTUNE)\n \n # Optional Plot\n if do_plot:\n for __img, _, _ in _ds.take(1):\n plt.imshow(__img[..., 0])\n plt.title(\"EXAMPLE SPECTROGRAM\", fontweight=\"bold\")\n plt.show()\n\n # Pipeline preparing dataset for inference\n _ds = _ds.batch(_bs)\\\n .map(lambda img, row_id, t_span: (augment_batch(img, _pp), row_id, t_span), num_parallel_calls=tf.data.AUTOTUNE)\\\n .prefetch(tf.data.AUTOTUNE)\n\n return _ds, n_files\n\ndef get_pred_df(_df,\n _batch_size=192, \n _mlbl_fpath=\"../input/training-birdclef-22-multilabel-nocall/f1_effnet_b0_128x281x3_cwt_10__doublesoftloss\",\n _ncall_fpath=\"/kaggle/input/birdclef-2022-nocall-effnetb0/effnet_b0_callnocall__7S_128x281x3\",\n _pp_fn=tf.keras.applications.efficientnet.preprocess_input):\n\n # Load the models\n nc_model = tf.keras.models.load_model(_ncall_fpath, compile=False)\n mlbl_model = tf.keras.models.load_model(_mlbl_fpath, compile=False)\n\n # Get the dataset\n ds, n_ex = get_ds_for_inference(_df, _batch_size, _pp_fn, do_plot=True)\n\n # Inference Loop\n for i, (_img_batch, _row_id_batch, _timespan_batch) in tqdm(enumerate(ds), total=(n_ex//_batch_size+1)):\n if i==0:\n # I WAS using .predict() ... \n # however this leads to memory leakage and the resulting submission error\n mlbl_preds = mlbl_model(_img_batch, training=False)\n nc_preds = nc_model(_img_batch, training=False)\n ss_row_id_bases = _row_id_batch\n timespans = _timespan_batch\n else:\n # I WAS using .predict() ... but this leads to memory leakage and the resulting submission error\n # however this leads to memory leakage and the resulting submission error\n mlbl_preds = tf.concat([mlbl_preds, mlbl_model(_img_batch, training=False)], axis=0)\n nc_preds = tf.concat([nc_preds, nc_model(_img_batch, training=False)], axis=0)\n ss_row_id_bases = tf.concat([ss_row_id_bases, _row_id_batch], axis=0)\n timespans = tf.concat([timespans, _timespan_batch], axis=0)\n\n pred_df = pd.DataFrame({\"_id_w_timespan\":[a+\"/\"+b for a,b in zip(\n [x.decode() for x in ss_row_id_bases.numpy()], [x.decode() for x in timespans.numpy()]\n )]})\n\n pred_df = pred_df.join(pd.DataFrame(mlbl_preds.numpy()[:, np.array(SCORED_BIRD_INTS)], columns=SCORED_BIRD_STRS))\n pred_df[\"call_conf\"] = nc_preds.numpy()[:, 1]\n _df = _df.merge(pred_df, on=\"_id_w_timespan\")\n return _df\n\ndef get_preds_per_5sec(_df, base_row_id, to_bool=False,\n do_scored=True, scored_thresh=0.1, \n do_pwr=False, pwr=20, \n do_nocall=False, nocall_thresh=0.5, \n do_combo=False, combo_thresh=0.9):\n \"\"\" TBD \"\"\"\n \n # Initialize\n _sub_df = _df[_df.test_id==base_row_id]\n sub_pred_map = {\"row_id\":[], \"target\":[]}\n \n # Iterate over clip segments\n for i in range(5, 65, 5):\n # Get sub dataframe\n _span_sub_df = _sub_df[_sub_df[f\"clip_{i}_wt\"]!=0.0]\n \n # Get model predictions\n average_call_conf = (_span_sub_df.call_conf*_span_sub_df[f\"clip_{i}_wt\"]).sum()/_span_sub_df[f\"clip_{i}_wt\"].sum()\n average_preds = np.dot(_span_sub_df[f\"clip_{i}_wt\"], _span_sub_df[SCORED_BIRD_STRS].to_numpy())/_span_sub_df[f\"clip_{i}_wt\"].sum()\n \n if do_pwr: average_preds = average_preds**pwr\n \n if to_bool:\n if do_combo:\n thresh_preds = np.where(average_preds*average_call_conf>combo_thresh, True, False)\n elif do_nocall:\n if average_call_conf>nocall_thresh:\n thresh_preds = np.where(average_preds>scored_thresh, True, False)\n else:\n thresh_preds = [False,]*len(SCORED_BIRD_STRS)\n else:\n thresh_preds = np.where(average_preds>scored_thresh, True, False)\n else:\n if do_combo:\n thresh_preds = average_preds*average_call_conf\n else:\n thresh_preds = average_preds\n \n for _bird, _pred in zip(SCORED_BIRD_STRS, thresh_preds):\n sub_pred_map[\"row_id\"].append(base_row_id+f\"_{_bird}_{i}\")\n sub_pred_map[\"target\"].append(_pred)\n \n return pd.DataFrame(sub_pred_map)\n\n# def get_preds_per_5sec(_df, base_row_id, \n# do_scored=True, scored_thresh=0.1, \n# do_pwr=False, pwr=10, \n# do_nocall=False, nocall_thresh=0.1, \n# do_combo=False, combo_thresh=0.9, take_top_only=False):\n# _sub_df = _df[_df.test_id==base_row_id]\n# sub_pred_map = {\"row_id\":[], \"target\":[]}\n \n# for i in range(5, 65, 5):\n# _span_sub_df = _sub_df[_sub_df[f\"clip_{i}_wt\"]!=0.0]\n# average_call_conf = (_span_sub_df.call_conf*_span_sub_df[f\"clip_{i}_wt\"]).sum()/_span_sub_df[f\"clip_{i}_wt\"].sum()\n# average_preds = np.dot(_span_sub_df[f\"clip_{i}_wt\"], _span_sub_df[SCORED_BIRD_STRS].to_numpy())/_span_sub_df[f\"clip_{i}_wt\"].sum()\n \n# if not take_top_only:\n# if do_pwr:\n# average_preds = average_preds**pwr\n# if combo_thresh:\n# thresh_preds = np.where(average_preds*average_call_conf>combo_thresh, True, False)\n# elif do_nocall:\n# if average_call_conf>nocall_thresh:\n# thresh_preds = np.where(average_preds>scored_thresh, True, False)\n# else:\n# thresh_preds = [False,]*len(SCORED_BIRD_STRS)\n# else:\n# thresh_preds = np.where(average_preds>scored_thresh, True, False)\n# else:\n# if (average_call_conf>nocall_thresh) and (average_preds.max()>0.2):\n# max_conf = average_preds.max()\n# min_yes = max_conf-(max_conf**2)*0.25\n# if i==60:\n# if min_yes<0.25:\n# print(min_yes)\n# thresh_preds = np.where(average_preds>=min_yes, True, False)\n# else:\n# thresh_preds = [False,]*len(SCORED_BIRD_STRS)\n \n# for _bird, _pred in zip(SCORED_BIRD_STRS, thresh_preds):\n# sub_pred_map[\"row_id\"].append(base_row_id+f\"_{_bird}_{i}\")\n# sub_pred_map[\"target\"].append(_pred)\n# return pd.DataFrame(sub_pred_map)","metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-27T23:00:33.522184Z","iopub.execute_input":"2022-03-27T23:00:33.522461Z","iopub.status.idle":"2022-03-27T23:00:33.589913Z","shell.execute_reply.started":"2022-03-27T23:00:33.522429Z","shell.execute_reply":"2022-03-27T23:00:33.589028Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load dataframe\ntrain_df = pd.read_csv(os.path.join(DATA_DIR, \"train_metadata.csv\"))\n\n# Add filepath to training dataframe\ntrain_df[\"f_path\"] = train_df.filename.progress_apply(lambda x: DATA_DIR+\"/train_audio/\"+x)\n\n# Initialize Some Constants/Details\nstr2int_lbl_map = {k:i for i, k in enumerate(sorted(list(train_df.primary_label.unique())))}\nint2str_lbl_map = {v:k for k,v in str2int_lbl_map.items()}\nN_CLASSES = len(str2int_lbl_map)\nwith open('../input/birdclef-2022/scored_birds.json', 'r') as f: SCORED_BIRD_STRS=json.load(f)\nSCORED_BIRD_INTS = [str2int_lbl_map[x] for x in SCORED_BIRD_STRS]\n\n# Add duration information to training dataframe\ntrain_df = train_df.progress_apply(add_audio_to_df, axis=1)\n\n# Reduce dataframe while maintaining integrity of class distribution (targeting 60 second clips)\n#\n# NOTE: The 249.048 clip is to capture one hawaiian bird that only has one training example\ntrain_df = train_df[((train_df.duration>3) & (train_df.duration<180)) | (train_df.duration==249.048)].reset_index(drop=True)\n\n# This samples the reduced dataframe to further reduce it \n# so that the total duration of samples is 1/5th of the test set\n#\n# NOTE: This will offer preference to clips that are close to 60 seconds\n# but will cycle through species while doing so (so some species with\n# clips that are not near 60 seconds may have more extreme clip durations)\n# NOTE: duration_goal of 6_600 is 1/50th, 33_000 is 1/10th and 66_000 is 1/5th.\ntrain_df = get_reduced_df(train_df, return_indexed_df=True, verbose=0, duration_goal=66_000)\n\n# Create a new dataframe that mimics the layout of the test dataframe\ntrain_for_test_df = train_df[[\"filename\", \"f_path\"]]\ntrain_for_test_df.columns=[\"file_id\", \"ogg_f_path\"]\ntrain_for_test_df.file_id = train_for_test_df.file_id.apply(lambda x: x[:-4])\n\n# Do some printing\nprint(f\"\\n... REDUCED TRAIN DATAFRAME - {len(train_df)} ROWS ...\\n\")\ndisplay(train_df.head(3))\n\n# Save files to disk... [small difference here]\nmp_segment_audio(train_for_test_df, _verbose=1)\n\n# Add weights that show the overlap between the 7-second segment\n# and the 5-second scoring window\nclip_df = add_clip_weighting(train_for_test_df)\n\n# Get the prediction dataframe\npred_df = get_pred_df(clip_df,)\n\n# Get `sample_submission.csv` format inference dataframe\nviz_ss_df = pd.concat([\n get_preds_per_5sec(pred_df, _test_id, nocall_thresh=0.1, do_nocall=True, scored_thresh=0.025, ) \\\n for _test_id in tqdm(pred_df.test_id.unique(), \n total=len(pred_df.test_id.unique()))\n]).reset_index(drop=True)\ndisplay(viz_ss_df)","metadata":{"execution":{"iopub.status.busy":"2022-03-27T23:00:35.418899Z","iopub.execute_input":"2022-03-27T23:00:35.419375Z","iopub.status.idle":"2022-03-27T23:14:27.093804Z","shell.execute_reply.started":"2022-03-27T23:00:35.419338Z","shell.execute_reply":"2022-03-27T23:14:27.092971Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n

2  VISUALIZATION FUNCTIONALITY    ⤒

\n\n---\n","metadata":{}},{"cell_type":"code","source":"def get_sub_pred_df(_pred_df, _row_id=None, return_row_id=False):\n \"\"\" Ease of use fn \n \n If _row_id is none a random example is pulled\n \"\"\"\n if _row_id is None:\n _row_id = np.random.choice(viz_ss_df.row_id.apply(lambda x: x.split(\"_\", 1)[0]).unique())\n _sub_pred_df = _pred_df.copy()[_pred_df.row_id.str.contains(_row_id)].reset_index(drop=True)\n \n if return_row_id:\n return _sub_pred_df, _row_id\n else:\n return _sub_pred_df\n\ndef plot_clip_preds(sub_pred_df, \n row_identifier=\"Not Specified\",\n label_grid_values=True,\n n_segments=12, \n _cmap=\"magma\",\n _fig_size=(20,13),\n _xlabel=\"5 Second Segments\",\n _ylabel=\"Species Label (Identifier)\",\n n_float_decimal_places=3,):\n \"\"\" TBD \"\"\"\n # Plotting Helpers\n _x_start, _x_end, _y_start, _y_end = 3.0, 9.0, 6.0, 12.0\n _extent = [_x_start, _x_end, _y_start, _y_end]\n n_species = len(SCORED_BIRD_STRS)\n \n # For later\n float_style = sub_pred_df[\"target\"][0].dtype==\"float\"\n \n # Get predictions as numpy array (n_segments, n_bird_species)\n pred_arr = sub_pred_df[\"target\"].to_numpy().reshape(-1, n_species).T\n\n # Make figure\n plt.figure(figsize=_fig_size)\n plt.xlabel(_xlabel, fontweight=\"bold\")\n plt.ylabel(_ylabel, fontweight=\"bold\")\n plt.title(f\"Audio Clip Predictions - Visualization - {row_identifier}\", fontweight=\"bold\")\n \n # The x and y axis ticks and labels\n plt.xticks(ticks=np.linspace(_x_start, \n _x_end, \n n_segments+1), \n labels=[str(x) for x in np.arange(0, 65, 5)], fontweight=\"bold\")\n plt.yticks(ticks=np.linspace(_y_start+6/(n_species+1)/2, \n _y_end-(6/(n_species+1))/2, \n n_species), \n labels=SCORED_BIRD_STRS, fontweight=\"bold\")\n \n # Do the grid numbering if desired\n if label_grid_values:\n _jump_x = (_x_end-_x_start)/(2.0*n_segments)\n _jump_y = (_y_end-_y_start)/(2.0*n_species)\n _x_positions = np.linspace(start=_x_start, stop=_x_end, num=n_segments, endpoint=False)\n _y_positions = np.linspace(start=_y_start, stop=_y_end, num=n_species, endpoint=False)\n for y_index, y in enumerate(_y_positions):\n for x_index, x in enumerate(_x_positions):\n _label = pred_arr[y_index, x_index]\n _text_x = x+_jump_x\n _text_y = y+_jump_y\n if float_style: _label=round(_label, n_float_decimal_places)\n plt.text(_text_x, _text_y, _label, color='black' if _label>(pred_arr.max()*0.4) else 'white', ha='center', va='center')\n\n # Display\n plt.imshow(pred_arr, cmap=_cmap, extent=_extent, origin='lower', interpolation='None')\n \n if float_style:\n plt.colorbar()\n plt.clim(0.0, 1.0)\n \n plt.tight_layout()\n plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-03-27T23:14:38.014247Z","iopub.execute_input":"2022-03-27T23:14:38.014586Z","iopub.status.idle":"2022-03-27T23:14:38.032619Z","shell.execute_reply.started":"2022-03-27T23:14:38.014549Z","shell.execute_reply":"2022-03-27T23:14:38.031685Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Below Is The Macro F1 Score**\n\nWe know a few things about the test set and we have endeavoured to make our train set similar to this\n* Only 60 second clips\n* If we submit all False (no-call) then we should score 0.45-0.50\n* All non-scored birds are no calls\n\nHere is some information about how we determine some of the ground truth data.\n* If no scored-bird is found in the primary or secondary labels column than all segments from that audio clip are considered to be False for all scored birds\n* If a scored-bird is found in the primary column than we naively assume that all segments are that bird\n* If a scored-bird is found in the secondary column we will assume it is not present (this is erroneous ... but we can't help it right now).","metadata":{}},{"cell_type":"code","source":"filename_to_primary = train_df.groupby(\"filename\").primary_label.first().to_dict()\nfilename_to_primary = {k.split(\"/\")[-1][:-4]:v for k,v in filename_to_primary.items()}\n\ndef get_gt_lbl(row_id):\n xc_id, _species, _seg = row_id.split(\"_\")\n gt_primary = filename_to_primary[xc_id]\n if gt_primary==_species:\n return 1.0\n else:\n return 0.0\n \nviz_ss_df[\"gt_lbl\"] = viz_ss_df.row_id.progress_apply(get_gt_lbl)\nviz_ss_df","metadata":{"execution":{"iopub.status.busy":"2022-03-26T13:52:29.416822Z","iopub.execute_input":"2022-03-26T13:52:29.417115Z","iopub.status.idle":"2022-03-26T13:52:30.119446Z","shell.execute_reply.started":"2022-03-26T13:52:29.417079Z","shell.execute_reply":"2022-03-26T13:52:30.118666Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def comp_metric(y_true, y_pred, epsilon=1e-9):\n \"\"\" Function to calculate competition metric in an sklearn like fashion\n\n Args:\n y_true{array-like, sparse matrix} of shape (n_samples, n_outputs)\n - Ground truth (correct) target values.\n y_pred{array-like, sparse matrix} of shape (n_samples, n_outputs)\n - Estimated targets as returned by a classifier.\n Returns:\n The single calculated score representative of this competitions evaluation\n \"\"\"\n \n # Get representative confusion matrices for each label\n mlbl_cms = sklearn.metrics.multilabel_confusion_matrix(y_true, y_pred)\n\n # Get two scores (TP and TN SCORES)\n tp_scores = np.array([\n mlbl_cm[1, 1]/(epsilon+mlbl_cm[:, 1].sum()) \\\n for mlbl_cm in mlbl_cms\n ])\n \n tn_scores = np.array([\n mlbl_cm[0, 0]/(epsilon+mlbl_cm[:, 0].sum()) \\\n for mlbl_cm in mlbl_cms\n ])\n\n # Get average\n tp_mean = tp_scores.mean()\n tn_mean = tn_scores.mean()\n\n return round((tp_mean+tn_mean)/2, 8)\n\ndef get_species_arrs(pred_y, true_y, thresh=0.5, dynamic_thresh=True):\n \"\"\" Get species-wise categorical binary predictions \"\"\"\n # Prevent problems\n if thresh==0.0: thresh+=1e-9\n \n n_species = len(SCORED_BIRD_STRS)\n n_ex = (len(pred_y)//n_species)\n pred_arr = np.zeros((n_ex, n_species+1), dtype=np.uint8)\n gt_arr = np.zeros((n_ex, n_species+1), dtype=np.uint8)\n \n for i, idx in enumerate(range(0, len(pred_y), n_species)):\n max_conf = average_preds.max()\n min_yes = max_conf-(max_conf**2)*0.25\n thresh_preds = np.where(average_preds>=min_yes, True, False)\n \n pred_arr[i, :n_species] = np.where(pred_y[idx:(idx+n_species)]>=thresh, 1, 0)\n gt_arr[i, :n_species] = np.where(true_y[idx:(idx+n_species)]>=thresh, 1, 0)\n pred_arr[i, n_species]=(1 if pred_arr[i].sum()==0 else 0)\n gt_arr[i, n_species]=(1 if gt_arr[i].sum()==0 else 0)\n \n return pred_arr, gt_arr \n\n\nthresholds = []\nf1_scores = []\ncomp_scores = []\n# for _thresh in tqdm(np.arange(0, 1.005, 0.05), total=len(np.arange(0, 1, 0.005))+1):\npred_sps_arrs, gt_sps_arrs = get_species_arrs(viz_ss_df.target.to_numpy(), viz_ss_df.gt_lbl.to_numpy(), thresh=0.5)\n_f1_score = sklearn.metrics.f1_score(gt_sps_arrs[:, :-1], pred_sps_arrs[:, :-1], average='macro')\n_comp_score = comp_metric(gt_sps_arrs[:, :-1], pred_sps_arrs[:, :-1])\n\n# thresholds.append(_thresh)\n# f1_scores.append(_f1_score)\n# comp_scores.append(_comp_score)\n\nprint(f\"\\n\\n\\nSTARTING TEST w/ THRESH={_thresh}%\\n\")\nprint(f\"\\n\\t--> MACRO F1 SCORE: {_f1_score}\")\nprint(f\"\\t--> CLASSWISE COMPETITION METRIC ACCURACY: {_comp_score}\\n\")","metadata":{"execution":{"iopub.status.busy":"2022-03-26T13:45:45.597028Z","iopub.execute_input":"2022-03-26T13:45:45.59747Z","iopub.status.idle":"2022-03-26T13:45:46.000594Z","shell.execute_reply.started":"2022-03-26T13:45:45.597432Z","shell.execute_reply":"2022-03-26T13:45:45.999772Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = px.line(x=thresholds, y=[f1_scores, comp_scores], \n labels={\"wide_variable_0\":\"F1 Score\", \"wide_variable_1\":\"Competition Metric\"},\n range_x=(0.0, 1.0), range_y=(0.0, 1.0), \n title=\"Threshold v. Model Performance (F1 & Comp)\",\n )\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-03-25T18:38:55.476607Z","iopub.execute_input":"2022-03-25T18:38:55.476888Z","iopub.status.idle":"2022-03-25T18:38:55.548525Z","shell.execute_reply.started":"2022-03-25T18:38:55.476854Z","shell.execute_reply":"2022-03-25T18:38:55.547694Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# TOP COMP SCORE - THRESH=0.9874\n# LAST STABLE - THRESH=0.95-0.96\n\npred_sps_arrs, gt_sps_arrs = get_species_arrs(viz_ss_df.target.to_numpy(), viz_ss_df.gt_lbl.to_numpy(), thresh=0.95)\nprint(sklearn.metrics.f1_score(gt_sps_arrs[:, :-1], pred_sps_arrs[:, :-1], average='macro'))\nprint(comp_metric(gt_sps_arrs[:, :-1], pred_sps_arrs[:, :-1]))","metadata":{"execution":{"iopub.status.busy":"2022-03-25T18:42:40.902195Z","iopub.execute_input":"2022-03-25T18:42:40.902887Z","iopub.status.idle":"2022-03-25T18:42:41.310018Z","shell.execute_reply.started":"2022-03-25T18:42:40.902824Z","shell.execute_reply":"2022-03-25T18:42:41.309233Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(pred_sps_arrs.sum(axis=0))\nprint()\nprint(gt_sps_arrs.sum(axis=0))","metadata":{"execution":{"iopub.status.busy":"2022-03-25T18:42:45.991428Z","iopub.execute_input":"2022-03-25T18:42:45.992441Z","iopub.status.idle":"2022-03-25T18:42:46.001387Z","shell.execute_reply.started":"2022-03-25T18:42:45.992387Z","shell.execute_reply":"2022-03-25T18:42:46.00066Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"SCORED_BIRD_STRS[11]","metadata":{"execution":{"iopub.status.busy":"2022-03-25T18:42:30.986446Z","iopub.execute_input":"2022-03-25T18:42:30.986986Z","iopub.status.idle":"2022-03-25T18:42:30.995892Z","shell.execute_reply.started":"2022-03-25T18:42:30.986944Z","shell.execute_reply":"2022-03-25T18:42:30.99471Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(pred_sps_arrs.sum(axis=0))\nprint()\nprint(gt_sps_arrs.sum(axis=0))","metadata":{"execution":{"iopub.status.busy":"2022-03-25T18:41:10.660181Z","iopub.execute_input":"2022-03-25T18:41:10.660449Z","iopub.status.idle":"2022-03-25T18:41:10.66707Z","shell.execute_reply.started":"2022-03-25T18:41:10.660419Z","shell.execute_reply":"2022-03-25T18:41:10.666185Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def comp_metric(y_true, y_pred, epsilon=1e-9):\n \"\"\" Function to calculate competition metric in an sklearn like fashion\n\n Args:\n y_true{array-like, sparse matrix} of shape (n_samples, n_outputs)\n - Ground truth (correct) target values.\n y_pred{array-like, sparse matrix} of shape (n_samples, n_outputs)\n - Estimated targets as returned by a classifier.\n Returns:\n The single calculated score representative of this competitions evaluation\n \"\"\"\n \n # Get representative confusion matrices for each label\n mlbl_cms = sklearn.metrics.multilabel_confusion_matrix(y_true, y_pred)\n\n # Get two scores (TP and TN SCORES)\n tp_scores = np.array([\n mlbl_cm[1, 1]/(epsilon+mlbl_cm[:, 1].sum()) \\\n for mlbl_cm in mlbl_cms\n ])\n \n tn_scores = np.array([\n mlbl_cm[0, 0]/(epsilon+mlbl_cm[:, 0].sum()) \\\n for mlbl_cm in mlbl_cms\n ])\n\n print(tp_scores)\n print(tn_scores)\n \n # Get average\n tp_mean = tp_scores.mean()\n tn_mean = tn_scores.mean()\n\n return round((tp_mean+tn_mean)/2, 8)\n\nprint(comp_metric(gt_sps_arrs[:, :-1], pred_sps_arrs[:, :-1]))","metadata":{"execution":{"iopub.status.busy":"2022-03-26T13:46:07.183335Z","iopub.execute_input":"2022-03-26T13:46:07.183875Z","iopub.status.idle":"2022-03-26T13:46:07.208554Z","shell.execute_reply.started":"2022-03-26T13:46:07.183833Z","shell.execute_reply":"2022-03-26T13:46:07.207844Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"\\n\\n\\nSTARTING TEST w/ THRESH={_thresh}%\\n\")\npred_sps_arrs, gt_sps_arrs = get_species_arrs(viz_ss_df.target.to_numpy(), viz_ss_df.gt_lbl.to_numpy(), thresh=0.99647)\n\nprint(f\"\\n\\t--> MACRO F1 SCORE: {sklearn.metrics.f1_score(gt_sps_arrs[:, -1], pred_sps_arrs[:, -1], average='macro')}\")\nprint(f\"\\t--> CLASSWISE COMPETITION METRIC ACCURACY: {comp_metric(gt_sps_arrs[:, -1], pred_sps_arrs[:, -1])}\\n\")","metadata":{"execution":{"iopub.status.busy":"2022-03-25T18:17:25.457357Z","iopub.execute_input":"2022-03-25T18:17:25.457621Z","iopub.status.idle":"2022-03-25T18:17:25.828826Z","shell.execute_reply.started":"2022-03-25T18:17:25.457591Z","shell.execute_reply":"2022-03-25T18:17:25.82786Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for _thresh in tqdm(range(950, 1005, 5), total=10):\n print(f\"\\n\\n\\nSTARTING TEST w/ THRESH={_thresh}%\\n\")\n pred_sps_arrs, gt_sps_arrs = get_species_arrs(viz_ss_df.target.to_numpy(), viz_ss_df.gt_lbl.to_numpy(), thresh=_thresh/1000)\n\n print(f\"\\n\\t--> MACRO F1 SCORE: {sklearn.metrics.f1_score(gt_sps_arrs[:, -1], pred_sps_arrs[:, -1], average='macro')}\")\n print(f\"\\t--> CLASSWISE COMPETITION METRIC ACCURACY: {comp_metric(gt_sps_arrs[:, -1], pred_sps_arrs[:, -1])}\\n\")","metadata":{"execution":{"iopub.status.busy":"2022-03-25T18:13:19.847964Z","iopub.execute_input":"2022-03-25T18:13:19.848988Z","iopub.status.idle":"2022-03-25T18:14:34.707595Z","shell.execute_reply.started":"2022-03-25T18:13:19.84893Z","shell.execute_reply":"2022-03-25T18:14:34.706876Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Below are some examples of scored species**\n* Notice that even in padded audio my model still predicts with some confidence... not good.","metadata":{}},{"cell_type":"code","source":"for _, _row in train_df[train_df.primary_label.isin(SCORED_BIRD_STRS)].groupby(\"primary_label\").sample(1).reset_index(drop=True).iterrows():\n xc_id = _row[\"filename\"].split(\"/\")[-1][:-4]\n print(f\"\\n\\n\\tRANDOM EXAMPLE FOR SCORED SPECIES --> {_row['primary_label'].upper()} ({xc_id}) \\n\")\n display(ipd.Audio(_row.f_path))\n display(pd.DataFrame(_row).T)\n \n sub_viz_df = get_sub_pred_df(viz_ss_df, _row_id=xc_id)\n plot_clip_preds(sub_viz_df, xc_id)","metadata":{"execution":{"iopub.status.busy":"2022-03-27T23:15:03.831581Z","iopub.execute_input":"2022-03-27T23:15:03.831844Z","iopub.status.idle":"2022-03-27T23:15:55.912296Z","shell.execute_reply.started":"2022-03-27T23:15:03.831815Z","shell.execute_reply":"2022-03-27T23:15:55.911599Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Below are some examples of non-scored species**\n\n* As you can see my model is very prone to false-positives...","metadata":{}},{"cell_type":"code","source":"for i, (_, _row) in enumerate(train_df[~train_df.primary_label.isin(SCORED_BIRD_STRS)].groupby(\"primary_label\").sample(1).reset_index(drop=True).iterrows()):\n if i==10: break\n xc_id = _row[\"filename\"].split(\"/\")[-1][:-4]\n print(f\"\\n\\n\\tRANDOM EXAMPLE FOR NON-SCORED SPECIES --> {_row['primary_label'].upper()} ({xc_id}) \\n\")\n display(ipd.Audio(_row.f_path))\n display(pd.DataFrame(_row).T)\n sub_viz_df = get_sub_pred_df(viz_ss_df, _row_id=xc_id)\n plot_clip_preds(sub_viz_df, xc_id)","metadata":{"execution":{"iopub.status.busy":"2022-03-27T23:20:36.386952Z","iopub.execute_input":"2022-03-27T23:20:36.387570Z","iopub.status.idle":"2022-03-27T23:20:52.086851Z","shell.execute_reply.started":"2022-03-27T23:20:36.387532Z","shell.execute_reply":"2022-03-27T23:20:52.086152Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Here's an example comparing the predictions for a clip containing a Canada Goose vs. a Hawaiian Goose**\n* This is just a simple example of the type of comparison you can do... obviously the functionality can be expanded","metadata":{}},{"cell_type":"code","source":"# ex_can_goose = \"XC609624\"\n# ex_haw_goose = \"XC326899\"\n\n# _row = train_df[train_df.filename.str.contains(ex_can_goose)].reset_index(drop=True).iloc[0]\n# xc_id = _row[\"filename\"].split(\"/\")[-1][:-4]\n# print(f\"\\n\\n\\tCANADA GOOSE EXAMPLE --> {_row['primary_label'].upper()} ({xc_id}) \\n\")\n# display(ipd.Audio(_row.f_path))\n# display(pd.DataFrame(_row).T)\n# sub_viz_df = get_sub_pred_df(viz_ss_df, _row_id=xc_id)\n# plot_clip_preds(sub_viz_df, xc_id)\n\n# _row = train_df[train_df.filename.str.contains(ex_haw_goose)].reset_index(drop=True).iloc[0]\n# xc_id = _row[\"filename\"].split(\"/\")[-1][:-4]\n# print(f\"\\n\\n\\tHAWAIIAN GOOSE EXAMPLE --> {_row['primary_label'].upper()} ({xc_id}) \\n\")\n# display(ipd.Audio(_row.f_path))\n# display(pd.DataFrame(_row).T)\n# sub_viz_df = get_sub_pred_df(viz_ss_df, _row_id=xc_id)\n# plot_clip_preds(sub_viz_df, xc_id)","metadata":{"execution":{"iopub.status.busy":"2022-03-25T17:52:13.196738Z","iopub.status.idle":"2022-03-25T17:52:13.197419Z","shell.execute_reply.started":"2022-03-25T17:52:13.197159Z","shell.execute_reply":"2022-03-25T17:52:13.197184Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Bool Version Of Visualization**","metadata":{}},{"cell_type":"code","source":"BOOL_THRESH = 0.75\nviz_ss_df.target = viz_ss_df.target>BOOL_THRESH\nviz_ss_df","metadata":{"execution":{"iopub.status.busy":"2022-03-25T17:52:13.198752Z","iopub.status.idle":"2022-03-25T17:52:13.199189Z","shell.execute_reply.started":"2022-03-25T17:52:13.198968Z","shell.execute_reply":"2022-03-25T17:52:13.19899Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for _, _row in train_df[train_df.primary_label.isin(SCORED_BIRD_STRS)].groupby(\"primary_label\").sample(1).reset_index(drop=True).iterrows():\n xc_id = _row[\"filename\"].split(\"/\")[-1][:-4]\n print(f\"\\n\\n\\tRANDOM EXAMPLE FOR SCORED SPECIES --> {_row['primary_label'].upper()} ({xc_id}) \\n\")\n display(ipd.Audio(_row.f_path))\n display(pd.DataFrame(_row).T)\n sub_viz_df = get_sub_pred_df(viz_ss_df, _row_id=xc_id)\n plot_clip_preds(sub_viz_df, xc_id)","metadata":{"execution":{"iopub.status.busy":"2022-03-25T17:52:13.200633Z","iopub.status.idle":"2022-03-25T17:52:13.201066Z","shell.execute_reply.started":"2022-03-25T17:52:13.200821Z","shell.execute_reply":"2022-03-25T17:52:13.200869Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}