{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Tensorflow Speech Recognition Challenge\n\n\nFormulation of the problem:\n\nWe are given a dataset of 64721 <=1 sec wav files, that should be classified into one of [yes, no, up, down, left, right, on, off, stop, go, silence, unknown]/. Most of the files are \"unknown\" in terms of these labels\nI think that the dataset of this audio length and the number of classes to predict can be treated as really simple for modern speech recognition. Probably SOTA networks can classify it almost perfectly. But let's assume we are solving a real problem: a simple set of words can be used for voice control of some application, and additionally the person who gave me this task, wrote that solution on pure TensorFlow is preferable. It means that created solution probably will be ported into another platform with TensorFlow support. Looks like our speech recognition system will be run locally in a browser or maybe a mobile application, so let's include computational effectiveness and prediction speed in our list of priorities.\n\nAdditionally, I would say that this is my first experience in speech recognition and I do not have a background in signal processing, so this task is especially interesting for me.\n\n

Table of Contents

\n
\n \n
","metadata":{}},{"cell_type":"markdown","source":"# Preparation\n\nIn this section I'm performing routine work with new data such as checking classes balance, defining train-val-test split and creating new silence and reducing unknown labels. Finally, all the data was moved into new folders, which is easy to load into any machine learning framework.","metadata":{}},{"cell_type":"code","source":"%%capture\n# unzip data\n!apt install --assume-yes p7zip-full\n!mkdir /kaggle/temp; cd /kaggle/temp && 7z x /kaggle/input/tensorflow-speech-recognition-challenge/train.7z","metadata":{"execution":{"iopub.status.busy":"2021-09-10T10:03:04.727310Z","iopub.execute_input":"2021-09-10T10:03:04.727692Z","iopub.status.idle":"2021-09-10T10:05:03.980862Z","shell.execute_reply.started":"2021-09-10T10:03:04.727661Z","shell.execute_reply":"2021-09-10T10:05:03.979766Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Metadata exploration","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom scipy.io import wavfile\nimport IPython.display as ipd\nimport os","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-09-10T10:05:03.983799Z","iopub.execute_input":"2021-09-10T10:05:03.984155Z","iopub.status.idle":"2021-09-10T10:05:03.992310Z","shell.execute_reply.started":"2021-09-10T10:05:03.984110Z","shell.execute_reply":"2021-09-10T10:05:03.989764Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"DATA_PATH = '/kaggle/temp/train'\nprint(os.listdir(DATA_PATH))","metadata":{"execution":{"iopub.status.busy":"2021-09-10T10:05:03.994431Z","iopub.execute_input":"2021-09-10T10:05:03.994972Z","iopub.status.idle":"2021-09-10T10:05:04.006068Z","shell.execute_reply.started":"2021-09-10T10:05:03.994937Z","shell.execute_reply":"2021-09-10T10:05:04.005146Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"audio_path = os.path.join(DATA_PATH, 'audio')\n\ndata_labels = os.listdir(audio_path)\ndata_labels.remove('_background_noise_')\nprint(f'Number of labels: {len(data_labels)}')\n\nLABELS = ['yes', 'no', 'up', 'down', 'left', 'right',\n 'on', 'off', 'stop', 'go', 'silence', 'unknown']\n\ndata_file_labels = dict()\nfor label in data_labels:\n files = os.listdir(audio_path + '/' + label)\n for f in files:\n data_file_labels[label + '/' + f] = label\n\nmeta_df = pd.DataFrame.from_dict(data_file_labels, orient='index')\nmeta_df = meta_df.reset_index(drop=False)\nmeta_df = meta_df.rename(columns={'index': 'filepath', 0: 'folder'})\nmeta_df = meta_df[['folder', 'filepath']]\nmeta_df = meta_df.sort_values('filepath')\nmeta_df = meta_df.reset_index(drop=True)\nprint(meta_df.shape)\n\ndef remove_label_from_file(label, fname):\n return fname[len(label)+1:]\n\nmeta_df['file'] = meta_df.apply(lambda x: remove_label_from_file(*x), axis=1)\nmeta_df[['speaker', 'utterance']] = meta_df['file'].str.split('_', expand=True)[[0, 2]]\nmeta_df['label'] = meta_df['folder'].apply(lambda x: x if x in LABELS else 'unknown')\n\ndef get_wv_metadata(filename):\n bitrate, data = wavfile.read(filename)\n return bitrate, len(data)\n\npaths = meta_df['filepath'].apply(lambda x: os.path.join(audio_path, x))\nmeta_df[['bitrate', 'length']] = paths.apply(get_wv_metadata).to_list()\nmeta_df.sample(5)","metadata":{"execution":{"iopub.status.busy":"2021-09-10T10:05:04.007845Z","iopub.execute_input":"2021-09-10T10:05:04.008513Z","iopub.status.idle":"2021-09-10T10:05:08.927833Z","shell.execute_reply.started":"2021-09-10T10:05:04.008439Z","shell.execute_reply":"2021-09-10T10:05:08.926874Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Bitrate and length","metadata":{}},{"cell_type":"code","source":"meta_df.describe(include=np.number)","metadata":{"execution":{"iopub.status.busy":"2021-09-10T10:05:08.929352Z","iopub.execute_input":"2021-09-10T10:05:08.929729Z","iopub.status.idle":"2021-09-10T10:05:08.964064Z","shell.execute_reply.started":"2021-09-10T10:05:08.929694Z","shell.execute_reply":"2021-09-10T10:05:08.962624Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have the same bitrate, but length sometimes is less than 1 sec. I'm going to fix that during preprocessing stage.","metadata":{}},{"cell_type":"markdown","source":"#### Validation and training lists\nWe are already given with lists of validation and testing samples. Let's check if they are constructed in an appropriate way, with respect to speakers. It is important because our model can overfit to speaker's voice or background sounds, and we should be able to catch sight of it.","metadata":{}},{"cell_type":"code","source":"validation_list = pd.read_csv(os.path.join(DATA_PATH, 'validation_list.txt'), header=None, squeeze=True).to_list()\ntesting_list = pd.read_csv(os.path.join(DATA_PATH, 'testing_list.txt'), header=None, squeeze=True).to_list()\n\nmeta_df['validation'] = meta_df['filepath'].isin(validation_list).astype('int')\nmeta_df['test'] = meta_df['filepath'].isin(testing_list).astype('int')\n\ntrain_idx = (meta_df['validation']==0) & (meta_df['test']==0)\nval_idx = (meta_df['validation']==1)\ntest_idx = (meta_df['test']==1)","metadata":{"execution":{"iopub.status.busy":"2021-09-10T10:05:08.972132Z","iopub.execute_input":"2021-09-10T10:05:08.973477Z","iopub.status.idle":"2021-09-10T10:05:09.126963Z","shell.execute_reply.started":"2021-09-10T10:05:08.973358Z","shell.execute_reply":"2021-09-10T10:05:09.126134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"meta_df[val_idx & test_idx]","metadata":{"execution":{"iopub.status.busy":"2021-09-10T10:05:09.130100Z","iopub.execute_input":"2021-09-10T10:05:09.130340Z","iopub.status.idle":"2021-09-10T10:05:09.144575Z","shell.execute_reply.started":"2021-09-10T10:05:09.130317Z","shell.execute_reply":"2021-09-10T10:05:09.143792Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_speakers = set(meta_df.loc[train_idx, ['folder', 'speaker']].sum(axis=1))\nval_speakers = set(meta_df.loc[val_idx, ['folder', 'speaker']].sum(axis=1))\ntest_speakers = set(meta_df.loc[test_idx, ['folder', 'speaker']].sum(axis=1))\n\ndef set_intersect(left, right):\n '''A∩B = A\\(A\\B)'''\n return left - (left - right)\n\nlen(set_intersect(train_speakers, val_speakers)), len(set_intersect(train_speakers, test_speakers)), len(set_intersect(val_speakers, test_speakers))","metadata":{"execution":{"iopub.status.busy":"2021-09-10T10:05:09.147164Z","iopub.execute_input":"2021-09-10T10:05:09.147744Z","iopub.status.idle":"2021-09-10T10:05:09.216052Z","shell.execute_reply.started":"2021-09-10T10:05:09.147706Z","shell.execute_reply":"2021-09-10T10:05:09.215048Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Great, speakers sets do not intersect, that means we can use this predefined split for model validation.","metadata":{}},{"cell_type":"markdown","source":"#### Classes balance","metadata":{}},{"cell_type":"code","source":"class_counts = pd.DataFrame([\n meta_df.loc[train_idx, 'label'].value_counts(),\n meta_df.loc[val_idx, 'label'].value_counts(),\n meta_df.loc[test_idx, 'label'].value_counts()\n], index=['train', 'validation', 'test'])\nclass_counts","metadata":{"execution":{"iopub.status.busy":"2021-09-10T10:05:09.217926Z","iopub.execute_input":"2021-09-10T10:05:09.218287Z","iopub.status.idle":"2021-09-10T10:05:09.251013Z","shell.execute_reply.started":"2021-09-10T10:05:09.218250Z","shell.execute_reply":"2021-09-10T10:05:09.250091Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class_counts.sum(axis=1)","metadata":{"execution":{"iopub.status.busy":"2021-09-10T10:05:09.252354Z","iopub.execute_input":"2021-09-10T10:05:09.252706Z","iopub.status.idle":"2021-09-10T10:05:09.259630Z","shell.execute_reply.started":"2021-09-10T10:05:09.252671Z","shell.execute_reply":"2021-09-10T10:05:09.258596Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"mean_class_samples = class_counts.drop(columns='unknown').mean(axis=1).round().astype('int')\n\nmean_class_samples","metadata":{"execution":{"iopub.status.busy":"2021-09-10T10:05:09.261094Z","iopub.execute_input":"2021-09-10T10:05:09.261429Z","iopub.status.idle":"2021-09-10T10:05:09.272631Z","shell.execute_reply.started":"2021-09-10T10:05:09.261396Z","shell.execute_reply":"2021-09-10T10:05:09.271392Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Classes have a similar number of samples, except 'unknown', which contains all the other words that we do not need to classify.\n\nAlso the \"silence\" class should be created, let's check the \\_background_noise_ folder","metadata":{}},{"cell_type":"code","source":"bg_noise = os.path.join(audio_path, '_background_noise_')\nsilence_filepath = os.listdir(bg_noise)\nsilence_filepath = [x for x in silence_filepath if x[-3:]=='wav']\nnoise_df = pd.DataFrame([get_wv_metadata(os.path.join(bg_noise, x)) for x in silence_filepath], columns=['bitrate', 'length'])\nnoise_df['filename'] = silence_filepath\nnoise_df","metadata":{"execution":{"iopub.status.busy":"2021-09-10T10:05:09.274138Z","iopub.execute_input":"2021-09-10T10:05:09.274544Z","iopub.status.idle":"2021-09-10T10:05:09.297877Z","shell.execute_reply.started":"2021-09-10T10:05:09.274503Z","shell.execute_reply":"2021-09-10T10:05:09.296775Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It's hard, to choose validation split for this class. For example, let's split each file into three disjoint parts (train-val-test, 80:10:10), and the create a samples with the moving window","metadata":{}},{"cell_type":"markdown","source":"## Data preparation\nMove validation and test into separate directories, create 'silence' samples, thin out 'unknown'","metadata":{}},{"cell_type":"code","source":"def sample_unknown(df, num_leave, random_state=567):\n '''\n Reduce number of rows with label=='unknown'\n uniformly sampling from the unknown-labeled folders.\n \n df - dataframe with metadata\n num_leave - how many samples should be chosen\n random_state - random sedd for pd.DataFrame.sample\n \n Returns np.array of df indicies\n '''\n redundant_labels = df.loc[df['label']=='unknown', 'folder'].unique()\n num_leave_per_label = num_leave // len(redundant_labels) + 1\n indexlist = [df[df['folder']==l].sample(num_leave_per_label, random_state=random_state).index for l in redundant_labels]\n return np.array(indexlist).ravel()\n\nidx_to_keep = np.hstack([\n sample_unknown(meta_df[train_idx], mean_class_samples['train']),\n sample_unknown(meta_df[val_idx], mean_class_samples['validation']),\n sample_unknown(meta_df[test_idx], mean_class_samples['test']),\n meta_df[meta_df['label']!='unknown'].index\n])\nmeta_df = meta_df.loc[idx_to_keep]","metadata":{"execution":{"iopub.status.busy":"2021-09-10T10:05:09.299429Z","iopub.execute_input":"2021-09-10T10:05:09.299829Z","iopub.status.idle":"2021-09-10T10:05:09.582784Z","shell.execute_reply.started":"2021-09-10T10:05:09.299791Z","shell.execute_reply":"2021-09-10T10:05:09.581946Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import itertools\nfrom pathlib import Path\n\nDATASET_PATH = '/kaggle/temp/data'\nTRAIN_DATASET_PATH = os.path.join(DATASET_PATH, 'train')\nVAL_DATASET_PATH = os.path.join(DATASET_PATH, 'validation')\nTEST_DATASET_PATH = os.path.join(DATASET_PATH, 'test')\n\nfor dataset, label in itertools.product([TRAIN_DATASET_PATH, VAL_DATASET_PATH, TEST_DATASET_PATH], LABELS):\n Path(os.path.join(dataset, label)).mkdir(parents=True, exist_ok=True)","metadata":{"execution":{"iopub.status.busy":"2021-09-10T10:05:09.584125Z","iopub.execute_input":"2021-09-10T10:05:09.584503Z","iopub.status.idle":"2021-09-10T10:05:09.595582Z","shell.execute_reply.started":"2021-09-10T10:05:09.584436Z","shell.execute_reply":"2021-09-10T10:05:09.594783Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Move files into new directories\ndef move_files(df, dataset_dir):\n for i, row in df.iterrows():\n cur_path = os.path.join(audio_path, row['filepath'])\n new_path = os.path.join(dataset_dir, row['label'], f\"{i}_{row['file']}\")\n os.replace(cur_path, new_path)\n\nmove_files(meta_df[(meta_df['validation']==0) & (meta_df['test']==0)], TRAIN_DATASET_PATH)\nmove_files(meta_df[meta_df['validation']==1], VAL_DATASET_PATH)\nmove_files(meta_df[meta_df['test']==1], TEST_DATASET_PATH)","metadata":{"execution":{"iopub.status.busy":"2021-09-10T10:05:09.596591Z","iopub.execute_input":"2021-09-10T10:05:09.596935Z","iopub.status.idle":"2021-09-10T10:05:13.019589Z","shell.execute_reply.started":"2021-09-10T10:05:09.596901Z","shell.execute_reply":"2021-09-10T10:05:13.018667Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_silence_path = os.path.join(TRAIN_DATASET_PATH, 'silence')\nval_silence_path = os.path.join(VAL_DATASET_PATH, 'silence')\ntest_silence_path = os.path.join(TEST_DATASET_PATH, 'silence')\n\nnum_samples_per_label = mean_class_samples//len(noise_df) + 1\n\ndef create_wav_samples(data, bitrate, num_samples, path):\n for x in np.linspace(0, len(data) - bitrate - 1, num_samples, dtype='int'):\n wavfile.write(os.path.join(path, str(x)+filename), bitrate, data[x: x+bitrate])\n\nfor filename in noise_df['filename']:\n bitrate, data = wavfile.read(os.path.join(bg_noise, filename))\n split1 = int(len(data)*0.8)\n split2 = int(len(data)*0.9)\n create_wav_samples(data[:split1], bitrate, num_samples_per_label['train'], train_silence_path)\n create_wav_samples(data[split1: split2], bitrate, num_samples_per_label['validation'], val_silence_path)\n create_wav_samples(data[split2:], bitrate, num_samples_per_label['test'], test_silence_path)","metadata":{"execution":{"iopub.status.busy":"2021-09-10T10:05:13.020967Z","iopub.execute_input":"2021-09-10T10:05:13.021301Z","iopub.status.idle":"2021-09-10T10:05:13.306502Z","shell.execute_reply.started":"2021-09-10T10:05:13.021265Z","shell.execute_reply":"2021-09-10T10:05:13.305663Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!rm -rf $DATA_PATH\ndel meta_df","metadata":{"execution":{"iopub.status.busy":"2021-09-10T10:05:13.307647Z","iopub.execute_input":"2021-09-10T10:05:13.308656Z","iopub.status.idle":"2021-09-10T10:05:15.055200Z","shell.execute_reply.started":"2021-09-10T10:05:13.308616Z","shell.execute_reply":"2021-09-10T10:05:15.054164Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we have nice directory hierarchy and the balanced dataset","metadata":{}},{"cell_type":"code","source":"!tree -d $DATASET_PATH","metadata":{"execution":{"iopub.status.busy":"2021-09-09T21:51:55.638831Z","iopub.execute_input":"2021-09-09T21:51:55.63947Z","iopub.status.idle":"2021-09-09T21:51:56.592834Z","shell.execute_reply.started":"2021-09-09T21:51:55.63943Z","shell.execute_reply":"2021-09-09T21:51:56.591932Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Modeling\n\n## Approaches overview:\nFrom speech recognition approaches two main groups can be distinguished:\n\n_Classic speech_ recognition approaches - Hidden Markov Models, Gaussian Mixtures, other statistical models. Fast&cheap, don't require GPU for training and running. Cons: they require hard data preprocessing and feature creation based on signal decomposition, and understanding of acoustic models. In short - you should really understand the field to apply them. Also, they seem to be inaccurate on noisy data. Takes a similar place in speech recognition as non-deep methods in computer vision.\n\n_Deep learning_. Typically deep learning is applied in an end-to-end manner, you give a sound - you receive a text. No acoustic and language models are required. Data is preprocessed with some transformations (e.g. Fourier - STFT (spectrogram)) into 2D input. From papers published in 2014-2016, we can imagine a typical deep speech recognition model as the sequence-to-sequence one that has convolutional layers at the beginning to catch whole sounds, then some recurrent layers, and the layer classifying the actual alphabet character of the input. Received prediction like \"c-c--a-a-a--t-t\" is mapped to actual word by combining adjacent characters (This is a part of CTC loss term).\n\nWith neural networks, we have two main ways: first - find and try (and, maybe use transfer learning, finetuning) a big pretrained model, second - develop and train a new one from scratch.\nI think that training a RNN with CTC loss is a too complicated approach for such a simple task and a small dataset - we simply do not have the required variety in recordings sounds and the model leads to overfitting because some of them appear only in specific combinations.\n\nBut actually, we don't even need a speech recognition model. Because of almost the same recordings length, we can simply convert them into equal-sized spectrograms and train a convolutional classifier of the whole word - which we can do because we need to classify only 13 of them. I believe that the classification of a single word is a real-world case, for example, voice control like voice dialing in old phones, so this approach is not a competition hacking.","metadata":{}},{"cell_type":"markdown","source":"## Preprocessing Pipeline","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow.audio import decode_wav\nfrom tensorflow import signal # here is the Short-time Fourier Transform used to create a spectrograms\n\nAUTOTUNE = tf.data.AUTOTUNE","metadata":{"execution":{"iopub.status.busy":"2021-09-10T10:05:42.171507Z","iopub.execute_input":"2021-09-10T10:05:42.171865Z","iopub.status.idle":"2021-09-10T10:05:46.482595Z","shell.execute_reply.started":"2021-09-10T10:05:42.171831Z","shell.execute_reply":"2021-09-10T10:05:46.481719Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def decode_audio(audio_binary):\n audio, _ = tf.audio.decode_wav(audio_binary)\n return tf.squeeze(audio, axis=-1)\n\n\ndef get_label(file_path):\n parts = tf.strings.split(file_path, os.path.sep)\n return parts[-2]\n\n\ndef get_waveform_and_label(file_path):\n label = get_label(file_path)\n audio_binary = tf.io.read_file(file_path)\n waveform = decode_audio(audio_binary)\n return waveform, label\n\n\ndef get_spectrogram(waveform):\n # Padding for files with less than 16000 samples\n zero_padding = tf.zeros([16000] - tf.shape(waveform), dtype=tf.float32)\n waveform = tf.cast(waveform, tf.float32)\n equal_length = tf.concat([waveform, zero_padding], 0)\n spectrogram = tf.signal.stft(\n equal_length, frame_length=255, frame_step=128\n )\n\n spectrogram = tf.abs(spectrogram)\n return spectrogram\n\n\ndef get_spectrogram_and_label_id(audio, label):\n spectrogram = get_spectrogram(audio)\n spectrogram = tf.expand_dims(spectrogram, -1)\n label_id = tf.argmax(label == LABELS)\n return spectrogram, label_id\n\n\ndef load_and_preprocess_dataset(dataset_glob):\n # actually, data loading in pytorch seem to be much more convenient for me\n filenames = tf.io.gfile.glob(dataset_glob)\n files_ds = tf.data.Dataset.from_tensor_slices(filenames)\n waveform_ds = files_ds.map(get_waveform_and_label, num_parallel_calls=AUTOTUNE)\n spectrogram_ds = waveform_ds.map(\n get_spectrogram_and_label_id, num_parallel_calls=AUTOTUNE)\n return spectrogram_ds","metadata":{"execution":{"iopub.status.busy":"2021-09-10T10:05:46.483987Z","iopub.execute_input":"2021-09-10T10:05:46.484320Z","iopub.status.idle":"2021-09-10T10:05:46.496820Z","shell.execute_reply.started":"2021-09-10T10:05:46.484285Z","shell.execute_reply":"2021-09-10T10:05:46.496130Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"DATASET_PATH = '/kaggle/temp/data'\nTRAIN_DATASET_PATH = os.path.join(DATASET_PATH, 'train')\nVAL_DATASET_PATH = os.path.join(DATASET_PATH, 'validation')\nTEST_DATASET_PATH = os.path.join(DATASET_PATH, 'test')\nLABELS = np.array(['yes', 'no', 'up', 'down', 'left', 'right',\n 'on', 'off', 'stop', 'go', 'silence', 'unknown'])\n\ntrain_ds = load_and_preprocess_dataset(TRAIN_DATASET_PATH + '/*/*')\nval_ds = load_and_preprocess_dataset(VAL_DATASET_PATH + '/*/*')\ntest_ds = load_and_preprocess_dataset(TEST_DATASET_PATH + '/*/*')","metadata":{"execution":{"iopub.status.busy":"2021-09-10T10:05:46.498310Z","iopub.execute_input":"2021-09-10T10:05:46.498919Z","iopub.status.idle":"2021-09-10T10:05:49.233065Z","shell.execute_reply.started":"2021-09-10T10:05:46.498869Z","shell.execute_reply":"2021-09-10T10:05:49.232220Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\ndef plot_spectrogram(spectrogram, ax):\n # Convert to frequencies to log scale and transpose so that the time is\n # represented in the x-axis (columns).\n log_spec = np.log(spectrogram.T)[::-1]\n ax.imshow(log_spec)\n\nrows = 3\ncols = 3\nn = rows*cols\nfig, axes = plt.subplots(rows, cols, figsize=(10, 10))\nfor i, (spectrogram, label_id) in enumerate(train_ds.take(n)):\n r = i // cols\n c = i % cols\n ax = axes[r][c]\n spectrogram = tf.image.resize(spectrogram, (32, 32))\n plot_spectrogram(np.squeeze(spectrogram.numpy()), ax)\n ax.set_title(LABELS[label_id.numpy()])\n ax.axis('off')\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-09-10T10:05:49.234546Z","iopub.execute_input":"2021-09-10T10:05:49.234899Z","iopub.status.idle":"2021-09-10T10:05:50.152075Z","shell.execute_reply.started":"2021-09-10T10:05:49.234865Z","shell.execute_reply":"2021-09-10T10:05:50.151196Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Nearest neighbors by spectrogram\nThe are similar patterns on same words spectrograms even by eye. So I decide to start with such simple method as kNN.\n\nIt gives 0.566 accuracy, while random guessing baseline is 1/12 = 0.083. Not bad.\n\nI believe if spend some time on feature engineering with signal transformations (with different parameters) - score of this simple model can be hightly improved.","metadata":{}},{"cell_type":"code","source":"%%time\n# I already created TF datasets, but it is much simpler to take knn from sklearn\ndef spectrogram_dataset_to_numpy(dataset):\n spectrograms = []\n labels = []\n for x, y in dataset:\n spectrograms.append(tf.image.resize(x, (32, 32)))\n labels.append(y)\n spectrograms = np.array(spectrograms).reshape(-1, 32*32)\n labels = np.array(labels)\n return spectrograms, labels\n\ntrain_spectrograms, train_labels = spectrogram_dataset_to_numpy(train_ds)\nval_spectrograms, val_labels = spectrogram_dataset_to_numpy(val_ds)\ntest_spectrograms, test_labels = spectrogram_dataset_to_numpy(test_ds)","metadata":{"execution":{"iopub.status.busy":"2021-09-09T21:52:04.144235Z","iopub.execute_input":"2021-09-09T21:52:04.144487Z","iopub.status.idle":"2021-09-09T21:53:11.51917Z","shell.execute_reply.started":"2021-09-09T21:52:04.144461Z","shell.execute_reply":"2021-09-09T21:53:11.518238Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.neighbors import KNeighborsClassifier\nfrom sklearn.metrics import accuracy_score\n\nscores = []\nks = np.arange(5, 55, 10)\nfor k in ks:\n knn = KNeighborsClassifier(k, metric='cosine', n_jobs=-1)\n knn.fit(train_spectrograms, train_labels)\n knn_pred = knn.predict(val_spectrograms)\n scores.append(accuracy_score(val_labels, knn_pred))\n \nplt.plot(ks, scores)\nprint(f'max validation accuracy: {np.max(scores)}')","metadata":{"execution":{"iopub.status.busy":"2021-09-09T21:53:11.521005Z","iopub.execute_input":"2021-09-09T21:53:11.521527Z","iopub.status.idle":"2021-09-09T21:53:26.651172Z","shell.execute_reply.started":"2021-09-09T21:53:11.521487Z","shell.execute_reply":"2021-09-09T21:53:26.650353Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"knn = KNeighborsClassifier(25, metric='cosine', n_jobs=-1)\nknn.fit(train_spectrograms, train_labels)\nknn_pred = knn.predict(test_spectrograms)\nprint('test accuracy', accuracy_score(test_labels, knn_pred))","metadata":{"execution":{"iopub.status.busy":"2021-09-09T21:53:26.652568Z","iopub.execute_input":"2021-09-09T21:53:26.652966Z","iopub.status.idle":"2021-09-09T21:53:29.441484Z","shell.execute_reply.started":"2021-09-09T21:53:26.652926Z","shell.execute_reply":"2021-09-09T21:53:29.440597Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del train_spectrograms, train_labels, val_spectrograms, val_labels, test_spectrograms, test_labels","metadata":{"execution":{"iopub.status.busy":"2021-09-09T21:53:29.443023Z","iopub.execute_input":"2021-09-09T21:53:29.443635Z","iopub.status.idle":"2021-09-09T21:53:29.448209Z","shell.execute_reply.started":"2021-09-09T21:53:29.443594Z","shell.execute_reply":"2021-09-09T21:53:29.447398Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## CNN from scratch (Keras)\nInspired by TensorFlow Tutorial [Simple audio recognition: Recognizing keywords](https://www.tensorflow.org/tutorials/audio/simple_audio), I also borroved some helping function from it.\n\nAudios are converted to spectrograms with STFT (I took parameters from the tutorial), then spectrograms resized to 32x32 to reduce the feature space then used as input for CNN. Classifier constructed in a standart way - with 3x3 convolutional kernels and max pooling, then single dense outputing layer on the top.","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras import layers, models\nfrom tensorflow.keras.layers.experimental import preprocessing\n\nfor spectrogram, _ in train_ds.take(1):\n input_shape = spectrogram.shape\nprint('Input shape:', input_shape)\nnum_labels = len(LABELS)\n\nresizing_layer = preprocessing.Resizing(32, 32)\n# fit the scaler\nnorm_layer = preprocessing.Normalization()\nnorm_layer.adapt(train_ds.map(lambda x, _: resizing_layer(x)))","metadata":{"execution":{"iopub.status.busy":"2021-09-10T10:05:51.321493Z","iopub.execute_input":"2021-09-10T10:05:51.321809Z","iopub.status.idle":"2021-09-10T10:06:39.922683Z","shell.execute_reply.started":"2021-09-10T10:05:51.321782Z","shell.execute_reply":"2021-09-10T10:06:39.921593Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = models.Sequential([\n layers.Input(shape=input_shape),\n resizing_layer,\n norm_layer,\n layers.Conv2D(16, 3, padding='same'),\n layers.BatchNormalization(),\n layers.Activation('relu'),\n layers.Conv2D(16, 3, padding='same'),\n layers.BatchNormalization(),\n layers.Activation('relu'),\n layers.MaxPooling2D(),\n layers.Dropout(0.2),\n \n layers.Conv2D(32, 3, padding='same'),\n layers.BatchNormalization(),\n layers.Activation('relu'),\n layers.Conv2D(48, 3, padding='same'),\n layers.BatchNormalization(),\n layers.Activation('relu'),\n layers.MaxPooling2D(),\n layers.Dropout(0.25),\n \n layers.Conv2D(64, 3, padding='same'),\n layers.BatchNormalization(),\n layers.Activation('relu'),\n layers.Conv2D(128, 3, padding='same'),\n layers.BatchNormalization(),\n layers.Activation('relu'),\n layers.GlobalMaxPooling2D(),\n\n layers.Flatten(),\n layers.Dropout(0.4),\n layers.Dense(num_labels),\n])\n\nmodel.summary()\n\nmodel.compile(\n tf.keras.optimizers.Adam(1e-4),\n loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n metrics=['accuracy']\n)","metadata":{"execution":{"iopub.status.busy":"2021-09-10T10:06:39.928373Z","iopub.execute_input":"2021-09-10T10:06:39.930578Z","iopub.status.idle":"2021-09-10T10:06:40.383241Z","shell.execute_reply.started":"2021-09-10T10:06:39.930539Z","shell.execute_reply":"2021-09-10T10:06:40.382200Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"batch_size = 128\ntrain_ds_batched = train_ds.cache().shuffle(len(train_ds), reshuffle_each_iteration=True).prefetch(AUTOTUNE).batch(batch_size)\nval_ds_batched = val_ds.cache().prefetch(AUTOTUNE).batch(batch_size)\n\nEPOCHS = 100\nhistory = model.fit(\n train_ds_batched, \n validation_data=val_ds_batched, \n epochs=EPOCHS,\n callbacks=tf.keras.callbacks.EarlyStopping('val_accuracy', verbose=1, patience=5, restore_best_weights=True),\n)","metadata":{"execution":{"iopub.status.busy":"2021-09-10T10:06:40.384905Z","iopub.execute_input":"2021-09-10T10:06:40.385151Z","iopub.status.idle":"2021-09-10T10:07:55.845434Z","shell.execute_reply.started":"2021-09-10T10:06:40.385125Z","shell.execute_reply":"2021-09-10T10:07:55.844565Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"0.89-0.90 on validation","metadata":{}},{"cell_type":"code","source":"fig, axs = plt.subplots(2, 1, tight_layout=True, figsize=(10, 8))\naxs[0].set_title('Loss')\naxs[1].set_title('Accuracy')\naxs[0].plot(history.history['loss'], label='train')\naxs[0].plot(history.history['val_loss'], label='validation')\naxs[1].plot(history.history['accuracy'], label='train')\naxs[1].plot(history.history['val_accuracy'], label='validation')\naxs[0].legend();","metadata":{"execution":{"iopub.status.busy":"2021-09-09T21:57:47.994124Z","iopub.execute_input":"2021-09-09T21:57:47.994534Z","iopub.status.idle":"2021-09-09T21:57:48.415889Z","shell.execute_reply.started":"2021-09-09T21:57:47.994495Z","shell.execute_reply":"2021-09-09T21:57:48.415098Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import seaborn as sns\ntest_audio = []\ntest_labels = []\n\nfor audio, label in val_ds:\n test_audio.append(audio.numpy())\n test_labels.append(label.numpy())\n\ntest_audio = np.array(test_audio)\ntest_labels = np.array(test_labels)\n \ny_pred = np.argmax(model.predict(test_audio), axis=1)\ny_true = test_labels\n \nconfusion_mtx = tf.math.confusion_matrix(y_true, y_pred) \nplt.figure(figsize=(10, 8))\nsns.heatmap(confusion_mtx, xticklabels=LABELS, yticklabels=LABELS, \n annot=True, fmt='g')\nplt.xlabel('Prediction')\nplt.ylabel('Label')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-09-09T21:57:48.417091Z","iopub.execute_input":"2021-09-09T21:57:48.41746Z","iopub.status.idle":"2021-09-09T21:57:56.259765Z","shell.execute_reply.started":"2021-09-09T21:57:48.417432Z","shell.execute_reply":"2021-09-09T21:57:56.258938Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Model often misclassify \"go\" and \"no\". Also model badly predict \"unknown\", it could be assumed - its hard to train model predict such thing in supervised way.","metadata":{}},{"cell_type":"code","source":"model.evaluate(test_ds.batch(64))","metadata":{"execution":{"iopub.status.busy":"2021-09-09T22:05:39.572239Z","iopub.execute_input":"2021-09-09T22:05:39.572598Z","iopub.status.idle":"2021-09-09T22:05:45.905078Z","shell.execute_reply.started":"2021-09-09T22:05:39.572568Z","shell.execute_reply":"2021-09-09T22:05:45.904284Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Little overview of pretrained model approach - DeepSpeech2\n\nClassic situation: this package does not have documentation. I decided to hastily create this little code snippet to try the model, but if will need the same model in production, we can reimplement it manually, and then load the weights.","metadata":{}},{"cell_type":"code","source":"%%capture\n!pip install tables google-cloud-storage python-speech-features\n!git clone https://github.com/rolczynski/Automatic-Speech-Recognition.git\n!mv Automatic-Speech-Recognition/automatic_speech_recognition automatic_speech_recognition\n!rm -rf Automatic-Speech-Recognition","metadata":{"execution":{"iopub.status.busy":"2021-09-09T22:06:24.704284Z","iopub.execute_input":"2021-09-09T22:06:24.704629Z","iopub.status.idle":"2021-09-09T22:06:38.196838Z","shell.execute_reply.started":"2021-09-09T22:06:24.704599Z","shell.execute_reply":"2021-09-09T22:06:38.195737Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import automatic_speech_recognition as asr\n\npipeline = asr.load('deepspeech2', lang='en')\npipeline.model.summary()","metadata":{"execution":{"iopub.status.busy":"2021-09-09T22:06:38.200435Z","iopub.execute_input":"2021-09-09T22:06:38.200715Z","iopub.status.idle":"2021-09-09T22:06:47.821044Z","shell.execute_reply.started":"2021-09-09T22:06:38.200687Z","shell.execute_reply":"2021-09-09T22:06:47.820181Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from tqdm.auto import tqdm\n\nval_filenames = tf.io.gfile.glob(VAL_DATASET_PATH + '/*/*')\ndeepspeech_pred = []\ntrue_labels = []\n\nfor file in tqdm(val_filenames[:25]):\n deepspeech_pred.append(pipeline.predict([asr.utils.read_audio(file)]))\n true_labels.append(get_label(file))","metadata":{"execution":{"iopub.status.busy":"2021-09-09T22:09:29.840318Z","iopub.execute_input":"2021-09-09T22:09:29.840673Z","iopub.status.idle":"2021-09-09T22:10:39.231779Z","shell.execute_reply.started":"2021-09-09T22:09:29.840643Z","shell.execute_reply":"2021-09-09T22:10:39.230973Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.array(deepspeech_pred).ravel()","metadata":{"execution":{"iopub.status.busy":"2021-09-09T22:10:39.233283Z","iopub.execute_input":"2021-09-09T22:10:39.233641Z","iopub.status.idle":"2021-09-09T22:10:39.242746Z","shell.execute_reply.started":"2021-09-09T22:10:39.233604Z","shell.execute_reply":"2021-09-09T22:10:39.241972Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.array(true_labels).ravel()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!rm -rf automatic_speech_recognition","metadata":{"execution":{"iopub.status.busy":"2021-09-09T22:10:39.244119Z","iopub.execute_input":"2021-09-09T22:10:39.244431Z","iopub.status.idle":"2021-09-09T22:10:40.165316Z","shell.execute_reply.started":"2021-09-09T22:10:39.244406Z","shell.execute_reply":"2021-09-09T22:10:40.164239Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It seems to work good, especially if we will finetune it, and maybe, use some heuristic based on Levenshtein distance to correct outputs.\n\nBut this is really big and slow even on GPU, so it's not the case for task we are trying to solve.","metadata":{"execution":{"iopub.status.busy":"2021-09-09T14:34:23.26533Z","iopub.execute_input":"2021-09-09T14:34:23.265586Z","iopub.status.idle":"2021-09-09T14:34:23.272322Z","shell.execute_reply.started":"2021-09-09T14:34:23.265563Z","shell.execute_reply":"2021-09-09T14:34:23.27157Z"}}},{"cell_type":"markdown","source":"## Conclusions\n\nI tried three methods for this task, but there is still a big room for improvement.\n\n**Nearest Neighbors**:\n- Pros:\n - Fast & simple\n - Understandable mechanism - no pitfalls\n- Cons:\n - Training set should be stored on a production platform\n - Quite inaccurate\n- Ways for improvement:\n - kNN shows that distance-based methods can have a chance, so maybe some clustering should be performed, and applied for the test set as \"k nearest centroids\"\n - Create additional features. There is a variety of decompositions in signal processing, also they have a lot of parameters, like window type and window size. Just need to find appropriate.\n \n**CNN**:\n- Pros:\n - the best model for now\n - still simple enough\n- Cons:\n - some systematic misclassification\n- Ways for improvement:\n - Additional data. I think these one-word commands can be found in other datasets.\n - Data augmentation. We can apply various sound effects to expand our dataset: pitch, reverb, compression, distortion, changing a loudness, adding a background noise...\n - Dealing with \"unknown\" label. The actual detection of the unknown is not even close to perfect. I think that there must another way to implement detection of new words, maybe with clustering of the last convolutional layer output.\n \n**Pretrained models**:\n- Pros:\n - I think they can give a pretty good prediction if we spent some time on pipeline organization\n- Cons:\n - Heavy. Spend more than 1.5 sec for a single prediction on GPU\n - Output requires additional processing\n- Ways for improvement:\n - Perform research of existing speech recognition models. Find the best for our task\n - Creating algorithm for labels fixing\n - Finetuning\n \nThe next step should be rewriting the model pure TensorFlow (but unfortunately, I have no time left for this) and deploying the best model to \"production\" (test on a competition) ","metadata":{}},{"cell_type":"code","source":"%%capture\n!cd /kaggle/temp && 7z x /kaggle/input/tensorflow-speech-recognition-challenge/test.7z","metadata":{"execution":{"iopub.status.busy":"2021-09-10T10:07:55.847079Z","iopub.execute_input":"2021-09-10T10:07:55.847416Z","iopub.status.idle":"2021-09-10T11:15:21.055239Z","shell.execute_reply.started":"2021-09-10T10:07:55.847377Z","shell.execute_reply":"2021-09-10T11:15:21.053940Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"TEST_PATH = '/kaggle/temp/test/audio/*'\ntest_ds = load_and_preprocess_dataset(TEST_PATH)\ntest_ds = test_ds.prefetch(AUTOTUNE).batch(64)\nfnames = tf.io.gfile.glob(TEST_PATH)\n\ntest_pred = model.predict(test_ds)\npred_labels = np.argmax(test_pred, axis=1).astype('int')\npred_labels = LABELS[pred_labels]\n\nsubm = pd.DataFrame({'fname': fnames, 'label': pred_labels})\nsubm.to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2021-09-10T11:27:11.050797Z","iopub.execute_input":"2021-09-10T11:27:11.051128Z","iopub.status.idle":"2021-09-10T11:27:12.054291Z","shell.execute_reply.started":"2021-09-10T11:27:11.051099Z","shell.execute_reply":"2021-09-10T11:27:12.052285Z"},"trusted":true},"execution_count":null,"outputs":[]}]}