{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# **MIDAS Task 1**\n\nFor this task, two different methods are tested and evaluated. The methods differ mainly on the audio feature used for classification. They are listed below as:\n1. Using Normalized Log Spectrogram of Audio Samples\n2. Using MFCC's and their higher order Deltas\n\n**This notebook only has the first method, which gavebest results. The other method is in the other experiments, placed in the experiments folder. This notebook has:**\n* Libraries Required\n* Data Unzip\n* Data Check\n* EDA\n* Data Loading, Preprocessing, Feature Extraction\n* Training andTesting\n\nReferences:\n\n1. David S., Speech representation and Data Exploration: Has been used to better understand the data","metadata":{}},{"cell_type":"markdown","source":"# Libraries","metadata":{}},{"cell_type":"code","source":"import numpy as np \nimport pandas as pd\nfrom tqdm import tqdm\nfrom glob import glob\nimport os\nimport random\nfrom os.path import isdir, join\nimport regex as re\nimport gc\nimport sklearn\nimport pickle\n\nimport librosa\nfrom scipy.fftpack import fft\nfrom scipy import signal\nfrom scipy.io import wavfile\nfrom sklearn.decomposition import PCA\nfrom sklearn.preprocessing import StandardScaler, MinMaxScaler\n\nimport matplotlib.pyplot as plt\nimport librosa.display\nimport IPython.display as ipd \n\nfrom keras import optimizers, losses, activations, models\nfrom keras.layers import Convolution2D, Dense, Input, Flatten, Dropout, MaxPooling2D, BatchNormalization, Conv1D\nfrom sklearn.model_selection import train_test_split\nimport keras\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Unzip","metadata":{}},{"cell_type":"code","source":"!apt-get install -y p7zip-full\n!7z x ../input/tensorflow-speech-recognition-challenge/train.7z\n!7z x ../input/tensorflow-speech-recognition-challenge/test.7z","metadata":{"_kg_hide-output":true,"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''\nCheck Training Audio Folder\n'''\n\ntrain_audio_path = 'train/audio/'\nprint(len(os.listdir(train_audio_path)))\nprint(os.listdir(train_audio_path))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Exploratory Data Analysis\n\nThe following Features are Visualised:\n\n1. Raw Audio\n2. MFCCs\n3. First and Second Order Deltas of MFCCs\n\n**These features will be considered for the feature set**","metadata":{}},{"cell_type":"code","source":"'''\nLoading a sample file to study\n'''\n\n# At 8000 \nfilename = '/yes/0a7c2a8d_nohash_0.wav'\nsamples2, sample_rate2 = librosa.load(str(train_audio_path) + filename, 8000)\n\n# At 16000 \nfilename = '/yes/0a7c2a8d_nohash_0.wav'\nsamples, sample_rate = librosa.load(str(train_audio_path) + filename, 16000)\nprint('Sample Rate: ', sample_rate)\nprint()\nipd.Audio(samples, rate=sample_rate)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''\nPlotting Raw Data\n'''\n\nplt.figure(figsize=(20, 5))\nlibrosa.display.waveplot(samples, sr=16000)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### MFCCs (First 13)","metadata":{}},{"cell_type":"code","source":"'''\nThis feature is one of the most important method to extract a feature of an audio signal and is used majorly whenever working on audio signals. \nThe mel frequency cepstral coefficients (MFCCs) of a signal are a small set of features (usually about 10–20) which concisely describe the overall shape of a spectral envelope.\n'''\n\nplt.figure(figsize=(20,5))\nmfcc = librosa.feature.mfcc(samples, sr=16000, n_mfcc=13)\nprint(mfcc.shape)\n\nlibrosa.display.specshow(mfcc, sr=16000, x_axis='time')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### First and Second Order Deltas of MFCCs (First 13)","metadata":{}},{"cell_type":"code","source":"'''\nDeltas are just the nth order derivative of the MFCCs. The provide a simpler overview of the sample.\n'''\n\ndelta1_mfcc = librosa.feature.delta(mfcc, order=1)\nprint(delta1_mfcc.shape)\n\nplt.figure(figsize=(12, 4))\nlibrosa.display.specshow(delta1_mfcc)\nplt.ylabel('MFCC coeffs')\nplt.xlabel('Time')\nplt.title('MFCC First Order Delta')\nplt.colorbar()\nplt.tight_layout()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"delta2_mfcc = librosa.feature.delta(mfcc, order=2)\nprint(delta2_mfcc.shape)\n\nplt.figure(figsize=(12, 4))\nlibrosa.display.specshow(delta2_mfcc)\nplt.ylabel('MFCC coeffs')\nplt.xlabel('Time')\nplt.title('MFCC Second Order Delta')\nplt.colorbar()\nplt.tight_layout()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''\nExample of final stacked feature set\n'''\n\nfeatures = np.concatenate((mfcc, delta1_mfcc, delta2_mfcc), axis=0)\n\nprint(features.shape)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data Loading, Preprocessing and Feature Extraction\n\n##### Details:\n\n1. All files are sampled at 16000 HZ\n2. The file length has been kept constant at 1 second.\n2. Zero Padding has been used in sound processing to make audio files of same length. The same has been implemented here.\n3. For extremely long audio files, they have been split into smaller 1 second samples.\n\n##### Features\n\n1. MFCCs and its first and second order deltas are loaded.\n2. These are stacked to create the feature set.","metadata":{}},{"cell_type":"code","source":"'''\nVariables to load data.\n\nsr - Sample Rate\nLegal Labels - Categories to Classify\n'''\n\nsr = 16000\nlegal_labels = 'yes no up down left right on off stop go silence unknown'.split()\n\n# Paths\ntrain_data_path = 'train/audio'\ntest_data_path = 'test'\n\n# Path Helper\next = 'wav'","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''\nUtility Function\n\nTask - Collect all file names.\n'''\n\ndef collect_files(path):\n    \n    # Simple Regex to collect paths\n    fpaths = glob(os.path.join(path, r'*/*' + ext))\n    pat = r'.+/(\\w+)/\\w+\\.' + ext + '$'\n    \n    labels = []\n    for fpath in fpaths:\n        fpath = fpath.replace('\\\\', '/')\n        r = re.match(pat, fpath)\n        if r:\n            labels.append(r.group(1))\n    pat = r'.+/(\\w+\\.' + ext + ')$'\n    fnames = []\n    for fpath in fpaths:\n        fpath = fpath.replace('\\\\', '/')\n        r = re.match(pat, fpath)\n        if r:\n            fnames.append(r.group(1))\n    return labels, fnames","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''\nGet paths and labels.\n'''\n\nlabels, filenames = collect_files(train_data_path)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''\nPadding -> Pad smaller samples with 0\n\nReduce -> Reduce audio to predefined size.\n\nLabellize -> Tranform label to suitable format for model.\n'''\n\ndef padding(samples):\n    if len(samples) >= sr: \n        return samples\n    else: \n        return np.pad(samples, pad_width=(sr - len(samples), 0), mode='constant', constant_values=(0, 0))\n\ndef reduce(samples, sr=16000, num=20):\n    for i in range(num):\n        cut = np.random.randint(0, len(samples) - sr)\n        yield samples[cut: cut + sr]\n\ndef labellize(labels):\n    nlabels = []\n    for label in labels:\n        if label == '_background_noise_':\n            nlabels.append('silence')\n        elif label not in legal_labels:\n            nlabels.append('unknown')\n        else:\n            nlabels.append(label)\n    return pd.get_dummies(pd.Series(nlabels))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''\nLoad MFCCS\n'''\n\nY_tr = []\nX_tr = []\n\nfor label, fname in tqdm(zip(labels, filenames)):\n    \n    samples, sample_rate = librosa.load(os.path.join(train_data_path, label, fname), sr = 16000)\n    samples = padding(samples)\n    if len(samples) > 16000:\n        n_samples = reduce(samples)\n    else: \n        n_samples = [samples]\n    for samples in n_samples:\n        \n        # Feature Extraction\n        mfcc = librosa.feature.mfcc(samples, sr = 16000, n_mfcc = 13)\n        delta1 = librosa.feature.delta(mfcc, order=1)\n        delta2 = librosa.feature.delta(mfcc, order=2)\n        \n        features = np.concatenate((mfcc, delta1, delta2), axis=0)\n        \n        Y_tr.append(label)\n        X_tr.append(features)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''\nConverting to Numpy Arrays and labellizing.\n'''\n\nX_train = np.array(X_tr)\nX_train = X_train.reshape(tuple(list(X_train.shape) + [1]))\nY_train = labellize(Y_tr)\nlabel_index = Y_train.columns.values\nY_train = Y_train.values\nY_train = np.array(Y_train)\ndel labels, filenames, X_tr, Y_tr\ngc.collect()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training","metadata":{}},{"cell_type":"markdown","source":"* Batch Normalization used to normalize all spectrogram.\n* Max Pooling and Dropout used to prevent over-fitting.\n* Simple Convolution Layer Used. \n* A heavy model was not used since RAM was limited to 16GB on kaggle.","metadata":{}},{"cell_type":"code","source":"'''\nDefine Params\n'''\n\ninput_shape = (39, 32, 1)\nnclass = 12\ninp = Input(shape=input_shape)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''\nBuild Model\n'''\n\nnorm_inp = BatchNormalization()(inp)\nimg_1 = Convolution2D(8, kernel_size=5, activation=activations.relu)(norm_inp)\nimg_1 = Convolution2D(8, kernel_size=5, activation=activations.relu)(norm_inp)\nimg_1 = Dropout(rate=0.2)(img_1)\nimg_1 = Convolution2D(16, kernel_size=3, activation=activations.relu)(img_1)\nimg_1 = Convolution2D(16, kernel_size=3, activation=activations.relu)(img_1)\nimg_1 = Dropout(rate=0.2)(img_1)\nimg_1 = Convolution2D(32, kernel_size=2, activation=activations.relu)(img_1)\nimg_1 = Convolution2D(32, kernel_size=2, activation=activations.relu)(img_1)\nimg_1 = Dropout(rate=0.3)(img_1)\nimg_1 = Flatten()(img_1)\n\ndense_1 = BatchNormalization()(Dense(64, activation=activations.relu)(img_1))\ndense_1 = BatchNormalization()(Dense(64, activation=activations.relu)(dense_1))\ndense_1 = Dense(nclass, activation=activations.softmax)(dense_1)\n\nmodel = models.Model(inputs=inp, outputs=dense_1)\nopt = optimizers.Adam()\n\nmodel.compile(optimizer=opt, loss=losses.binary_crossentropy, metrics = ['accuracy'])\nmodel.summary()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''\nSplit into train and validation and run.\n'''\n\nX_train, X_valid, Y_train, Y_valid = train_test_split(X_train, Y_train, test_size=0.1, random_state=1)\nmodel.fit(X_train, Y_train, batch_size=32, validation_data=(X_valid, Y_valid), epochs=20, shuffle=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Testing","metadata":{}},{"cell_type":"code","source":"'''\nCreate generator for test data. Since Data is too large.\n'''\n\ntest_data_path = 'test/audio'\n\ndef test_data_generator(batch=16):\n    fpaths = glob(os.path.join(test_data_path, '*wav'))\n    i = 0\n    for path in fpaths:\n        if i == 0:\n            imgs = []\n            fnames = []\n        i += 1\n        samples, rate = librosa.load(path, sr = 16000)\n        samples = padding(samples)\n        \n        mfcc = librosa.feature.mfcc(samples, sr = 16000, n_mfcc = 13)\n        delta1 = librosa.feature.delta(mfcc, order=1)\n        delta2 = librosa.feature.delta(mfcc, order=2)\n        \n        features = np.concatenate((mfcc, delta1, delta2), axis=0)\n        \n        imgs.append(features)\n        fnames.append(path.split('\\\\')[-1])\n        if i == batch:\n            i = 0\n            imgs = np.array(imgs)\n            imgs = imgs.reshape(tuple(list(imgs.shape) + [1]))\n            yield fnames, imgs\n    if i < batch:\n        imgs = np.array(imgs)\n        imgs = imgs.reshape(tuple(list(imgs.shape) + [1]))\n        yield fnames, imgs\n    raise StopIteration()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''\nClear RAM since Test Data is too large.\n'''\n\ndel X_train, Y_train\ngc.collect()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''\nTest and store results in a dataframe.\n'''\n\nindex = []\nresults = []\n\nfor fnames, specs in tqdm(test_data_generator(batch=32)):\n    predicts = model.predict(specs)\n    predicts = np.argmax(predicts, axis=1)\n    predicts = [label_index[p] for p in predicts]\n    index.extend(fnames)\n    results.extend(predicts)","metadata":{"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.DataFrame(columns=['fname', 'label'])\ndf['fname'] = index\ndf['label'] = results\n\n'''\nChecking Format of Result and saving\n'''\n\ndef cleanfname(s):\n    lis = s.split('/')\n    return str(lis[2])\n    \ndf.fname = df.fname.apply(cleanfname)\ndf.to_csv('submission_deltas.csv', index=False)\ndf.head(20)","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}