{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Improving on baseline model with bidirectional model\n\nMost code and baseline model from this kernel: https://www.kaggle.com/chewzy/gru-w-attention-baseline-model-curated [version 2]\n\n**Thanks Chewzy!**\n\n## What are we going to do here?\n\nWe will try to improve on the baseline model."},{"metadata":{},"cell_type":"markdown","source":"### References:\n* https://www.kaggle.com/maxwell110/beginner-s-guide-to-audio-data-2\n* https://www.kaggle.com/daisukelab/cnn-2d-basic-solution-powered-by-fast-ai\n* https://www.kaggle.com/christofhenkel/keras-baseline-lstm-attention-5-fold\n* https://yerevann.github.io/2016/06/26/combining-cnn-and-rnn-for-spoken-language-identification/\n\n### In this kernel, only train curated will be used.\n\nI'm taking 5 seconds of spectrograms for each video -> likely an overkill, to be fine-tuned later.\n\nTo use the noisy set for training, a data generator is required, as the complete spectograms won't fit into the memory."},{"metadata":{"trusted":false},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport os\nimport librosa\nimport matplotlib.pyplot as plt\nimport gc\n\nfrom tqdm import tqdm, tqdm_notebook\nfrom sklearn.metrics import label_ranking_average_precision_score\nfrom sklearn.model_selection import train_test_split\n\n#tqdm.pandas() #?","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def calculate_overall_lwlrap_sklearn(truth, scores):\n    \"\"\"Calculate the overall lwlrap using sklearn.metrics.lrap.\"\"\"\n    # sklearn doesn't correctly apply weighting to samples with no labels, so just skip them.\n    sample_weight = np.sum(truth > 0, axis=1)\n    nonzero_weight_sample_indices = np.flatnonzero(sample_weight > 0)\n    overall_lwlrap = label_ranking_average_precision_score(\n        truth[nonzero_weight_sample_indices, :] > 0, \n        scores[nonzero_weight_sample_indices, :], \n        sample_weight=sample_weight[nonzero_weight_sample_indices])\n    return overall_lwlrap","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def split_and_label(rows_labels):\n    \n    row_labels_list = []\n    for row in rows_labels:\n        row_labels = row.split(',')\n        labels_array = np.zeros((80))\n        \n        for label in row_labels:\n            index = label_mapping[label]\n            labels_array[index] = 1\n        \n        row_labels_list.append(labels_array)\n    \n    return row_labels_list","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"train_curated = pd.read_csv('../input/train_curated.csv')\ntrain_noisy = pd.read_csv('../input/train_noisy.csv')\ntest = pd.read_csv('../input/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"print(train_curated.shape, train_noisy.shape, test.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"label_columns = test.columns[1:]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"label_mapping = dict((label, index) for index, label in enumerate(label_columns))","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":false},"cell_type":"code","source":"label_mapping","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"for col in tqdm(label_columns):\n    train_curated[col] = 0\n    train_noisy[col] = 0\n    \nprint(train_curated.shape, train_noisy.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"train_curated_labels = split_and_label(train_curated['labels'])\ntrain_noisy_labels = split_and_label(train_noisy['labels'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"train_curated[label_columns] = train_curated_labels\ntrain_noisy[label_columns] = train_noisy_labels","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"train_curated['num_labels'] = train_curated[label_columns].sum(axis=1)\ntrain_noisy['num_labels'] = train_noisy[label_columns].sum(axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"plt.figure(figsize=(18,6))\n\nplt.subplot(121)\nax1 = train_curated['num_labels'].value_counts().plot(kind='bar')\nplt.xlabel('Number of labels')\nplt.ylabel('Counts')\nplt.xticks(rotation=0)\nplt.title('Curated Training Set')\n\nfor p in ax1.patches:\n    ax1.annotate(str(p.get_height()), \n                (p.get_x() + p.get_width()/2., p.get_height() * 1.005), \n                ha='center',\n                va='center',\n                xytext=(0,5), \n                textcoords='offset points')\n\nplt.subplot(122)\nax2 = train_noisy['num_labels'].value_counts().sort_index().plot(kind='bar', )\nplt.xlabel('Number of labels')\nplt.ylabel('Counts')\nplt.xticks(rotation=0)\nplt.title('Noisy Training Set')\n\nfor p in ax2.patches:\n    ax2.annotate(str(p.get_height()), \n                (p.get_x() + p.get_width()/2., p.get_height() * 1.005), \n                ha='center',\n                va='center',\n                xytext=(0,5), \n                textcoords='offset points')\n\n    \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Special thanks to https://github.com/makinacorpus/easydict/blob/master/easydict/__init__.py\n\nclass EasyDict(dict):\n\n    def __init__(self, d=None, **kwargs):\n        if d is None:\n            d = {}\n        if kwargs:\n            d.update(**kwargs)\n        for k, v in d.items():\n            setattr(self, k, v)\n        # Class attributes\n        for k in self.__class__.__dict__.keys():\n            if not (k.startswith('__') and k.endswith('__')) and not k in ('update', 'pop'):\n                setattr(self, k, getattr(self, k))\n\n    def __setattr__(self, name, value):\n        if isinstance(value, (list, tuple)):\n            value = [self.__class__(x)\n                     if isinstance(x, dict) else x for x in value]\n        elif isinstance(value, dict) and not isinstance(value, self.__class__):\n            value = self.__class__(value)\n        super(EasyDict, self).__setattr__(name, value)\n        super(EasyDict, self).__setitem__(name, value)\n\n    __setitem__ = __setattr__\n\n    def update(self, e=None, **f):\n        d = e or dict()\n        d.update(f)\n        for k in d:\n            setattr(self, k, d[k])\n\n    def pop(self, k, d=None):\n        delattr(self, k)\n        return super(EasyDict, self).pop(k, d)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"conf = EasyDict()\nconf.sampling_rate = 44100\nconf.duration = 5\nconf.hop_length = 347 # to make time steps 128\nconf.fmin = 20\nconf.fmax = conf.sampling_rate // 2\nconf.n_mels = 128\nconf.n_fft = conf.n_mels * 20\n\nconf.samples = conf.sampling_rate * conf.duration\n\ntrain_curated_path = '../input/train_curated/'\ntrain_noisy_path = '../input/train_noisy/'\ntest_path = '../input/test/'","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def read_audio(conf, pathname, trim_long_data):\n    y, sr = librosa.load(pathname, sr=conf.sampling_rate)\n    # trim silence\n    if 0 < len(y): # workaround: 0 length causes error\n        y, _ = librosa.effects.trim(y) # trim, top_db=default(60)\n    # make it unified length to conf.samples\n    if len(y) > conf.samples: # long enough\n        if trim_long_data:\n            y = y[0:0+conf.samples]\n    else: # pad blank\n        padding = conf.samples - len(y)    # add padding at both ends\n        offset = padding // 2\n        y = np.pad(y, (offset, conf.samples - len(y) - offset), 'constant')\n    return y\n\ndef audio_to_melspectrogram(conf, audio):\n    spectrogram = librosa.feature.melspectrogram(audio, \n                                                 sr=conf.sampling_rate,\n                                                 n_mels=conf.n_mels,\n                                                 hop_length=conf.hop_length,\n                                                 n_fft=conf.n_fft,\n                                                 fmin=conf.fmin,\n                                                 fmax=conf.fmax)\n    spectrogram = librosa.power_to_db(spectrogram)\n    spectrogram = spectrogram.astype(np.float32)\n    return spectrogram\n\ndef read_as_melspectrogram(conf, pathname, trim_long_data, debug_display=False):\n    x = read_audio(conf, pathname, trim_long_data)\n    mels = audio_to_melspectrogram(conf, x)\n    if debug_display:\n        IPython.display.display(IPython.display.Audio(x, rate=conf.sampling_rate))\n        show_melspectrogram(conf, mels)\n    return mels\n\ndef convert_wav_to_image(df, source):\n    X = []\n    for i, row in tqdm_notebook(df.iterrows()):\n        try:\n            x = read_as_melspectrogram(conf, f'{source[0]}/{str(row.fname)}', trim_long_data=True)\n        except:\n            x = read_as_melspectrogram(conf, f'{source[1]}/{str(row.fname)}', trim_long_data=True)\n\n        #x_color = mono_to_color(x)\n        X.append(x.transpose())\n        #df.loc[i, 'length'] = x.shape[1]\n    return X","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"#For baseline, noisy set is not used.\n#train = pd.concat([train_curated, train_noisy],axis=0)\n\n#del train_curated, train_noisy\n\n#gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"%%time\n\n#X = np.array(convert_wav_to_image(train, source=[train_curated_path, train_noisy_path]))\nX = np.array(convert_wav_to_image(train_curated, source=[train_curated_path]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"Y = train_curated[label_columns].values","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"from keras import backend as K\nfrom keras.engine.topology import Layer\nfrom keras import initializers, regularizers, constraints, optimizers, layers\nfrom keras.layers import Embedding, Input, Dense, CuDNNGRU,concatenate, Bidirectional, SpatialDropout1D, Conv1D, GlobalAveragePooling1D, GlobalMaxPooling1D, Dropout\nfrom keras.models import Model\nfrom keras.optimizers import Adam\nfrom keras.callbacks import EarlyStopping\nfrom keras.layers import *","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"class Attention(Layer):\n    def __init__(self, step_dim,\n                 W_regularizer=None, b_regularizer=None,\n                 W_constraint=None, b_constraint=None,\n                 bias=True, **kwargs):\n        self.supports_masking = True\n        self.init = initializers.get('glorot_uniform')\n\n        self.W_regularizer = regularizers.get(W_regularizer)\n        self.b_regularizer = regularizers.get(b_regularizer)\n\n        self.W_constraint = constraints.get(W_constraint)\n        self.b_constraint = constraints.get(b_constraint)\n\n        self.bias = bias\n        self.step_dim = step_dim\n        self.features_dim = 0\n        super(Attention, self).__init__(**kwargs)\n\n    def build(self, input_shape):\n        assert len(input_shape) == 3\n\n        self.W = self.add_weight((input_shape[-1],),\n                                 initializer=self.init,\n                                 name='{}_W'.format(self.name),\n                                 regularizer=self.W_regularizer,\n                                 constraint=self.W_constraint)\n        self.features_dim = input_shape[-1]\n\n        if self.bias:\n            self.b = self.add_weight((input_shape[1],),\n                                     initializer='zero',\n                                     name='{}_b'.format(self.name),\n                                     regularizer=self.b_regularizer,\n                                     constraint=self.b_constraint)\n        else:\n            self.b = None\n\n        self.built = True\n\n    def compute_mask(self, input, input_mask=None):\n        return None\n\n    def call(self, x, mask=None):\n        features_dim = self.features_dim\n        step_dim = self.step_dim\n\n        eij = K.reshape(K.dot(K.reshape(x, (-1, features_dim)),\n                        K.reshape(self.W, (features_dim, 1))), (-1, step_dim))\n\n        if self.bias:\n            eij += self.b\n\n        eij = K.tanh(eij)\n\n        a = K.exp(eij)\n\n        if mask is not None:\n            a *= K.cast(mask, K.floatx())\n\n        a /= K.cast(K.sum(a, axis=1, keepdims=True) + K.epsilon(), K.floatx())\n\n        a = K.expand_dims(a)\n        weighted_input = x * a\n        return K.sum(weighted_input, axis=1)\n\n    def compute_output_shape(self, input_shape):\n        return input_shape[0],  self.features_dim","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Baseline model\n\n**This is the model we try to improve on**"},{"metadata":{"trusted":false},"cell_type":"code","source":"def model_baseline(input_shape=(636,128)):\n\n    sequence_input = Input(shape=(636,128), dtype='float32')\n    x = CuDNNGRU(128, return_sequences=True)(sequence_input)\n\n    att = Attention(636)(x)\n    avg_pool = GlobalAveragePooling1D()(x)\n    max_pool = GlobalMaxPooling1D()(x) \n\n    x = concatenate([att, avg_pool, max_pool])\n\n    preds = Dense(80, activation='softmax')(x)\n\n    model = Model(sequence_input, preds)\n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model bidirectional\n\nThis model is the same as baseline, but now we step trough the input sequence both in forward and backward directions (bidirectional)"},{"metadata":{"scrolled":true,"trusted":false},"cell_type":"code","source":"def model_bi_gru(input_shape=(636,128)):\n    inp = Input(shape=input_shape)\n      \n    x = Bidirectional(CuDNNGRU(128, return_sequences=True))(inp)\n    #x = Bidirectional(CuDNNLSTM(64, return_sequences=True))(x)\n  \n    att = Attention(input_shape[0])(x)\n    avg_pool = GlobalAveragePooling1D()(x)\n    max_pool = GlobalMaxPooling1D()(x) \n    \n    x = concatenate([att, avg_pool, max_pool])\n    \n    x = Dense(80, activation=\"softmax\")(x)\n\n    model = Model(inputs=inp, outputs=x)\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Multiple runs baseline model \n\n**How unstable is our network?**\n\nTo get a good idea of the quality of this (stochastic) model, We will let the model train multiple times on the same data. I would like to do more than 5 runs, but kernel time is limited."},{"metadata":{"scrolled":true,"trusted":false},"cell_type":"code","source":"runs = 5\n\n# collect data across multiple repeats\ntrain = pd.DataFrame()\nval = pd.DataFrame()\n\nfor i in range(runs):\n    # define model\n    model = model_baseline()\n    \n    # compile model\n    model.compile(loss='categorical_crossentropy',optimizer=Adam(0.005),metrics=['acc'])\n    \n    # train / val data\n    x_train, x_val, y_train, y_val = train_test_split(X, Y, test_size=0.2, random_state=123)\n    \n    # fit model\n    history = model.fit(np.array(x_train),\n          y_train,\n          batch_size=1024,\n          epochs=35,\n          validation_data=(np.array(x_val), y_val),\n          #callbacks = [es]\n                   )\n    # store history\n    train[str(i)] = history.history[ 'loss' ]\n    val[str(i)] = history.history[ 'val_loss' ]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"baseline_val = [ val[str(i)][val[str(0)].shape[0]-1] for i in range(runs)] # we will need this later to compare","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# plot train and validation loss across\nplt.plot(train, color= 'blue' , label= 'train')\nplt.plot(val, color= 'orange' , label= 'validation')\nplt.title( 'model train vs validation loss')\nplt.ylabel( 'loss' )\nplt.xlabel( 'epoch' )\n\nplt.grid()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Validation loss (orange lines) seem to bottom out after 30 epochs. I think 35 epochs is right between an under- and overfit."},{"metadata":{},"cell_type":"markdown","source":"## Multiple runs bidirectional model"},{"metadata":{"trusted":false},"cell_type":"code","source":"# collect data across multiple repeats\ntrain = pd.DataFrame()\nval = pd.DataFrame()\nfor i in range(runs):\n    # define model\n    model = model_bi_gru()\n    \n    # compile model\n    model.compile(loss='categorical_crossentropy',optimizer=Adam(0.005),metrics=['acc'])\n    \n    # train / val data\n    x_train, x_val, y_train, y_val = train_test_split(X, Y, test_size=0.2, random_state=123)\n    \n    # fit model\n    history = model.fit(np.array(x_train),\n          y_train,\n          batch_size=1024,\n          epochs=50,\n          validation_data=(np.array(x_val), y_val),\n          #callbacks = [es]\n                   )\n    # store history\n    train[str(i)] = history.history[ 'loss' ]\n    val[str(i)] = history.history[ 'val_loss' ]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"bidirectional_val = [ val[str(i)][val[str(0)].shape[0]-1] for i in range(runs)] # we will need this later to compare","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# plot train and validation loss across\nplt.plot(train, color= 'blue' , label= 'train')\nplt.plot(val, color= 'orange' , label= 'validation')\nplt.title( 'model train vs validation loss')\nplt.ylabel( 'loss' )\nplt.xlabel( 'epoch' )\n\nplt.grid()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Model doesnt seem to improve anymore. 50 epochs is plenty"},{"metadata":{},"cell_type":"markdown","source":"## Comparing baseline with bidirectional model"},{"metadata":{"trusted":false},"cell_type":"code","source":"scores = pd.DataFrame()\nscores['model_baseline'] = baseline_val\nscores['model_bidirectional'] = bidirectional_val\n\n# box and whisker plot of results\nscores.boxplot()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Did we really improve on baseline model?** \n\nBeeing a datascientist is all about taking risks, rolling the dice, making snap decisions. So we are going with a **Maybe**. \n\nWelcome to the dangerzone."},{"metadata":{},"cell_type":"markdown","source":"## Making a submission"},{"metadata":{"trusted":false},"cell_type":"code","source":"y_train_pred = model.predict(np.array(x_train))\ny_val_pred = model.predict(np.array(x_val))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"train_lwlrap = calculate_overall_lwlrap_sklearn(y_train, y_train_pred)\nval_lwlrap = calculate_overall_lwlrap_sklearn(y_val, y_val_pred)\n\nprint(f'Training LWLRAP : {train_lwlrap:.4f}')\nprint(f'Validation LWLRAP : {val_lwlrap:.4f}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"%%time\nX_test = np.array(convert_wav_to_image(test, source=[test_path]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"predictions = model.predict(np.array(X_test))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"test[label_columns] = predictions","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.7"}},"nbformat":4,"nbformat_minor":1}