{"cells":[{"metadata":{},"cell_type":"markdown","source":"**Freesound General-Purpose Audio Tagging Challenge**\n\nAim: To build a general-purpose automatic audio tagging system using a dataset of audio files covering a wide range of real-world environments.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"import tensorflow as tf\nprint(tf.__version__)\nfrom IPython.display import Image\nImage(\"../input/blockdiagram/block diia.png\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Method**"},{"metadata":{"trusted":true},"cell_type":"code","source":"Image(\"../input/methodology/Method.png\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Generally, ML problems are solved by using data in relevant form that can be fed to train a model, which can further classify/predict outputs for unseen data. In this problem, the input data consists of audio files split into training and testing datasets. The audio clips (.wav) are represented in the form of spectrogram, which along with labels is given to the model for training. Spectrogram is explained in a later section.\n\nThe sequence of events is represented in the block diagram above."},{"metadata":{"trusted":true},"cell_type":"code","source":"# import dependencies\n\nimport numpy as np\nimport pandas as pd\nimport librosa\nimport librosa.display\nimport wave\nfrom sklearn.preprocessing import LabelEncoder, OneHotEncoder\nfrom keras import backend as K\nfrom keras.models import Sequential\nfrom keras.layers import (Conv2D, GlobalAveragePooling2D, BatchNormalization, Flatten,\n                          GlobalMaxPool2D, MaxPool2D, concatenate, Activation, Input, Dense)\nimport keras.optimizers\nimport matplotlib.pyplot as plt\nfrom keras.utils import plot_model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Data Analysis**\n\nFreesound Dataset Kaggle 2018 (or FSDKaggle2018 for short) is an audio dataset containing 18,873 audio files annotated with labels from Google's AudioSet Ontology. The audio clips are unequally distributed in 41 categories. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# load data\ntrain_data = pd.read_csv(\"../input/freesound-audio-tagging/train.csv\")\ntest_data = pd.read_csv(\"../input/freesound-audio-tagging/sample_submission.csv\")\nprint(train_data.head())\nprint(\"Number of training examples=\", train_data.shape[0], \"  Number of classes=\", len(train_data.label.unique()))\nprint(test_data.head())\nprint(\"Number of testing examples=\", test_data.shape[0], \"  Number of classes=\", len(test_data.label.unique()))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The training data includes 9,473 samples and testing data includes 9400 samples. From the output, it can be seen that the correct labels of testing data are not given."},{"metadata":{},"cell_type":"markdown","source":"**Distribution of Data**"},{"metadata":{"trusted":true},"cell_type":"code","source":"category_group = train_data.groupby(['label']).count()\nplot = category_group.unstack().plot(kind='bar', stacked=True, title=\"Number of Audio Samples per Category\", figsize=(16,10))\nplot.set_xlabel(\"Category\")\nplot.set_ylabel(\"Number of Samples\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The histogram shows that the number of audio samples per category is not uniform. The minimum number of samples per category is 94 and the maximum is 300."},{"metadata":{},"cell_type":"markdown","source":"**Frame Length**"},{"metadata":{},"cell_type":"markdown","source":"Majority of the files are short and not constant. The length of audio files range from 300 ms to 30s.\nAs the model takes data in equal sizes, the data needs to be preprocessed. The audio clips that are shorter than the required sample length are padded. Whereas the audio clips that are longer than the required sample length are cut-short.\n\nCitation: Eduardo Fonseca, Manoj Plakal, Frederic Font, Daniel P. W. Ellis, Xavier Favory, Jordi Pons, Xavier Serra. General-purpose Tagging of Freesound Audio with AudioSet Labels: Task Description, Dataset, and Baseline. In Proceedings of DCASE2018 Workshop, 2018. URL: https://arxiv.org/abs/1807.09902"},{"metadata":{},"cell_type":"markdown","source":"**Spectrogram**\n\nA spectrogram is a visual representation of frequencies of a signal as it varies. An example of spectrogram can be seen below."},{"metadata":{"trusted":true},"cell_type":"code","source":"fname = \"../input/freesound-audio-tagging/audio_test/audio_test/8319139c.wav\"\nsamples, sample_rate = librosa.core.load(fname, sr=44100)\ns = librosa.feature.melspectrogram(samples,sr=sample_rate)\ns = librosa.power_to_db(s)\ns = s.astype(np.float32)\nplt.figure(figsize=(10, 4))\nlibrosa.display.specshow(s,y_axis='mel', fmax=8000,x_axis='time')\nplt.colorbar(format='%+2.0f dB')\nplt.title('Mel spectrogram')\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"A spectrogram has three dimensions. X-axis represents time, Y-axis represents frequency and the final dimension represents the amplitude of a particular frequency at a particular time is represented by the intensity or color of each point in the image."},{"metadata":{},"cell_type":"markdown","source":"**Data Preprocessing**\n\nBefore the data is fed into the model, it needs to be preprocessed into a desired form. Preprocessing of data includes the following steps:\n1. Extract samples from audio, adjust the length and extract spectrograms.\n2. Normalize the data"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# Loading and processing Data\n\ndef get_spectrogram(filename):\n    '''\n    Input: A dataframe of filepaths of the audio clips\n    Returns: A numpy array of spectrograms of the clips in the dataframe.\n    \n    For every audio clip in the dataset, samples are extracted using Librosa library. The length of the samples are\n    adjusted as required.\n    Spectrogram for the samples is calculated using the same library Librosa which is then appended to a numpy array\n    '''\n    import tqdm\n    x = []\n    duration = 5\n    sample_length = 44100 * duration\n    for fname in tqdm.tqdm(filename):\n        samples, sample_rate = librosa.core.load(fname, sr=44100)\n        if len(samples) > sample_length: # long enough\n            samples = samples[0:sample_length]\n        else: # pad blank\n            padding = sample_length - len(samples)\n            offset = padding // 2\n            if len(samples) == 0:\n                samples = np.pad(samples, (offset, sample_length - len(samples) - offset), 'constant')\n            else:\n                while(len(samples)<sample_length):\n                    padding = sample_length - len(samples)\n                    samples = np.append(samples, samples[0:padding], axis=0)\n        #mfcc = librosa.feature.mfcc(samples,sr=sample_rate)\n        s = librosa.feature.melspectrogram(samples,sr=sample_rate)\n        s = librosa.power_to_db(s)\n        s = s.astype(np.float32)\n        x.append(s)\n    # Stack them using axis=0.\n    x = np.stack(x)\n    print(x.shape)\n    x = x.reshape(x.shape[0], x.shape[1], x.shape[2],1)\n    print(\"done\")\n    return x\n\n\ndef get_labels(y):\n    '''\n    Input: A list of Labels\n    Returns: A list of number encoded labels and a map of labels to number code\n    '''\n    le = LabelEncoder()\n    le.fit(y)\n    y = le.fit_transform(y)\n    le_mapping = dict(zip(le.transform(le.classes_),le.classes_))\n    return y, le_mapping\n\n\ndef normalize(X):\n    '''\n    Input: Dataset\n    Returns: Z-Score normalization of the data in the dataset.\n    '''\n    eps = 0.001\n    normalized_dataset = []\n    for img in X:\n        if np.std(img) != 0:\n            img = (img - np.mean(img)) / np.std(img)\n        else:\n            img = (img - np.mean(img)) / eps\n        normalized_dataset.append(img)\n    return np.array(normalized_dataset)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def process_data(train_data,test_data):\n    '''\n    Input: Dataframes of training and testing data given in the workspace of kaggle\n    Returns: Processed training and testing data along with labels and label mapping.\n    '''\n    filename_train = '../input/freesound-audio-tagging/audio_train/audio_train/' + train_data['fname']\n    filename_test = '../input/freesound-audio-tagging/audio_test/audio_test/' + test_data['fname']\n    x_train = get_spectrogram(filename_train)\n    x_test = get_spectrogram(filename_test)\n    x_train = normalize(x_train)\n    x_test = normalize(x_test)\n    y_train, le_mapping = get_labels(train_data['label'])\n    y_test = np.array(test_data['label'])\n    return x_train, y_train,x_test, y_test, le_mapping\n\n\ntrain_data = pd.read_csv(\"../input/freesound-audio-tagging/train.csv\")\ntest_data = pd.read_csv(\"../input/freesound-audio-tagging/sample_submission.csv\")\n\n# transform and extract relevant data\nx_train,y_train,x_test,y_test, le_mapping = process_data(train_data,test_data)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Model**\n\nModel defined is a 2D Convolution Neural Network. The architecture is defined using Keras."},{"metadata":{"trusted":true},"cell_type":"code","source":"def train(x_train,y_train):\n    '''\n    Input: Training data\n    Returns: A trained model and training metrics\n    '''\n    \n    batch_size = 50\n    optimizer = keras.optimizers.SGD(lr=0.01)\n    input_shape=(x_train.shape[1], x_train.shape[2],1)\n    model = Sequential()\n    model.add(Conv2D(64, (3,6),input_shape=input_shape))\n    model.add(BatchNormalization())\n    model.add(Activation(\"tanh\"))\n    model.add(MaxPool2D())\n    \n    model.add(Conv2D(32, (3,6)))\n    model.add(BatchNormalization())\n    model.add(Activation(\"relu\"))\n    model.add(MaxPool2D())\n    \n    model.add(Conv2D(32, (3,6)))\n    model.add(BatchNormalization())\n    model.add(Activation(\"relu\"))\n    model.add(MaxPool2D())\n    \n    model.add(Flatten())\n    model.add(Dense(41,activation='softmax'))\n    model.summary()\n    model.compile(loss = 'sparse_categorical_crossentropy', optimizer=optimizer,metrics = ['accuracy'])\n    history = model.fit(x_train, y_train,batch_size=batch_size,epochs=50,verbose=1,validation_split=0.3)\n    \n    return model, history\n\nmodel, history = train(x_train,y_train)\nmodel.save('my_keras_model.h5')\nplot_model(model, to_file='model_plot.png', show_shapes=True, show_layer_names=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Training Metrics**"},{"metadata":{"trusted":true},"cell_type":"code","source":"model.evaluate(x_train,y_train)\nprint(\"Output: \",model.outputs)\nprint(\"Input: \",model.inputs)\nx_axis = np.linspace(1,50,50)\nplt.subplot(2, 1, 1)\nplt.plot(x_axis, history.history['acc'], history.history['val_acc'])\nplt.title('Training metrics')\nplt.ylabel('Accuracy')\n\nplt.subplot(2, 1, 2)\nplt.plot(x_axis, history.history['loss'], history.history['val_loss'])\nplt.xlabel('Epochs')\nplt.ylabel('Loss')\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Testing**\n\nTested a sample of test data with the model"},{"metadata":{"trusted":true},"cell_type":"code","source":"def test(x_test,le_mapping, model,y_test):\n    y_pred = model.predict_classes(x_test)\n    pred = pd.DataFrame([],columns = ['Predicted','Actual'])\n    for i in range(len(y_pred)):\n        pred = pred.append({'Predicted':le_mapping[y_pred[i]], 'Actual':y_test[i]}, ignore_index=True)\n    print(pred)\n        \n# test model\nx_test  = ['0038a046.wav','007759c4.wav','00ae03f6.wav','00eac343.wav','010a0b3a.wav','01a5a2a3.wav','02107093.wav','02960f07.wav'\n          ,'02fb6c5b.wav']\nx_test = pd.DataFrame(x_test,columns=['fname'])\ny_test = [\"Bass_drum\",'Saxophone','Chime','Electric_piano','Shatter','bark','Electric_piano','Scissors','Knock']\nfilename_test = '../input/freesound-audio-tagging/audio_test/audio_test/' + x_test['fname']\nx_test = get_spectrogram(filename_test)\nx_test = normalize(x_test)\ntest(x_test,le_mapping,model,y_test)        \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def export_model_for_mobile(model_name, input_node_names, output_node_name):\n    tf.train.write_graph(K.get_session().graph_def, 'out', \\\n        model_name + '_graph.pbtxt')\n\n    tf.train.Saver().save(K.get_session(), 'out/' + model_name + '.chkp')\n\n    freeze_graph.freeze_graph('out/' + model_name + '_graph.pbtxt', None, \\\n        False, 'out/' + model_name + '.chkp', output_node_name, \\\n        \"save/restore_all\", \"save/Const:0\", \\\n        'out/frozen_' + model_name + '.pb', True, \"\")\n\n    input_graph_def = tf.GraphDef()\n    with tf.gfile.Open('out/frozen_' + model_name + '.pb', \"rb\") as f:\n        input_graph_def.ParseFromString(f.read())\n\n    output_graph_def = optimize_for_inference_lib.optimize_for_inference(\n            input_graph_def, input_node_names, [output_node_name],\n            tf.float32.as_datatype_enum)\n\n    with tf.gfile.FastGFile('out/tensorflow_lite_' + model_name + '.pb', \"wb\") as f:\n        f.write(output_graph_def.SerializeToString())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras import backend as K\nimport tensorflow as tf\n\ndef freeze_session(session, keep_var_names=None, output_names=None, clear_devices=True):\n\n    from tensorflow.python.framework.graph_util import convert_variables_to_constants\n    graph = session.graph\n    with graph.as_default():\n        freeze_var_names = list(set(v.op.name for v in tf.global_variables()).difference(keep_var_names or []))\n        output_names = output_names or []\n        output_names += [v.op.name for v in tf.global_variables()]\n        # Graph -> GraphDef ProtoBuf\n        input_graph_def = graph.as_graph_def()\n        if clear_devices:\n            for node in input_graph_def.node:\n                node.device = \"\"\n        frozen_graph = convert_variables_to_constants(session, input_graph_def,\n                                                      output_names, freeze_var_names)\n        return frozen_graph\n\n\nfrozen_graph = freeze_session(K.get_session(),\n                              output_names=[out.op.name for out in model.outputs])\ntf.train.write_graph(frozen_graph, \"model\", \"my_model.pb\", as_text=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(le_mapping)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}