{"cells":[{"metadata":{},"cell_type":"markdown","source":"In this notebook I present the basic keypoints of my attempt to tackle the problem. I hope you all find it helpfull. The key-points are:\n* I choose a `np.log` version of `scipy.signal.spectrogram` to transform 1-D audio signal to 2-D imaging.\n* The model uses softmax activation function.\n* During training I select random 1-seconds parts from the audio signals (see `DataGenerator`).\n* I use a stratified version of k-fold based on the *first* class. That means that for stratification purposes I choose *one of the many classes* (see dictionary `first_labels_set`) \n* I calculate out-of-fold predictions (`oof_y`) which I use to estimate the lwlrap score.\n* The test set is evaluated and averaged `k=5` times. The evaluation is done using `TestDataGenerator` hence introducing some randomness and non-determinism (`res_Y = model.predict_generator(test_gen, verbose=1)`)\n* The training process is optimizing `categorical_crossentropy` and monitors `categorical_accuracy` for termination.\n* By using early stopping in each fold I implicitly introduce some overfitting on the validation set. I expect more stable estimation by using fixed number of iterations, but is is not my style :) \n* The whole kernel takes ~2 hours to train and a couple of minutes for inference if you use the pretrained weights I already attach. \n* (GIBA) Increased Window time to 1.5 * 44100\n* (GIBA) Added TTA at test time"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nfrom os.path import isdir, join\nfrom pathlib import Path\nimport pandas as pd\nimport time\n\n# Math\nimport numpy as np\nfrom scipy.fftpack import fft\nfrom scipy import signal\nfrom scipy.io import wavfile\nimport librosa\n\nfrom sklearn.decomposition import PCA\n\n# Visualization\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport IPython.display as ipd\nimport librosa.display\n\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nimport plotly.tools as tls\nimport pandas as pd\n\n%matplotlib inline\nt1 = time.time()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##\n## Read necessary files and folders \ntrain_audio_path = '../input/freesound-audio-tagging-2019/train_curated/'\ntrain_files = os.listdir(train_audio_path)\ntrain_annot = pd.read_csv('../input/freesound-audio-tagging-2019/train_curated.csv')\ntest_audio_path = '../input/freesound-audio-tagging-2019/test/'\ntest_files = np.sort(os.listdir(test_audio_path))\nprint (test_files[:5])\n##\n## I 've created the weights in previous run. Just use them\nUSE_WEIGHTS = False\nlen(train_files), len(test_files)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##\n## Here we calculate unique labels (80) and create the necessary data structures\n## for binary encoding the multi-labeled input\n##\n## label_dict : dictionary with classes as keys and counts per classes as values\n##              {'Bark': 74, 'Raindrop': 74, 'Finger_snapping': 74, 'Run': 74, 'Whispering': 74, .... }\n## classes: 80 sound classes \n##              ['Bark', 'Raindrop', 'Finger_snapping', 'Run', 'Whispering', ...]\n## all_labels_set (size: 4970): List of sets. The same size as training sounds. Each set correspond to the classes of the i-th sound\n##              [{'Bark'},  {'Raindrop'},  {'Finger_snapping'},  {'Run'},  {'Finger_snapping'},  {'Whispering'},  {'Acoustic_guitar', 'Strum'},  ...]\n## first_labels_set (size: 4970) : List containing only first class for each training pattern..  to be used as approximation stratification \n##              ['Bark', 'Raindrop', 'Finger_snapping', 'Run', 'Finger_snapping', 'Whispering', 'Acoustic_guitar', ...]\n\n##\n## L = 1 * SAMPLE_RATE -> 1 second\n## L = 2 * SAMPLE_RATE -> 2 seconds ....\n\nSAMPLE_RATE  = 44100\nL = int( 1.5 * SAMPLE_RATE )\n\ndef create_unique_labels(all_labels):\n    label_dict = {}\n    all_labels_set = []\n    first_labels_set = []\n    for labs in all_labels:\n        lab = labs.split(',')\n        for l in lab:\n            if l in label_dict:\n                label_dict[l] = label_dict[l]  + 1\n            else:\n                label_dict[l]= 0\n\n        all_labels_set.append(set(lab))\n        first_labels_set.append(lab[0])\n    classes = list(label_dict.keys())\n    \n    return label_dict, classes, all_labels_set, first_labels_set","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"label_dict, classes, all_labels_set, first_labels_set = create_unique_labels(train_annot.labels)\nfiles = train_annot.fname\nprint (len(files), len(train_files))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##\n## Y_split are the binary labels used for stratification\n## Y is the target\nfrom sklearn.preprocessing import MultiLabelBinarizer, LabelEncoder\nbinarize = MultiLabelBinarizer(classes=classes)\nencode = LabelEncoder()\nY_split = encode.fit_transform(first_labels_set)\nY = binarize.fit_transform(all_labels_set)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##\n## Read all training files and keep them in memory\nfrom tqdm import tqdm_notebook\nX_raw = []\nfor f in tqdm_notebook(files):\n    sample_rate, sample_ = wavfile.read(str(train_audio_path) + f)\n    X_raw.append(sample_)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##\n## Nice helper functions for padding, random sampling L samples\n\ndef pad_audio(samples):\n    if len(samples) >= L: return samples\n    else: return np.pad(samples, pad_width=(L - len(samples), 0), mode='constant', constant_values=(0, 0))\n\n# 150000 , 44100 ->  [0, .,......., (150000-44100)]\ndef chop_audio(samples):\n    beg = np.random.randint(0, len(samples) - L)\n    return samples[beg: beg + L]\n        \n        \ndef log_specgram(audio, \n                 sample_rate, \n                 window_size=20,\n                 step_size=10, eps=1e-10):\n    nperseg = int(round(window_size * sample_rate / 1e3))\n    noverlap = int(round(step_size * sample_rate / 1e3))\n    freqs, times, spec = signal.spectrogram(audio,\n                                    fs=sample_rate,\n                                    window='hann',\n                                    nperseg=nperseg,\n                                    noverlap=noverlap,\n                                    detrend=False)\n    return freqs, times, np.log(spec.astype(np.float32) + eps)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##\n## DataGenerator based on keras.utils.Sequence. The nice thing about it is the random part selection that works like augmentation.\n## TestDataGenerator is bad software engineering from my part... Essentially the same generator used only for inference...\n\nimport numpy as np\nimport keras\n\nclass DataGenerator(keras.utils.Sequence):\n    'Generates data for Keras'\n    def __init__(self, list_IDs, labels, batch_size=32, dim=(256,256,1), n_channels=1,\n                 n_classes=80, shuffle=True):\n        'Initialization'\n        self.dim = dim\n        self.batch_size = batch_size\n        self.labels = labels\n        self.list_IDs = list_IDs\n        self.n_channels = n_channels\n        self.n_classes = n_classes\n        self.shuffle = shuffle\n        self.on_epoch_end()\n\n    def __len__(self):\n        'Denotes the number of batches per epoch'\n        return int(np.ceil(len(self.list_IDs) / self.batch_size))\n\n    def __getitem__(self, index):\n        'Generate one batch of data'\n        # Generate indexes of the batch\n        max_index = min((index+1)*self.batch_size, len(self.list_IDs))\n        indexes = self.indexes[index*self.batch_size:max_index]\n\n        # Find list of IDs\n        list_IDs_temp = [self.list_IDs[k] for k in indexes]\n\n        # Generate data\n        X, y = self.__data_generation(list_IDs_temp)\n\n        return X, y\n\n    def on_epoch_end(self):\n        'Updates indexes after each epoch'\n        self.indexes = np.arange(len(self.list_IDs))\n        if self.shuffle == True:\n            np.random.shuffle(self.indexes)\n\n    def __data_generation(self, list_IDs_temp):\n        'Generates data containing batch_size samples' # X : (n_samples, *dim, n_channels)\n        # Initialization\n        X = []# np.empty((self.batch_size, *self.dim, self.n_channels))\n        y = []# np.empty((self.batch_size), dtype=int)\n        t1 = time.time()\n        #print (list_IDs_temp)\n\n        # Generate data\n        for i, ID in enumerate(list_IDs_temp):\n            #print (i, ID)\n            # Store samplw\n            xx = X_raw[ID].copy()\n    \n            xx = pad_audio(xx)\n            if len(xx) > L:\n                xx = chop_audio(xx)\n            _, _, specgram = log_specgram(xx, sample_rate=SAMPLE_RATE,  window_size=10, step_size=5)\n            X.append(specgram)\n\n            # Store class\n            y.append(self.labels[ID, :])\n            \n        t2 = time.time()\n        #print (t2-t1)\n        y = np.array(y, dtype='float32')\n        X = np.expand_dims(np.array(X), -1)\n        return X, y\n\nclass TestDataGenerator(keras.utils.Sequence):\n    'Generates data for Keras'\n    def __init__(self, test_files, test_base_path, batch_size=32, dim=(256,256,1), n_channels=1,\n                 n_classes=80):\n        'Initialization'\n        self.dim = dim\n        self.batch_size = batch_size\n        self.test_base_path = test_base_path\n        self.test_files = test_files\n        self.n_channels = n_channels\n        self.n_classes = n_classes\n        self.on_epoch_end()\n\n    def __len__(self):\n        'Denotes the number of batches per epoch'\n        return int(np.ceil(len(self.test_files) / self.batch_size))\n\n    def __getitem__(self, index):\n        'Generate one batch of data'\n        # Generate indexes of the batch\n        max_index = min((index+1)*self.batch_size, len(self.test_files))\n        indexes = self.indexes[index*self.batch_size:max_index]\n\n        # Find list of IDs\n        list_IDs_temp = [self.test_files[k] for k in indexes]\n\n        # Generate data\n        X = self.__data_generation(list_IDs_temp)\n\n        return X\n\n    def on_epoch_end(self):\n        'Updates indexes after each epoch'\n        self.indexes = np.arange(len(self.test_files))\n\n\n    def __data_generation(self, list_IDs_temp):\n        'Generates data containing batch_size samples' # X : (n_samples, *dim, n_channels)\n        # Initialization\n        X = []# np.empty((self.batch_size, *self.dim, self.n_channels))\n\n        # Generate data\n        for i, ID in enumerate(list_IDs_temp):\n            # Store samplw\n            sample_rate, xx = wavfile.read(str(self.test_base_path) + ID)\n            xx = pad_audio(xx)\n            if len(xx) > L:\n                xx = chop_audio(xx)\n            _, _, specgram = log_specgram(xx, sample_rate=SAMPLE_RATE,  window_size=10, step_size=5)\n            X.append(specgram)\n\n        X = np.expand_dims(np.array(X), -1)\n        return X","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.layers import (Convolution2D, GlobalAveragePooling2D, BatchNormalization, Flatten,\n                          GlobalMaxPool2D, MaxPool2D, concatenate, Activation, Input, Dense, Dropout)\nfrom keras.utils import Sequence, to_categorical\nfrom keras.optimizers import Adam\nfrom keras.losses import binary_crossentropy\nfrom keras.models import Model\nfrom keras import backend as K\n\ndef get_2d_conv_model(input_shape= (221, 298, 1), n_classes=80, learning_rate=0.001):\n    \n    nclass = n_classes\n    \n    inp = Input(shape=(input_shape[0],input_shape[1],1))\n    x = Convolution2D(96, (4,10), padding=\"same\")(inp)\n    x = BatchNormalization()(x)\n    x = Activation(\"relu\")(x)\n    x = MaxPool2D()(x)\n    \n    x = Convolution2D(64, (4,10), padding=\"same\")(x)\n    x = BatchNormalization()(x)\n    x = Activation(\"relu\")(x)\n    x = MaxPool2D()(x)\n    \n    x = Convolution2D(48, (4,10), padding=\"same\")(x)\n    x = BatchNormalization()(x)\n    x = Activation(\"relu\")(x)\n    x = MaxPool2D()(x)\n    \n    x = Convolution2D(32, (4,10), padding=\"same\")(x)\n    x = BatchNormalization()(x)\n    x = Activation(\"relu\")(x)\n    x = MaxPool2D()(x)\n    x = Flatten()(x)\n    \n    x = Dropout(0.5)(x)\n    x = Dense(80)(x)\n    x = BatchNormalization()(x)\n    x = Activation(\"relu\")(x)\n    \n    x = Dense(80)(x)\n    x = BatchNormalization()(x)\n    x = Activation(\"relu\")(x)\n    \n    out = Dense(nclass, activation='softmax')(x)\n\n    model = Model(inputs=inp, outputs=out)\n    opt = Adam(learning_rate)\n\n    model.compile(optimizer=opt, loss=binary_crossentropy, metrics=['categorical_accuracy'])\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.callbacks import ModelCheckpoint, EarlyStopping, ReduceLROnPlateau\nmodelname = 'custom-v1-'\ntest_gen = TestDataGenerator(test_files, test_audio_path, batch_size=128)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":false},"cell_type":"code","source":"from sklearn.model_selection import StratifiedKFold\noof_y = np.zeros_like(Y, dtype='float32')\ntest_Y = np.zeros((len(test_files), 80), dtype='float32')\n\n#90%/10% Split\nkfold = StratifiedKFold(10, True, 6996 )\nifold = 0\nfor train_index, valid_index in kfold.split(X_raw, Y_split):\n    if ifold<=1:\n        print(\"--------------------------------------------\")\n\n        checkpoint = ModelCheckpoint(modelname + str(ifold) + '.hdf5', monitor='val_categorical_accuracy', verbose=1, \n                                     save_best_only=True, save_weights_only=True, mode='auto', period=1)\n        early = EarlyStopping(monitor='val_categorical_accuracy', min_delta=0.001, patience=12, verbose=1, mode='auto', restore_best_weights=True)\n\n        reduce_lr = ReduceLROnPlateau(monitor='val_categorical_accuracy', factor=0.5, patience=6, verbose=1)\n\n        model = get_2d_conv_model()\n        train_gen = DataGenerator(train_index, Y, batch_size=32, n_classes=80, shuffle=True)\n        valid_gen = DataGenerator(valid_index, Y, batch_size=128, n_classes=80, shuffle=False)\n\n#         if USE_WEIGHTS == True:\n#             print ('Loading from ', '../input/custom-cnn-1-sec/' + modelname + str(ifold) + '.hdf5')\n#             model.load_weights('../input/custom-cnn-1-sec/' + modelname + str(ifold) + '.hdf5')\n#         else:\n        model.fit_generator(train_gen, epochs=60, verbose=0, callbacks=[checkpoint, early, reduce_lr], validation_data=valid_gen)\n\n        res = model.predict_generator(valid_gen, verbose=0)\n        #for bag in range(9):\n        #    res += model.predict_generator(valid_gen, verbose=0)\n\n        res_Y = model.predict_generator(test_gen, verbose=0)\n        for bag in range(9):\n            res_Y += model.predict_generator(test_gen, verbose=0)\n\n        oof_y[valid_index, ] = res\n        test_Y = test_Y + res_Y / 10.\n    ifold = ifold + 1\n    \ntest_Y = test_Y / 5\n","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"import numpy as np\nimport sklearn.metrics\n# Core calculation of label precisions for one test sample.\n\ndef _one_sample_positive_class_precisions(scores, truth):\n  \"\"\"Calculate precisions for each true class for a single sample.\n  \n  Args:\n    scores: np.array of (num_classes,) giving the individual classifier scores.\n    truth: np.array of (num_classes,) bools indicating which classes are true.\n\n  Returns:\n    pos_class_indices: np.array of indices of the true classes for this sample.\n    pos_class_precisions: np.array of precisions corresponding to each of those\n      classes.\n  \"\"\"\n  num_classes = scores.shape[0]\n  pos_class_indices = np.flatnonzero(truth > 0)\n  # Only calculate precisions if there are some true classes.\n  if not len(pos_class_indices):\n    return pos_class_indices, np.zeros(0)\n  # Retrieval list of classes for this sample. \n  retrieved_classes = np.argsort(scores)[::-1]\n  # class_rankings[top_scoring_class_index] == 0 etc.\n  class_rankings = np.zeros(num_classes, dtype=np.int)\n  class_rankings[retrieved_classes] = range(num_classes)\n  # Which of these is a true label?\n  retrieved_class_true = np.zeros(num_classes, dtype=np.bool)\n  retrieved_class_true[class_rankings[pos_class_indices]] = True\n  # Num hits for every truncated retrieval list.\n  retrieved_cumulative_hits = np.cumsum(retrieved_class_true)\n  # Precision of retrieval list truncated at each hit, in order of pos_labels.\n  precision_at_hits = (\n      retrieved_cumulative_hits[class_rankings[pos_class_indices]] / \n      (1 + class_rankings[pos_class_indices].astype(np.float)))\n  return pos_class_indices, precision_at_hits\n\n# All-in-one calculation of per-class lwlrap.\n\ndef calculate_per_class_lwlrap(truth, scores):\n  \"\"\"Calculate label-weighted label-ranking average precision.\n  \n  Arguments:\n    truth: np.array of (num_samples, num_classes) giving boolean ground-truth\n      of presence of that class in that sample.\n    scores: np.array of (num_samples, num_classes) giving the classifier-under-\n      test's real-valued score for each class for each sample.\n  \n  Returns:\n    per_class_lwlrap: np.array of (num_classes,) giving the lwlrap for each \n      class.\n    weight_per_class: np.array of (num_classes,) giving the prior of each \n      class within the truth labels.  Then the overall unbalanced lwlrap is \n      simply np.sum(per_class_lwlrap * weight_per_class)\n  \"\"\"\n  assert truth.shape == scores.shape\n  num_samples, num_classes = scores.shape\n  # Space to store a distinct precision value for each class on each sample.\n  # Only the classes that are true for each sample will be filled in.\n  precisions_for_samples_by_classes = np.zeros((num_samples, num_classes))\n  for sample_num in range(num_samples):\n    pos_class_indices, precision_at_hits = (\n      _one_sample_positive_class_precisions(scores[sample_num, :], \n                                            truth[sample_num, :]))\n    precisions_for_samples_by_classes[sample_num, pos_class_indices] = (\n        precision_at_hits)\n  labels_per_class = np.sum(truth > 0, axis=0)\n  weight_per_class = labels_per_class / float(np.sum(labels_per_class))\n  # Form average of each column, i.e. all the precisions assigned to labels in\n  # a particular class.\n  per_class_lwlrap = (np.sum(precisions_for_samples_by_classes, axis=0) / \n                      np.maximum(1, labels_per_class))\n  # overall_lwlrap = simple average of all the actual per-class, per-sample precisions\n  #                = np.sum(precisions_for_samples_by_classes) / np.sum(precisions_for_samples_by_classes > 0)\n  #           also = weighted mean of per-class lwlraps, weighted by class label prior across samples\n  #                = np.sum(per_class_lwlrap * weight_per_class)\n  return per_class_lwlrap, weight_per_class\n\n# Calculate the overall lwlrap using sklearn.metrics function.\n\ndef calculate_overall_lwlrap_sklearn(truth, scores):\n  \"\"\"Calculate the overall lwlrap using sklearn.metrics.lrap.\"\"\"\n  # sklearn doesn't correctly apply weighting to samples with no labels, so just skip them.\n  sample_weight = np.sum(truth > 0, axis=1)\n  nonzero_weight_sample_indices = np.flatnonzero(sample_weight > 0)\n  overall_lwlrap = sklearn.metrics.label_ranking_average_precision_score(\n      truth[nonzero_weight_sample_indices, :] > 0, \n      scores[nonzero_weight_sample_indices, :], \n      sample_weight=sample_weight[nonzero_weight_sample_indices])\n  return overall_lwlrap\n\n\n# Accumulator object version.\n\nclass lwlrap_accumulator(object):\n  \"\"\"Accumulate batches of test samples into per-class and overall lwlrap.\"\"\"  \n\n  def __init__(self):\n    self.num_classes = 0\n    self.total_num_samples = 0\n  \n  def accumulate_samples(self, batch_truth, batch_scores):\n    \"\"\"Cumulate a new batch of samples into the metric.\n    \n    Args:\n      truth: np.array of (num_samples, num_classes) giving boolean\n        ground-truth of presence of that class in that sample for this batch.\n      scores: np.array of (num_samples, num_classes) giving the \n        classifier-under-test's real-valued score for each class for each\n        sample.\n    \"\"\"\n    assert batch_scores.shape == batch_truth.shape\n    num_samples, num_classes = batch_truth.shape\n    if not self.num_classes:\n      self.num_classes = num_classes\n      self._per_class_cumulative_precision = np.zeros(self.num_classes)\n      self._per_class_cumulative_count = np.zeros(self.num_classes, \n                                                  dtype=np.int)\n    assert num_classes == self.num_classes\n    for truth, scores in zip(batch_truth, batch_scores):\n      pos_class_indices, precision_at_hits = (\n        _one_sample_positive_class_precisions(scores, truth))\n      self._per_class_cumulative_precision[pos_class_indices] += (\n        precision_at_hits)\n      self._per_class_cumulative_count[pos_class_indices] += 1\n    self.total_num_samples += num_samples\n\n  def per_class_lwlrap(self):\n    \"\"\"Return a vector of the per-class lwlraps for the accumulated samples.\"\"\"\n    return (self._per_class_cumulative_precision / \n            np.maximum(1, self._per_class_cumulative_count))\n\n  def per_class_weight(self):\n    \"\"\"Return a normalized weight vector for the contributions of each class.\"\"\"\n    return (self._per_class_cumulative_count / \n            float(np.sum(self._per_class_cumulative_count)))\n\n  def overall_lwlrap(self):\n    \"\"\"Return the scalar overall lwlrap for cumulated samples.\"\"\"\n    return np.sum(self.per_class_lwlrap() * self.per_class_weight())\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"truth = Y\nscores = oof_y\nprint(\"lwlrap from sklearn.metrics =\", calculate_overall_lwlrap_sklearn(truth, scores))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{"trusted":true},"cell_type":"code","source":"sort_idx = np.argsort(classes).astype(int)\nsample_sub = pd.read_csv('../input/freesound-audio-tagging-2019/sample_submission.csv')\ntest_Y_sort = test_Y[:, sort_idx]\nsample_sub.iloc[:, 1:] =  test_Y_sort\nsample_sub.to_csv('submission.csv', index=False)\n\nt2 = time.time()\nprint ('Total time: ', (t2-t1))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_sub.head()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}