{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Inference kernel using fastai\n\n[![header](https://raw.githubusercontent.com/ebouteillon/freesound-audio-tagging-2019/master/images/header.png)](https://github.com/ebouteillon/freesound-audio-tagging-2019/)\n\nThis is the inference kernel I used as submission to the *Freesound Audio Tagging 2019* competition with some additional editorial changes to make things clearer.\n\nOn [my github repository](https://github.com/ebouteillon/freesound-audio-tagging-2019/), you will find more about the training of the models used here (e.g. about the **warm-up pipeline** training or **SpecMix** data augmentation used."},{"metadata":{"trusted":true,"_kg_hide-input":false,"_kg_hide-output":false},"cell_type":"code","source":"import os\nfrom pathlib import Path\nimport pickle\nimport random\nimport time\nfrom io import StringIO\nfrom csv import writer\nimport gc\n\nimport numpy as np\nimport pandas as pd\nimport librosa\nimport librosa.display\nimport matplotlib.pyplot as plt\nfrom tqdm import tqdm_notebook\nimport IPython\nimport IPython.display\n# import PIL\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\n\nfrom fastai import *\nfrom fastai.vision import *\nfrom fastai.vision.data import *","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We define here the list of models that will use to predict our solution:"},{"metadata":{"trusted":true},"cell_type":"code","source":"models_list = (\n    # trained weights of CNN-model-1 inspired by mhiro2\n    (Path('../input/freesoundaudiotagging2019ebouteillonsolution/fat2019ssl4multistage/work/'), 'stage-2_fold-{fold}.pkl'),\n    (Path('../input/freesoundaudiotagging2019ebouteillonsolution/fat2019ssl4multistage/work/'), 'stage-10_fold-{fold}.pkl'),\n    (Path('../input/freesoundaudiotagging2019ebouteillonsolution/fat2019ssl4multistage/work/'), 'stage-11_fold-{fold}.pkl'),\n    # trained weights of VGG-16\n    (Path('../input/freesoundaudiotagging2019ebouteillonsolution/fat2019ssl8vgg16full/work'), 'stage-2_fold-{fold}.pkl'),\n    (Path('../input/freesoundaudiotagging2019ebouteillonsolution/fat2019ssl8vgg16full/work'), 'stage-10_fold-{fold}.pkl'),\n    (Path('../input/freesoundaudiotagging2019ebouteillonsolution/fat2019ssl8vgg16full/work'), 'stage-11_fold-{fold}.pkl'),\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"PREDICTION_WINDOW_SHIFT = 48  # predict every PREDICTION_WINDOW_SHIFT time sample\nn_splits = 10\nDATA = Path('../input/freesound-audio-tagging-2019')\nDATA_TEST = DATA/'test'\nCSV_SUBMISSION = DATA/'sample_submission.csv'\ntest_df = pd.read_csv(CSV_SUBMISSION)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Preprocessing\n\nNow we are going to generate all the mel-spectrograms of all test samples and keep them in memory. We are doing this once to save up time.\n\nThis code is borrowed from great [daisuke kernel](https://www.kaggle.com/daisukelab/creating-fat2019-preprocessed-data)"},{"metadata":{"trusted":true},"cell_type":"code","source":"def read_audio(conf, pathname, trim_long_data):\n    y, sr = librosa.load(pathname, sr=conf.sampling_rate)\n    # trim silence\n    if 0 < len(y): # workaround: 0 length causes error\n        y, _ = librosa.effects.trim(y) # trim, top_db=default(60)\n    # make it unified length to conf.samples\n    if len(y) > conf.samples: # long enough\n        if trim_long_data:\n            y = y[0:0+conf.samples]\n    else: # pad blank\n        padding = conf.samples - len(y)    # add padding at both ends\n        offset = padding // 2\n        y = np.pad(y, (offset, conf.samples - len(y) - offset), 'constant')\n    return y\n\ndef audio_to_melspectrogram(conf, audio):\n    spectrogram = librosa.feature.melspectrogram(audio, \n                                                 sr=conf.sampling_rate,\n                                                 n_mels=conf.n_mels,\n                                                 hop_length=conf.hop_length,\n                                                 n_fft=conf.n_fft,\n                                                 fmin=conf.fmin,\n                                                 fmax=conf.fmax)\n    spectrogram = librosa.power_to_db(spectrogram)\n    spectrogram = spectrogram.astype(np.float32)\n    return spectrogram\n\ndef show_melspectrogram(conf, mels, title='Log-frequency power spectrogram'):\n    librosa.display.specshow(mels, x_axis='time', y_axis='mel', \n                             sr=conf.sampling_rate, hop_length=conf.hop_length,\n                            fmin=conf.fmin, fmax=conf.fmax)\n    plt.colorbar(format='%+2.0f dB')\n    plt.title(title)\n    plt.show()\n\ndef read_as_melspectrogram(conf, pathname, trim_long_data, debug_display=False):\n    x = read_audio(conf, pathname, trim_long_data)\n    mels = audio_to_melspectrogram(conf, x)\n    if debug_display:\n        IPython.display.display(IPython.display.Audio(x, rate=conf.sampling_rate))\n        show_melspectrogram(conf, mels)\n    return mels\n\n\nclass conf:\n    # Preprocessing settings\n    sampling_rate = 44100\n    duration = 2\n    hop_length = 347*duration # to make time steps 128\n    fmin = 20\n    fmax = sampling_rate // 2\n    n_mels = 128\n    n_fft = n_mels * 20\n    samples = sampling_rate * duration\n\n\ndef mono_to_color(X, mean=None, std=None, norm_max=None, norm_min=None, eps=1e-6):\n    # Stack X as [X,X,X]\n    X = np.stack([X, X, X], axis=-1)\n\n    # Standardize\n    mean = mean or X.mean()\n    std = std or X.std()\n    Xstd = (X - mean) / (std + eps)\n    _min, _max = Xstd.min(), Xstd.max()\n    norm_max = norm_max or _max\n    norm_min = norm_min or _min\n    if (_max - _min) > eps:\n        # Scale to [0, 255]\n        V = Xstd\n        V[V < norm_min] = norm_min\n        V[V > norm_max] = norm_max\n        V = 255 * (V - norm_min) / (norm_max - norm_min)\n        V = V.astype(np.uint8)\n    else:\n        # Just zero\n        V = np.zeros_like(Xstd, dtype=np.uint8)\n    return V\n\ndef convert_wav_to_image(df, source, img_dest):\n    print(f'Converting {source} -> {img_dest}')\n    X = []\n    for i, row in tqdm_notebook(df.iterrows(), total=df.shape[0]):\n        x = read_as_melspectrogram(conf, source/str(row.fname), trim_long_data=False)\n        x_color = mono_to_color(x)\n        X.append(x_color)\n    return X","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_test = convert_wav_to_image(test_df, source=DATA_TEST, img_dest=None)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Fastai callbacks\n\nWeights were pickled and have references to callbacks we implemented and used during training. We provide them here or a useless conterparts to make everyone happy."},{"metadata":{"trusted":true},"cell_type":"code","source":"# This implemented my new data augmentation technique 'SpecMix', see my github for more details :)\nclass MyMixUpCallback(LearnerCallback):\n    def __init__(self, learn:Learner):\n        super().__init__(learn)\n        \nclass Lwlrap(Callback):\n    def on_epoch_begin(self, **kwargs):\n        pass","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# from official code https://colab.research.google.com/drive/1AgPdhSp7ttY18O3fEoHOQKlt_3HJDLi8#scrollTo=cRCaCIb9oguU\ndef _one_sample_positive_class_precisions(scores, truth):\n    \"\"\"Calculate precisions for each true class for a single sample.\n\n    Args:\n      scores: np.array of (num_classes,) giving the individual classifier scores.\n      truth: np.array of (num_classes,) bools indicating which classes are true.\n\n    Returns:\n      pos_class_indices: np.array of indices of the true classes for this sample.\n      pos_class_precisions: np.array of precisions corresponding to each of those\n        classes.\n    \"\"\"\n    num_classes = scores.shape[0]\n    pos_class_indices = np.flatnonzero(truth > 0)\n    # Only calculate precisions if there are some true classes.\n    if not len(pos_class_indices):\n        return pos_class_indices, np.zeros(0)\n    # Retrieval list of classes for this sample.\n    retrieved_classes = np.argsort(scores)[::-1]\n    # class_rankings[top_scoring_class_index] == 0 etc.\n    class_rankings = np.zeros(num_classes, dtype=np.int)\n    class_rankings[retrieved_classes] = range(num_classes)\n    # Which of these is a true label?\n    retrieved_class_true = np.zeros(num_classes, dtype=np.bool)\n    retrieved_class_true[class_rankings[pos_class_indices]] = True\n    # Num hits for every truncated retrieval list.\n    retrieved_cumulative_hits = np.cumsum(retrieved_class_true)\n    # Precision of retrieval list truncated at each hit, in order of pos_labels.\n    precision_at_hits = (\n            retrieved_cumulative_hits[class_rankings[pos_class_indices]] /\n            (1 + class_rankings[pos_class_indices].astype(np.float)))\n    return pos_class_indices, precision_at_hits\n\n\ndef calculate_per_class_lwlrap(truth, scores):\n    \"\"\"Calculate label-weighted label-ranking average precision.\n\n    Arguments:\n      truth: np.array of (num_samples, num_classes) giving boolean ground-truth\n        of presence of that class in that sample.\n      scores: np.array of (num_samples, num_classes) giving the classifier-under-\n        test's real-valued score for each class for each sample.\n\n    Returns:\n      per_class_lwlrap: np.array of (num_classes,) giving the lwlrap for each\n        class.\n      weight_per_class: np.array of (num_classes,) giving the prior of each\n        class within the truth labels.  Then the overall unbalanced lwlrap is\n        simply np.sum(per_class_lwlrap * weight_per_class)\n    \"\"\"\n    assert truth.shape == scores.shape\n    num_samples, num_classes = scores.shape\n    # Space to store a distinct precision value for each class on each sample.\n    # Only the classes that are true for each sample will be filled in.\n    precisions_for_samples_by_classes = np.zeros((num_samples, num_classes))\n    for sample_num in range(num_samples):\n        pos_class_indices, precision_at_hits = (\n            _one_sample_positive_class_precisions(scores[sample_num, :],\n                                                  truth[sample_num, :]))\n        precisions_for_samples_by_classes[sample_num, pos_class_indices] = (\n            precision_at_hits)\n    labels_per_class = np.sum(truth > 0, axis=0)\n    weight_per_class = labels_per_class / float(np.sum(labels_per_class))\n    # Form average of each column, i.e. all the precisions assigned to labels in\n    # a particular class.\n    per_class_lwlrap = (np.sum(precisions_for_samples_by_classes, axis=0) /\n                        np.maximum(1, labels_per_class))\n    # overall_lwlrap = simple average of all the actual per-class, per-sample precisions\n    #                = np.sum(precisions_for_samples_by_classes) / np.sum(precisions_for_samples_by_classes > 0)\n    #           also = weighted mean of per-class lwlraps, weighted by class label prior across samples\n    #                = np.sum(per_class_lwlrap * weight_per_class)\n    return per_class_lwlrap, weight_per_class\n\n\n# Accumulator object version.\n\nclass lwlrap_accumulator(object):\n  \"\"\"Accumulate batches of test samples into per-class and overall lwlrap.\"\"\"  \n\n  def __init__(self):\n    self.num_classes = 0\n    self.total_num_samples = 0\n  \n  def accumulate_samples(self, batch_truth, batch_scores):\n    \"\"\"Cumulate a new batch of samples into the metric.\n    \n    Args:\n      truth: np.array of (num_samples, num_classes) giving boolean\n        ground-truth of presence of that class in that sample for this batch.\n      scores: np.array of (num_samples, num_classes) giving the \n        classifier-under-test's real-valued score for each class for each\n        sample.\n    \"\"\"\n    assert batch_scores.shape == batch_truth.shape\n    num_samples, num_classes = batch_truth.shape\n    if not self.num_classes:\n      self.num_classes = num_classes\n      self._per_class_cumulative_precision = np.zeros(self.num_classes)\n      self._per_class_cumulative_count = np.zeros(self.num_classes, \n                                                  dtype=np.int)\n    assert num_classes == self.num_classes\n    for truth, scores in zip(batch_truth, batch_scores):\n      pos_class_indices, precision_at_hits = (\n        _one_sample_positive_class_precisions(scores, truth))\n      self._per_class_cumulative_precision[pos_class_indices] += (\n        precision_at_hits)\n      self._per_class_cumulative_count[pos_class_indices] += 1\n    self.total_num_samples += num_samples\n\n  def per_class_lwlrap(self):\n    \"\"\"Return a vector of the per-class lwlraps for the accumulated samples.\"\"\"\n    return (self._per_class_cumulative_precision / \n            np.maximum(1, self._per_class_cumulative_count))\n\n  def per_class_weight(self):\n    \"\"\"Return a normalized weight vector for the contributions of each class.\"\"\"\n    return (self._per_class_cumulative_count / \n            float(np.sum(self._per_class_cumulative_count)))\n\n  def overall_lwlrap(self):\n    \"\"\"Return the scalar overall lwlrap for cumulated samples.\"\"\"\n    return np.sum(self.per_class_lwlrap() * self.per_class_weight())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Model inspired from mhiro2 and implemented by [daisuke kernel](https://www.kaggle.com/daisukelab/cnn-2d-basic-3-using-simple-model) :"},{"metadata":{"trusted":true},"cell_type":"code","source":"class ConvBlock(nn.Module):\n    def __init__(self, in_channels, out_channels):\n        super().__init__()\n        \n        self.conv1 = nn.Sequential(\n            nn.Conv2d(in_channels, out_channels, 3, 1, 1),\n            nn.BatchNorm2d(out_channels),\n            nn.ReLU(),\n        )\n        self.conv2 = nn.Sequential(\n            nn.Conv2d(out_channels, out_channels, 3, 1, 1),\n            nn.BatchNorm2d(out_channels),\n            nn.ReLU(),\n        )\n\n        self._init_weights()\n        \n    def _init_weights(self):\n        for m in self.modules():\n            if isinstance(m, nn.Conv2d):\n                nn.init.kaiming_normal_(m.weight)\n                if m.bias is not None:\n                    nn.init.zeros_(m.bias)\n            elif isinstance(m, nn.BatchNorm2d):\n                nn.init.constant_(m.weight, 1)\n                nn.init.zeros_(m.bias)\n        \n    def forward(self, x):\n        x = self.conv1(x)\n        x = self.conv2(x)\n        x = F.avg_pool2d(x, 2)\n        return x\n    \nclass Classifier(nn.Module):\n    def __init__(self, num_classes=1000): # <======== modificaition to comply fast.ai\n        super().__init__()\n        \n        self.conv = nn.Sequential(\n            ConvBlock(in_channels=3, out_channels=64),\n            ConvBlock(in_channels=64, out_channels=128),\n            ConvBlock(in_channels=128, out_channels=256),\n            ConvBlock(in_channels=256, out_channels=512),\n        )\n        self.avgpool = nn.AdaptiveAvgPool2d((1, 1)) # <======== modificaition to comply fast.ai\n        self.fc = nn.Sequential(\n            nn.Dropout(0.2),\n            nn.Linear(512, 128),\n            nn.PReLU(),\n            nn.BatchNorm1d(128),\n            nn.Dropout(0.1),\n            nn.Linear(128, num_classes),\n        )\n\n    def forward(self, x):\n        x = self.conv(x)\n        #x = torch.mean(x, dim=3)   # <======== modificaition to comply fast.ai\n        #x, _ = torch.max(x, dim=2) # <======== modificaition to comply fast.ai\n        x = self.avgpool(x)         # <======== modificaition to comply fast.ai\n        x = self.fc(x)\n        return x","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In the custom image opening function for fastai, we are providing as input in the filename: an identifier of the mel-spectrogram to use, as well as the position of the window to crop on the mel-spectrogram.\n\nIn order to shave some seconds on the inference, I removed the conversion from and to image PIL format."},{"metadata":{"trusted":true},"cell_type":"code","source":"# !!! use globals CUR_X_FILES, CUR_X\ndef open_fat2019_image(fn, convert_mode, after_open)->Image:\n    # open\n    fname = fn.split('/')[-1]\n    if '!' in fname:\n        fname, crop_x = fname.split('!')\n        crop_x = int(crop_x)\n    else:\n        crop_x = -1\n    idx = CUR_X_FILES.index(fname)\n    x = CUR_X[idx]\n    # crop\n    base_dim, time_dim, _ = x.shape\n    if crop_x == -1:\n        crop_x = random.randint(0, time_dim - base_dim)\n    x = x[0:base_dim, crop_x:crop_x+base_dim, :]\n    x = np.transpose(x, (1, 0, 2))\n    x = np.transpose(x, (2, 1, 0))\n    # standardize\n    return Image(torch.from_numpy(x.astype(np.float32, copy=False)).div_(255))\n\n\nvision.data.open_image = open_fat2019_image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"CUR_X_FILES, CUR_X = list(test_df.fname.values), X_test","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Create list of elements to predict\n\nfastai uses a dataframe as input to define list of elements to predict. Therefore, we create here a list of elements with shift in time to predict (see variable `PREDICTION_WINDOW_SHIFT`).\n\nInserting new rows in a pandas dataframe is awfully slow, instead we create a CSV file in-memory and then load it as a pandas dataframe, it is **much** faster."},{"metadata":{"trusted":true},"cell_type":"code","source":"output = StringIO()\ncsv_writer = writer(output)\ncsv_writer.writerow(test_df.columns)\n\nfor _, row in tqdm_notebook(test_df.iterrows(), total=test_df.shape[0]):\n    idx = CUR_X_FILES.index(row.fname)\n    time_dim = CUR_X[idx].shape[1]\n    s = math.ceil((time_dim-conf.n_mels) / PREDICTION_WINDOW_SHIFT) + 1\n    \n    fname = row.fname\n    for crop_x in [int(np.around((time_dim-conf.n_mels)*x/(s-1))) if s != 1 else 0 for x in range(s)]:\n        row.fname = fname + '!' + str(crop_x)\n        csv_writer.writerow(row)\n\noutput.seek(0)\ntest_df_multi = pd.read_csv(output)\n\ndel row, test_df, output, csv_writer; gc.collect();","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Prediction time!\n\nAveragre prediction given for each entry in the dataframe by our models."},{"metadata":{"trusted":true},"cell_type":"code","source":"test = ImageList.from_df(test_df_multi, models_list[0][0])\n\nfor model_nb, (work, name) in enumerate(models_list):\n    for fold in range(n_splits):\n        learn = load_learner(work, name.format(fold=fold), test=test)\n        preds, _ = learn.get_preds(ds_type=DatasetType.Test)\n        preds = preds.cpu().numpy()\n        if (fold == 0) and (model_nb == 0):\n            predictions = preds\n        else:\n            predictions += preds\n\npredictions /= (n_splits * len(models_list))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Average all predictions for a same test sample audio clip:"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df_multi[learn.data.classes] = predictions\ntest_df_multi['fname'] = test_df_multi.fname.apply(lambda x: x.split('!')[0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Generate submission.csv file\n\nGenerate file and display first lines to visually check everything is OK."},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = test_df_multi.infer_objects().groupby('fname').mean().reset_index()\nsubmission.to_csv('submission.csv', index=False)\nsubmission.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Display the most probable target for some test sample audio clip:"},{"metadata":{"trusted":true},"cell_type":"code","source":"submission.set_index('fname').idxmax(1)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.3"}},"nbformat":4,"nbformat_minor":1}