{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Freesound Audio Tagging 2019\n\n![Logo](https://storage.googleapis.com/kaggle-competitions/kaggle/10700/logos/header.png)\n\nThank you for all the competitors, kaggle teams, and the host of this competition!  \nWe enjoyed a lot during the competition and learned many things.  \n\nNone of us had experience with audio data, and we also didn't get used to use Nerual Networks.  \nWe especially want to thank [@daisukelab](https://www.kaggle.com/daisukelab) for his clear instruction with great kernels and datasets, [@mhiro2](https://www.kaggle.com/mhiro2) for sharing excellent training framework, and [@sailorwei](https://www.kaggle.com/sailorwei) for showing his Inception v3 model in his public kernel. We were helped a lot with public kernels below shared by them.\n\nhttps://www.kaggle.com/daisukelab/cnn-2d-basic-solution-powered-by-fast-ai  \nhttps://www.kaggle.com/mhiro2/simple-2d-cnn-classifier-with-pytorch  \nhttps://www.kaggle.com/sailorwei/fat2019-2d-cnn-with-mixup-lb-0-673\n\nWe've created a kernel to explain team [kaggler-ja] Shirogane's solution.\n\nIn short, our solution is 3 models ensemble, one is **Custom CNN** and the other two are **Inception v3** models.  \nWe used **Log-Melspectrogram** features made with the manner introduced by @daisukelab in his great dataset.\n\nhttps://www.kaggle.com/daisukelab/fat2019_prep_mels1  \n\nWe used noisy data for **warm-up**, just like pre-training the model, and curated data for **fine-tuning** the model, which was trained in earlier stage with noisy data.\n\n**All the works for this competition were done with kaggle kernels**, and we'd like to thank a lot to kaggle for giving us great opportunities and computing resources."},{"metadata":{},"cell_type":"markdown","source":"## Contents\n\n1. [Data Augmentation](#Data_Augmentation)\n  - [pitch shift](#pitch_shift)\n  - [fade](#Fade)\n  - [reverb](#Reverb)\n  - [treble and bass](#Treble_and_Bass)\n  - [equalizing](#Equalizing)\n2. [Crop Policy](#Crop_Policy)\n  - [Uniform Crop](#Uniform_Crop)\n  - [Strength Adaptive Crop](#Strength_Adaptive_Crop)\n3. [Model Structure](#Model_Structure)\n  - [Custom CNN](#Custom_CNN)\n  - [Inception v3](#Inception_v3)\n4. [Augmentation in Batch](#Augmentation_in_Batch)\n  - [blend image augmentation](#blend_image_augmentation)\n5. [Training Strategy](#Training_Strategy)\n  - [Pretrain with Noisy Data](#Pretrain_with_Noisy_Data)\n  - [Training with Curated Data](#Training_with_Curated_Data)\n  - [TTA for Validation](#TTA_for_Validation)\n6. [Ensemble](#Ensemble)"},{"metadata":{},"cell_type":"markdown","source":"## Data Augmentation"},{"metadata":{},"cell_type":"markdown","source":"We created augmented training dataset with [sox](http://sox.sourceforge.net/).  \nSox is an open-source synthesis tool, which is easy to use from command line interface, and is installed to Kaggle kernel by default.\n\nWe can use sox like this."},{"metadata":{"trusted":true},"cell_type":"code","source":"!sox ../input/freesound-audio-tagging-2019/train_curated/004ca909.wav output.wav pitch \"-500\" gain -h","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now check the result of audio transformation."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import gc\nimport random\nimport IPython.display\n\nimport subprocess as sp\n\nimport librosa\nimport librosa.display\n\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\n\nfrom PIL import Image, ImageOps\nfrom torch.utils.data import Dataset\n\nfrom pathlib import Path\nfrom tqdm import tqdm_notebook\n\nfrom fastprogress import master_bar, progress_bar","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def read_audio(conf, pathname, trim_long_data):\n    y, sr = librosa.load(pathname, sr=conf.sampling_rate)\n    if 0 < len(y):\n        y, _ = librosa.effects.trim(y)\n\n    if len(y) > conf.samples:\n        if trim_long_data:\n            y = y[0:0+conf.samples]\n    else:\n        padding = conf.samples - len(y)\n        offset = padding // 2\n        y = np.pad(y, (offset, conf.samples - len(y) - offset), conf.padmode)\n    return y\n\n\ndef audio_to_melspectrogram(conf, audio):\n    spectrogram = librosa.feature.melspectrogram(audio, \n                                                 sr=conf.sampling_rate,\n                                                 n_mels=conf.n_mels,\n                                                 hop_length=conf.hop_length,\n                                                 n_fft=conf.n_fft,\n                                                 fmin=conf.fmin,\n                                                 fmax=conf.fmax)\n    spectrogram = librosa.power_to_db(spectrogram)\n    spectrogram = spectrogram.astype(np.float32)\n    return spectrogram\n\n\ndef show_melspectrogram(conf, mels, title='Log-frequency power spectrogram'):\n    librosa.display.specshow(mels, x_axis='time', y_axis='mel', \n                             sr=conf.sampling_rate, hop_length=conf.hop_length,\n                            fmin=conf.fmin, fmax=conf.fmax)\n    plt.colorbar(format='%+2.0f dB')\n    plt.title(title)\n    plt.show()\n\n\ndef read_as_melspectrogram(conf, pathname, trim_long_data, debug_display=False):\n    x = read_audio(conf, pathname, trim_long_data)\n    mels = audio_to_melspectrogram(conf, x)\n    if debug_display:\n        IPython.display.display(IPython.display.Audio(x, rate=conf.sampling_rate))\n        show_melspectrogram(conf, mels)\n    return mels\n\n\nclass conf:\n    sampling_rate = 44100\n    duration = 2 # sec\n    hop_length = 347*duration # to make time steps 128\n    fmin = 20\n    fmax = sampling_rate // 2\n    n_mels = 128\n    n_fft = n_mels * 20\n    padmode = 'constant'\n    samples = sampling_rate * duration","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The original audio data and its melspectrogram representation are below."},{"metadata":{"trusted":true},"cell_type":"code","source":"curated_dir = Path(\"../input/freesound-audio-tagging-2019/train_curated\")\nread_as_melspectrogram(\n    conf=conf, \n    debug_display=True, \n    trim_long_data=False, \n    pathname=curated_dir / \"004ca909.wav\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And the transformed audio data is like this."},{"metadata":{"trusted":true},"cell_type":"code","source":"read_as_melspectrogram(\n    conf=conf, \n    debug_display=True, \n    trim_long_data=False, \n    pathname=\"output.wav\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We used sox from Python using `subprocess` package and created augmented features with kaggle kernels.\nHere's one of the kernel which created some of the augmented features with `train_curated` dataset.\n\nhttps://www.kaggle.com/hidehisaarai1213/freesound-sox-pitch-fade-reverb-treble-bas\n\nWe created 7 augmented datasets whose size are 4970 each, which means the same size as curated one.  \nHere's the types of augmentation.  \nWe used those augmentation which certainly alter what it sounds like but still can be recoginized as the same class."},{"metadata":{},"cell_type":"markdown","source":"### pitch shift"},{"metadata":{},"cell_type":"markdown","source":"This augmentation changes the pitch of the sound.  \nWe applied both pitch-up and pitch-down.  \nThe degree of pitch shift was fixed to 500 cent."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def show_before_and_after(conf, \n                          mels1, \n                          mels2, \n                          title=\"Log-melspectrogram comparison\"):\n    fig, (axL, axR) = plt.subplots(ncols=2, figsize=(15,4), sharex=True)\n    plt.title(title)\n    plt.subplot(1, 2, 1)\n    librosa.display.specshow(mels1, x_axis='time', y_axis='mel', \n                             sr=conf.sampling_rate, hop_length=conf.hop_length,\n                             fmin=conf.fmin, fmax=conf.fmax)\n    axL.set_title(\"original\")\n    plt.colorbar(format='%+2.0f dB')\n    \n    plt.subplot(1, 2, 2)\n    librosa.display.specshow(mels2, x_axis='time', y_axis='mel', \n                             sr=conf.sampling_rate, hop_length=conf.hop_length,\n                             fmin=conf.fmin, fmax=conf.fmax)\n    axR.set_title(\"transformed\")\n    plt.colorbar(format='%+2.0f dB')\n    plt.show()\n\n\ndef transform_and_compare(conf, pathname: Path, cmd=[]):\n    x = read_audio(conf, pathname, trim_long_data=False)\n    mels = audio_to_melspectrogram(conf, x)        \n    IPython.display.display(IPython.display.Audio(x, rate=conf.sampling_rate))\n\n    cmd = [\"sox\", str(pathname), \"output.wav\"] + cmd\n    sp.run(cmd)\n\n    augmented = read_audio(conf, Path(\"output.wav\"), trim_long_data=False)\n    mels_augmented = audio_to_melspectrogram(conf, augmented)\n    IPython.display.display(IPython.display.Audio(augmented, rate=conf.sampling_rate))\n\n    show_before_and_after(conf, mels, mels_augmented)\n    return mels, mels_augmented","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# 500 cent pitch-down\ntransform_and_compare(conf, curated_dir / \"004ca909.wav\", cmd=[\"pitch\", \"-500\"])\n\n# 500 cent pitch-up\ntransform_and_compare(conf, curated_dir / \"004ca909.wav\", cmd=[\"pitch\", \"+500\"]);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Fade\n\nThis augmentation alter the volume of the sound.   "},{"metadata":{"trusted":true},"cell_type":"code","source":"# fade in and out cyclically\ntransform_and_compare(conf, curated_dir / \"004ca909.wav\", cmd=[\"fade\", \"q\", \"3\"]);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Reverb\n\nThis augmentation add reverberation to the original sound."},{"metadata":{"trusted":true},"cell_type":"code","source":"transform_and_compare(conf, curated_dir / \"004ca909.wav\", cmd=[\"reverb\"]);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Treble and Bass\n\nThis augmentation emphasize or weaken treble and bass range of the sound."},{"metadata":{"trusted":true},"cell_type":"code","source":"# emphasize\ntransform_and_compare(conf, curated_dir / \"004ca909.wav\", cmd=[\"gain\", \"-h\", \"treble\", \"+20\", \"bass\", \"+20\"]);\n\n# weaken\ntransform_and_compare(conf, curated_dir / \"004ca909.wav\", cmd=[\"gain\", \"-h\", \"treble\", \"-30\", \"bass\", \"-30\"]);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Equalizing\n\nThis augmentation adjust custom frequency range."},{"metadata":{"trusted":true},"cell_type":"code","source":"transform_and_compare(conf, curated_dir / \"004ca909.wav\", cmd=[\"gain\", \"-h\", \"equalizer\", \"2400\", \"3q\", \"8\"]);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We applied these augmentations to the whole curated data.  \nWhen we train the model with curated data, we used all of these augmented features, so the size of the train dataset was 8 times larger(7 augmentations + 1 original) than the original curated dataset.\n\nWe also applied pitch shift to the noisy data."},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here's the table which summarize how much each augmentation worked on the score.  \nNote that this score was calcurated with Custom CNN model, and **the effect of the augmentation varies between models**.\n\n|score \\ augmentation|no aug| pitch down | pitch up | fade | reverb | treble and bass down | treble and bass up | equalize | pitch down, fade, reverb, treble and bass down| all|\n|---|---|---|---|---|---|---|---|---|---|\n|CV(no warm-up)|0.800|0.809|0.805|0.805|0.801|0.809|0.808|0.803|**0.829**|0.825|\n|CV(warm-up)|-|-|-|-|-|-|-|-|0.87+|**0.88**|\n|LB|0.690|-|-|-|-|-|-|-|0.720|**0.72x(better than 0.720)**|"},{"metadata":{},"cell_type":"markdown","source":"We didn't checked all the score improvement of Inception v3 model with these augmentation, but **between no aug and with aug, there exist around 0.001 improvement**.\n\n|score\\augmentation|no aug(no warm-up)|pitch down, fade, reverb, treble and bass down(no warm-up)|pitch down, fade, reverb, treble and bass down| all|\n|---|---|---|---|---|\n|CV|we don't remember|0.85|0.87|**0.87+**|\n|LB|0.699?|0.713|0.725|**0.725+**|"},{"metadata":{},"cell_type":"markdown","source":"## Crop Policy"},{"metadata":{},"cell_type":"markdown","source":"### Uniform Crop"},{"metadata":{},"cell_type":"markdown","source":"We use random crop, because we use fixed image size(128 * 128). Random crop got a little better cv than the first 2 seconds cropped. At first, it wa cropped uniformly as mhiro's kernel."},{"metadata":{},"cell_type":"markdown","source":"### Strength Adaptive Crop"},{"metadata":{},"cell_type":"markdown","source":"Many sound crip has important information at the first few seconds. Some sample has it in the middle of crip. However, due to the nature of recording, it is rare to have important information at th end of sounds. The score drops about 0.03~0.04 when learning only the last few seconds.\n\nThen, We introduce Strength Adaptive Crop. We tried to crop the place where the total of db is high preferentially.\n\nThis method is very effective because most samples contain important information in places where the sound is loud.\n\nCV 0.01 up\nLB 0.004~0.005 up"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def _get_strength(x):\n    strength_list_train = []\n    time_dim, base_dim = x.shape[1], x.shape[0]\n    x_sum = []\n    x_max = 0\n    x_min = np.inf\n    for j in range(0,time_dim-base_dim + 1):\n        #x_temp = x[:,j:j + base_dim].sum()\n        x_temp = x[0:84,j:j + base_dim].sum()\n        x_sum.append(x_temp)\n        if x_temp > x_max:\n            x_max = x_temp\n        if x_temp < x_min:\n            x_min = x_temp\n    if (x_max == x_min):\n        height = x_max\n    else:\n        height = x_max - x_min\n    strength_list_train.append([x_sum, height, x_max, x_min, time_dim, base_dim])\n    return strength_list_train\n\n#strength_list_train = _get_strength_list(x_train)\ndef mono_to_color(X, mean=None, std=None, norm_max=None, norm_min=None, eps=1e-6):\n    # Stack X as [X,X,X]\n\n    # Standardize\n    mean = mean or X.mean()\n    X = X - mean\n    std = std or X.std()\n    Xstd = X / (std + eps)\n    _min, _max = Xstd.min(), Xstd.max()\n    norm_max = norm_max or _max\n    norm_min = norm_min or _min\n    if (_max - _min) > eps:\n        # Normalize to [0, 255]\n        V = Xstd\n        V[V < norm_min] = norm_min\n        V[V > norm_max] = norm_max\n        V = 255 * (V - norm_min) / (norm_max - norm_min)\n        V = V.astype(np.uint8)\n    else:\n        # Just zero\n        V = np.zeros_like(Xstd, dtype=np.uint8)\n    return V\n\ndef convert_wav_to_image(df, source):\n    X = []\n    for i, row in tqdm_notebook(df.iterrows()):\n        x = read_as_melspectrogram(conf, source/str(row.fname), trim_long_data=False)\n        x_color = mono_to_color(x)\n        X.append(x_color)\n    return X","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"sample_x = read_as_melspectrogram(\n    conf=conf, \n    debug_display=False, \n    trim_long_data=False, \n    pathname=curated_dir / \"0164cba5.wav\");\nsample_x = mono_to_color(sample_x)\nimage = Image.fromarray(sample_x)\nim_flip = ImageOps.flip(image)\n\nplt.figure(figsize=(4, 2))\nplt.title('Mel Spectrogram')\nplt.imshow(im_flip)\nplt.tick_params(labelleft=False, labelbottom=False)\nstrength_list = _get_strength(sample_x)\nplt.figure(figsize=(3, 1.3))\nplt.title('Strength of db')\nplt.plot(strength_list[0][0])\nplt.tick_params(labelleft=False, labelbottom=False)\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This sample has important information earlier, but there is a little information from the end of sound."},{"metadata":{},"cell_type":"markdown","source":"We prepare strength_list in advance and apply strength adaptive crop in FATTrainDataset.Here the code."},{"metadata":{"trusted":true},"cell_type":"code","source":"def _get_strength_list(x_train):\n    strength_list_train = []\n    for i in range(len(x_train)):\n        time_dim, base_dim = x_train[i].shape[1], x_train[i].shape[0]\n        x = x_train[i]\n        x_sum = []\n        x_max = 0\n        x_min = np.inf\n        for j in range(0,time_dim-base_dim + 1):\n            #x_temp = x[:,j:j + base_dim].sum()\n            x_temp = x[0:84,j:j + base_dim].sum()\n            x_sum.append(x_temp)\n            if x_temp > x_max:\n                x_max = x_temp\n            if x_temp < x_min:\n                x_min = x_temp\n        if (x_max == x_min):\n            height = x_max\n        else:\n            height = x_max - x_min\n        strength_list_train.append([x_sum, height, x_max, x_min, time_dim, base_dim])\n    return strength_list_train\n\n#strength_list_train = _get_strength_list(x_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class FATTrainDataset(Dataset):\n    def __init__(self, mels, labels, transforms, strength_list):\n        super().__init__()\n        self.mels = mels\n        self.labels = labels\n        self.transforms = transforms\n        self.strength_list = strength_list\n\n        \n    def __len__(self):\n        return len(self.mels)\n    \n    def __getitem__(self, idx):      \n        x = self.mels[idx]\n        x_sum = self.strength_list[idx][0]\n        height = self.strength_list[idx][1]\n        x_max = self.strength_list[idx][2]\n        x_min = self.strength_list[idx][3]\n        time_dim = self.strength_list[idx][4]\n        base_dim = self.strength_list[idx][5]\n        flag = True\n        while flag:\n            crop = random.randint(0, int(height * (time_dim- base_dim)))\n            # all 0\n            if height == 0:\n                crop_idx = 0\n                value = 0\n            else:\n                crop_idx = int(crop // height)\n                value = int(crop % height)                \n            if ((x_sum[crop_idx] - x_min) - value) >= 0:\n                flag = False\n        crop = crop_idx\n\n        image = Image.fromarray(x, mode='RGB') \n        image = image.crop([crop, 0, crop + base_dim, base_dim])\n        image = self.transforms(image).div_(255)\n        \n        label = self.labels[idx]\n        label = torch.from_numpy(label).float()\n        \n        return image, label","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model Structure"},{"metadata":{},"cell_type":"markdown","source":"### Custom CNN"},{"metadata":{},"cell_type":"markdown","source":"At first, we worked a lot on our custom CNN model. We started from improving [CVSSP Baseline](https://www.kaggle.com/c/freesound-audio-tagging-2019/discussion/89382#latest-534349).   \nWe changed kernel size, number of Convolution Blocks, **Pooling policy**, and the **structure of the fully connected layer**. We also added **spatial and channel attention**.  \nThe original CVSSP Baseline model structure is below."},{"metadata":{"trusted":true},"cell_type":"code","source":"class ConvBlock(nn.Module):\n    def __init__(self, in_channels, out_channels):\n        \n        super(ConvBlock, self).__init__()\n        \n        self.conv1 = nn.Conv2d(in_channels=in_channels, \n                              out_channels=out_channels,\n                              kernel_size=(3, 3), stride=(1, 1),\n                              padding=(1, 1), bias=False)\n                              \n        self.conv2 = nn.Conv2d(in_channels=out_channels, \n                              out_channels=out_channels,\n                              kernel_size=(3, 3), stride=(1, 1),\n                              padding=(1, 1), bias=False)\n                              \n        self.bn1 = nn.BatchNorm2d(out_channels)\n        self.bn2 = nn.BatchNorm2d(out_channels)\n        \n        self.init_weights()\n        \n    def init_weights(self):\n        \n        init_layer(self.conv1)\n        init_layer(self.conv2)\n        init_bn(self.bn1)\n        init_bn(self.bn2)\n        \n    def forward(self, input, pool_size=(2, 2), pool_type='avg'):\n        \n        x = input\n        x = F.relu_(self.bn1(self.conv1(x)))\n        x = F.relu_(self.bn2(self.conv2(x)))\n        if pool_type == 'max':\n            x = F.max_pool2d(x, kernel_size=pool_size)\n        elif pool_type == 'avg':\n            x = F.avg_pool2d(x, kernel_size=pool_size)\n        else:\n            raise Exception('Incorrect argument!')\n        \n        return x\n    \n    \nclass Cnn_9layers_AvgPooling(nn.Module):\n    \n    def __init__(self, classes_num):\n        super(Cnn_9layers_AvgPooling, self).__init__()\n\n        self.conv_block1 = ConvBlock(in_channels=1, out_channels=64)\n        self.conv_block2 = ConvBlock(in_channels=64, out_channels=128)\n        self.conv_block3 = ConvBlock(in_channels=128, out_channels=256)\n        self.conv_block4 = ConvBlock(in_channels=256, out_channels=512)\n\n        self.fc = nn.Linear(512, classes_num, bias=True)\n\n        self.init_weights()\n\n    def init_weights(self):\n\n        init_layer(self.fc)\n\n    def forward(self, input):\n        '''\n        Input: (batch_size, times_steps, freq_bins)'''\n        \n        x = input[:, None, :, :]\n        '''(batch_size, 1, times_steps, freq_bins)'''\n        \n        x = self.conv_block1(x, pool_size=(2, 2), pool_type='avg')\n        x = self.conv_block2(x, pool_size=(2, 2), pool_type='avg')\n        x = self.conv_block3(x, pool_size=(2, 2), pool_type='avg')\n        x = self.conv_block4(x, pool_size=(1, 1), pool_type='avg')\n        '''(batch_size, feature_maps, time_steps, freq_bins)'''\n        \n        x = torch.mean(x, dim=3)        # (batch_size, feature_maps, time_stpes)\n        (x, _) = torch.max(x, dim=2)    # (batch_size, feature_maps)\n        output = torch.sigmoid(self.fc(x))\n        \n        return output","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We altered the model structure like the model below."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Attention Layers\nclass SpatialAttention2d(nn.Module):\n    def __init__(self, channel):\n        super(SpatialAttention2d, self).__init__()\n        self.squeeze = nn.Conv2d(channel, 1, kernel_size=1, bias=False)\n        self.sigmoid = nn.Sigmoid()\n\n    def forward(self, x):\n        z = self.squeeze(x)\n        z = self.sigmoid(z)\n        return x * z\n\n\nclass GAB(nn.Module):\n    def __init__(self, input_dim, reduction=4):\n        super(GAB, self).__init__()\n        self.global_avgpool = nn.AdaptiveAvgPool2d(1)\n        self.conv1 = nn.Conv2d(input_dim, input_dim // reduction, kernel_size=1, stride=1)\n        self.conv2 = nn.Conv2d(input_dim // reduction, input_dim, kernel_size=1, stride=1)\n        self.relu = nn.ReLU(inplace=True)\n        self.sigmoid = nn.Sigmoid()\n\n    def forward(self, x):\n        z = self.global_avgpool(x)\n        z = self.relu(self.conv1(z))\n        z = self.sigmoid(self.conv2(z))\n        return x * z\n\n    \nclass SCse(nn.Module):\n    def __init__(self, dim):\n        super(SCse, self).__init__()\n        self.satt = SpatialAttention2d(dim)\n        self.catt = GAB(dim)\n\n    def forward(self, x):\n        return self.satt(x) + self.catt(x)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We borrowed nice implementation from https://github.com/ybabakhin/kaggle_salt_bes_phalanx.  \nWe'd like to appreciate to [@phalanx](https://www.kaggle.com/phalanx) for these implementations.\n\nWe added some modification in the ConvBlock and the model structure.  \nHere's our model."},{"metadata":{"trusted":true},"cell_type":"code","source":"class ConvBlock(nn.Module):\n    def __init__(self, in_channels, out_channels, reduction=16):\n        super(ConvBlock, self).__init__()\n        # changed kernel size\n        # we don't know why, but this is better than (3, 3) kernels.\n        self.conv1 = nn.Conv2d(in_channels=in_channels,\n                               out_channels=out_channels,\n                               kernel_size=(2, 2),\n                               stride=(1, 1),\n                               padding=(1, 1),\n                               bias=False)\n        self.conv2 = nn.Conv2d(in_channels=out_channels,\n                               out_channels=out_channels,\n                               kernel_size=(2, 2),\n                               stride=(1, 1),\n                               padding=(1, 1),\n                               bias=False)\n        self.bn1 = nn.BatchNorm2d(out_channels)\n        self.bn2 = nn.BatchNorm2d(out_channels)\n        \n        # Spatial and Channel Squeeze and Exitation\n        self.scse = SCse(out_channels)\n        \n        self.init_weights()\n        \n    def init_weights(self):\n        init_layer(self.conv1)\n        init_layer(self.conv2)\n        init_bn(self.bn1)\n        init_bn(self.bn2)\n        \n    def forward(self, inp, pool_size=(2, 2), pool_type=\"avg\"):\n        x = inp\n        x = F.relu_(self.bn1(self.conv1(x)))\n        x = F.relu_(self.scse(self.bn2(self.conv2(x))))\n        if pool_type == \"max\":\n            x = F.max_pool2d(x, kernel_size=pool_size)\n        elif pool_type == \"avg\":\n            x = F.avg_pool2d(x, kernel_size=pool_size)\n        # Added 'both' pool.\n        elif pool_type == \"both\":\n            x1 = F.max_pool2d(x, kernel_size=pool_size)\n            x2 = F.avg_pool2d(x, kernel_size=pool_size)\n            x = x1 + x2\n        else:\n            import pdb\n            pdb.set_trace()\n        return x\n    \n    \nclass ConvNet(nn.Module):\n    def __init__(self, n_classes=80):\n        super(ConvNet, self).__init__()\n        # 5 ConvBlocks\n        self.conv1 = ConvBlock(1, 32)\n        self.conv2 = ConvBlock(32, 64)\n        self.conv3 = ConvBlock(64, 128)\n        self.conv4 = ConvBlock(128, 256)\n        self.conv5 = ConvBlock(256, 512)\n        \n        self.bn1 = nn.BatchNorm1d((1 + 4 + 20) * 512)\n        self.drop1 = nn.Dropout(0.4)\n        self.fc1 = nn.Linear((1 + 4 + 20) * 512, 512)\n        self.prelu = nn.PReLU()\n        self.bn2 = nn.BatchNorm1d(512)\n        self.drop2 = nn.Dropout(0.2)\n        self.fc2 = nn.Linear(512, n_classes)\n        \n    def init_weight(self):\n        init_layer(self.fc1)\n        init_layer(self.fc2)\n        init_bn(self.bn1)\n        init_bn(self.bn2)\n    \n    def forward(self, x):\n        x = x.view(x.size(0), 1, x.size(1), x.size(2))\n        # Changed pooling policy.\n        x = self.conv1(x, pool_size=(1, 1), pool_type=\"both\")\n        x = self.conv2(x, pool_size=(4, 1), pool_type=\"both\")\n        x = self.conv3(x, pool_size=(1, 3), pool_type=\"both\")\n        x = self.conv4(x, pool_size=(4, 1), pool_type=\"both\")\n        x = self.conv5(x, pool_size=(1, 3), pool_type=\"both\")\n        \n        # Cutting the feature map to arbitrary size.\n        x1_max = F.max_pool2d(x, (5, 8))\n        x1_mean = F.avg_pool2d(x, (5, 8))\n        x1 = (x1_max + x1_mean).reshape(x.size(0), -1)\n        \n        x2_max = F.max_pool2d(x, (2, 4))\n        x2_mean = F.avg_pool2d(x, (2, 4))\n        x2 = (x2_max + x2_mean).reshape(x.size(0), -1)\n        \n        x = torch.mean(x, dim=3)\n        x, _ = torch.max(x, dim=2)\n        \n        x = torch.cat([x, x1, x2], dim=1)\n        x = self.drop1(self.bn1(x))\n        x = self.prelu(self.fc1(x))\n        x = self.drop2(self.bn2(x))\n        x = self.fc2(x)\n         \n        return x","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The difference from the original model is...\n\n#### kernel size\n\nWe don't know why, but the size (2, 2) was the best.\n\n#### The nuber of convolution blocks\n\nWe used 5 blocks to get 512 channels feature map. This is because we designed our model considering the meaning of each blocks (,although we're not sure whether it is working as we imaged originally).  \nWe found [@omalleyt](https://www.kaggle.com/omalleyt)'s 2nd place solution for Tensorflow Speech Recognition Competition very helpful.  \nhttps://www.kaggle.com/c/tensorflow-speech-recognition-challenge/forums/t/47715/2nd-place-solution?forumMessageId=270205#post270205\n\nThe *design philosophy* behind the placement of convolution blocks are as follows.\n\n  - First Convolution Block: channel 1 -> 32, no pooling.  \n    We put this block to collect primitive patterns of the log-melspectrogram.\n  - Second Convolution Block: channel 32 -> 64, (4, 1) pooling.  \n    Clearly, **spectrogram has different axis, and we need to treat them separately**. We took careful **not to aggregate the feature map along to the time axis and to the frequency axis simultaneously**. In this block, we'll get the primitive patterns connected to the patterns nearby in the time axis.\n  - Third Convolution Block: channel 64 -> 128, (1, 3) pooling.  \n    In this block, we'll get the primitive pattens connected in the frequency axis.\n  - Fourth Convolution Block: channel 128 -> 256, (4, 1) pooling.  \n    This block allows patterns to move in the time axis.\n  - Fifth Convolution Block: channel 256 -> 512, (1, 3) pooling.  \n    As explained in this [blog post](https://t.co/IgUKfFOYaa), moving the pattern in the frequency can be problematic. But that isn't the case when the degree of the translation is not so big. This block allows patterns to move in the frequency axis a little.\n    \n#### Cutting output feature map to arbitrary size\n\nWe found that the position in the feature map is important when treating spectrogram.  \nThe same pattern at the bottom of the image and at the top of the image are totally different. We should not allow patterns translate along the frequency axis. That is a little bit different from treating natural images.\n\nTo treat the position in the feature map, we used **[spatial pyramid pooling](https://arxiv.org/pdf/1406.4729.pdf)** to catch the local features.  \nWith this trick, CV lwlrap score improved **0.015 - 0.03**."},{"metadata":{},"cell_type":"markdown","source":"### Inception v3\n\nWe used two Inception v3 models. One is all the same as the original Inception v3 architecture, so we simply used the one in `torchvision.models` package. The other one was 1ch version of the original Inception v3, so we just simply altered the first layer.  \nThe score of the 1ch version was not as stable as 3ch version, but we found that model a little bit different from the original one.  "},{"metadata":{},"cell_type":"markdown","source":"## Augmentation in Batch"},{"metadata":{},"cell_type":"markdown","source":"### blend image augmentation\n\nWe used below augmentation strategy in batch\n\nhttps://pillow.readthedocs.io/en/stable/reference/Image.html"},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport pickle\nfrom PIL import Image\n\ndef array2img(array):    \n    image = Image.fromarray(array, mode='RGB')        \n    time_dim, base_dim = image.size\n    crop = random.randint(0, time_dim - base_dim)\n    image = image.crop([crop, 0, crop + base_dim, base_dim])\n    return image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with open(Path('../input/fat2019_prep_mels1') / 'mels_train_curated.pkl', 'rb') as curated:\n    x_train = pickle.load(curated)\n    \nprint(x_train[0].shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"im1 = array2img(x_train[0])\nim2 = array2img(x_train[1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(im1);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(im2);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lam = np.random.beta(1.0, 1.0)\n\nimg_blend = Image.blend(im1=im1, im2=im2, alpha=lam)\nplt.imshow(img_blend);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So, We got mixed Image data"},{"metadata":{},"cell_type":"markdown","source":"We define below function to apply to above strategy in our training"},{"metadata":{"trusted":true},"cell_type":"code","source":"def _get_blend_arr(x_trn, y_trn, lam = np.random.beta(1.0, 1.0)):\n    # get mix data\n    blend_arr_x = []\n    blend_arr_y = []\n    for i in range(1000):\n        r1, r2 = random.randint(0, len(x_trn)-1), random.randint(0, len(x_trn)-1)\n        \n        img0 = array2img(x_trn[r1])\n        img1 = array2img(x_trn[r2])\n        img_blend = Image.blend(im1=img0, im2=img1, alpha=lam)\n\n        y_blend = y_trn[r1]*(1-lam) + y_trn[r2]*lam\n        img_blend_arr = np.asarray(img_blend)\n        blend_arr_x.append(img_blend_arr)\n        blend_arr_y.append(y_blend)    \n    idx_ex = np.arange(len(blend_arr_x))\n    return blend_arr_x, blend_arr_y, idx_ex\n\n# blend_arr_x, blend_arr_y, idx_ex = _get_blend_arr(x_trn, y_trn, lam = np.random.beta(1.0, 1.0))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Training Strategy"},{"metadata":{},"cell_type":"markdown","source":"This is our pipline."},{"metadata":{"trusted":true},"cell_type":"code","source":"from IPython.display import Image\nImage(\"../input/freesound-solution-pic/Freesound_solution_pic1.png\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Pretrain with Noisy Data"},{"metadata":{},"cell_type":"markdown","source":"We used noisy data to 'pre-train' our model.  \nWe had two policy to train and validate with noisy data.  \nYou can see how we did this in [this kernel](https://www.kaggle.com/hidehisaarai1213/convnet-noisy-all-pitch-pretraining).\n\n1. Train with all the noisy data, validate with curated data.\n2. Split noisy data with `train_test_split`. Both train and validate are done with only noisy data.\n\nWe tried both policy, and found it surprising that policy 2 was slightly better than policy 1. We believe it was just a seed magic, but the model trained with policy12 were at least as better as the model trained with policy1.  \n\nWe used [@mhiro2's way](https://www.kaggle.com/mhiro2/simple-2d-cnn-classifier-with-pytorch) to train, which means \n\n- change learning rate with CosineAnnealingLR cyclickally in the range between 1e-3 and 1e-5. The cycle was 10 epochs.\n- use Adam optimizer.\n- optimize BCEWithLogitLoss\n\nWe also used augmented data in this stage.  \nWe created a `pitch` augmented dataset from train_noisy dataset and used it only for training.  \nWe only used train_noisy dataset for validation.\n\nIn the pre-training of Inception models, we applied RandomResizedCrop to the data, and it required us to train longer, so we introduced stage 2 pre-training.  \nIn the stage2 pre-training, we didn't changed any settings except for the initial weight of the network.\n\nWith pre-training, local lwlrap jumped from **0.83 to 0.88** with Custom CNN, and **0.86 to 0.87** with Inception v3.  \nUnfortunately, we don't know how much the LB lwlrap score improved with Custom CNN, but the LB lwlrap score of Inception v3 improved **0.01** with this trick."},{"metadata":{},"cell_type":"markdown","source":"### Training with Curated Data"},{"metadata":{},"cell_type":"markdown","source":"We used curated data to 'finetune' the models, which were 'pre-trained' with noisy data.  \nWe used augmented dataset with original train_curated dataset, which made the whole training dataset 8 times larger than the original one.  \n\nWe used normal `KFold` to get cross validation, and split the dataset into 5 subsets.  \nSince the size of each augmented dataset is all the same as the original one, which means 4970, and the order of them are all the same, it was easy to avoid data leak.  \nWe simply used the same indices for splitting original dataset and augmented datasets.  \nOne thing to note is that we didn't used augmented dataset for validation. The reason is simple, the size was to large.\n\n[Here](https://www.kaggle.com/hidehisaarai1213/freesound-dameoshi-aug-fold5)'s one of the kernel to conduct learning with curated data."},{"metadata":{},"cell_type":"markdown","source":"### TTA for Validation"},{"metadata":{},"cell_type":"markdown","source":"It is very effective to use tta because we use random crop. Especially, since we use random resized crop, tta is very important. When RandomResizedCrop is used, val score fluctuate,so if val tta is not used, an appropriate epoch can not be selected. So, we used tta for validation to ensure that validation can be properly evaluated."},{"metadata":{"trusted":true},"cell_type":"code","source":"class FATValidDataset(Dataset):\n    def __init__(self, fnames, mels, labels, transforms, strength_list, tta=5):\n        super().__init__()\n        self.fnames = fnames\n        self.mels = mels\n        self.labels = labels\n        self.transforms = transforms\n        self.strength_list = strength_list\n        self.tta = tta\n\n    def __len__(self):\n        return len(self.fnames) * self.tta\n    \n    def __getitem__(self, idx):       \n        new_idx = idx % len(self.fnames)      \n        x = self.mels[new_idx]\n        x_sum = self.strength_list[new_idx][0]\n        height = self.strength_list[new_idx][1]\n        x_max = self.strength_list[new_idx][2]\n        x_min = self.strength_list[new_idx][3]\n        time_dim = self.strength_list[new_idx][4]\n        base_dim = self.strength_list[new_idx][5]\n        flag = True\n        while flag:\n            crop = random.randint(0, int(height * (time_dim- base_dim)))\n            if height == 0:\n                crop_idx = 0\n                value = 0\n            else:\n                crop_idx = int(crop // height)\n                value = int(crop % height)                \n            if ((x_sum[crop_idx] - x_min) - value) >= 0:\n                flag = False\n        crop = crop_idx\n        image = Image.fromarray(x, mode='RGB') \n        image = image.crop([crop, 0, crop + base_dim, base_dim])\n        image = self.transforms(image).div_(255)\n        label = self.labels[new_idx]\n        label = torch.from_numpy(label).float()\n        \n        fname = self.fnames[new_idx]\n        \n        return image, label, fname\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Ensemble"},{"metadata":{},"cell_type":"markdown","source":"We end up with simply blending the output with the same weight, which means `(CustomCNN + Inception v3 3ch + Inception v3 1ch) / 3` but we tried a lot to use many models with comparatively low scores around `0.700 - 0.710`.  \nAlso we faced the problem of **Blend Many Models** vs **Increasing the number of TTA**.  \n\nWith careful experiment, we found that Inception model was comparatively unstable and requires comparatively large number of TTA, which was around 20 - 25 times.  \nFinaly, we decided to use 20 TTA for each model, and blend 5 fold * 3models = 15model with equally weighted manner."},{"metadata":{},"cell_type":"markdown","source":"## What didn't work"},{"metadata":{},"cell_type":"markdown","source":"* Augmentation\n  - CutMix https://github.com/knjcode/pytorch-finetuner\n  - GaussianNoise https://imgaug.readthedocs.io/en/latest/source/augmenters.html#additivegaussiannoise\n  - SpecAugment http://ai.googleblog.com/2019/04/specaugment-new-data-augmentation.html\n  \nAll the above had no impact on CV.\n\n* Validation\n  - Stratified K fold for Multilabel\n\n* Models\n  - DenseNet\n  - XceptionNet\n  - SE-Inception\n  - ResNet\n  - ResNeXt\n  - SE-ResNeXt\n  - MobileNet\n  - ConvLSTM\n  - CapsuleNet\n  \nWe've tried many models, but we didn't find them better than Custom CNN or Inception v3 model. Xception model may have been better, but we couldn't train it properly as we didn't have enough time to try it in the last period of the competition. \n\n\n* Specific CNN structures\n  - [CoordConv](https://arxiv.org/abs/1807.03247)\n  - [OctaveConv](https://arxiv.org/pdf/1904.05049v2.pdf)\n  \nWe thought that the position in the image was very important in this competition, and tried CoordConv because it says it can handle the position, but we found it not useful.  \n\nWe also tried OctaveConv, but couldn't find it useful for this competition.\n\n* Ensemble\n\nmany models (Inception, CustomCNN v1, CustomCNN v2, Se-ResNeXt, DenseNet)"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}