{"cells":[{"metadata":{},"cell_type":"markdown","source":"Hi Kagglers. Days back at the beginning of this comp', I've released a [dummy  kernel](https://www.kaggle.com/kneroma/inference-resnest-rfcx-audio-detection) that performs poorly because of the simple training pipeline and the poor data cropping strategy. I've updated a lot of things and I'm publishing a novel version which is more robust and has a clever training pipeline.\n\nThe goal of this work is to show that we can get a decent score with just a single **ResneSt50** architecture with no TTA of any fancy inference augmentations. As a great fan of the **open-source** philosophy, I will be releasing as much as I can, including my datasets, weights and tips. Unfortunately, my training is currently very dirty and will be hard to release. But, I will be hopefully cleaning and releasing it soon or later."},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training"},{"metadata":{},"cell_type":"markdown","source":"* The training is based on *intelligent* crops of **10 s**, with just a training set less than **500 Mo** !!! \n* I used a stratified KFold, with n_splits = 5\n* I found the learning rate scheduler very important\n* Higher learning rates seem to give me better results\n* For the cross validation metric, I was sticked on F1 score which seems to be more robust than the comps' metric"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Inference"},{"metadata":{},"cell_type":"markdown","source":"* The inference is based on these [resnest50 weights](https://www.kaggle.com/kneroma/kkiller-rfcx-species-detection-public-checkpoints). Please, don't forget upvoting the dataset to make it more visible for others\n* The inference pipeline is optimized as much as I can in order to reduce execution time\n* I'm using the pytorch native multi-worker data loading framework\n* You can try increasing the **DURATION** hyperparam, but it can lead to higher execution time\n* Reducing the **STRIDE** hyperparam may give better result, but  it can lead to higher execution time"},{"metadata":{},"cell_type":"markdown","source":"# Pre-computed MFCCs\n> In order to make inference crazingly faster (**< 10 mins on GPU**), I've created [this precomputed MFCC  dataset](https://www.kaggle.com/kneroma/kkiller-rfcx-test-mfcc-1-0400). Don't mind using it in your pipelines :) "},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<h2><font color=\"blue\">If you find this work useful, please don't forget upvoting :)</font></h2>"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"id":"ikMF-nQSdTbF","outputId":"041c6767-86ab-46e4-9b22-a4840121ecca","trusted":true},"cell_type":"code","source":"!nvidia-smi","execution_count":null,"outputs":[]},{"metadata":{"id":"y81uWCF8ngSs","trusted":true},"cell_type":"code","source":"!pip install  resnest > /dev/null","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# for TPU\n!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py > /dev/null\n!python pytorch-xla-env-setup.py --apt-packages libomp5 libopenblas-dev > /dev/null","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch_xla\nimport torch_xla.core.xla_model as xm","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","execution":{"iopub.execute_input":"2020-12-27T08:05:00.353947Z","iopub.status.busy":"2020-12-27T08:05:00.351498Z","iopub.status.idle":"2020-12-27T08:05:03.624446Z","shell.execute_reply":"2020-12-27T08:05:03.62339Z"},"papermill":{"duration":3.295502,"end_time":"2020-12-27T08:05:03.624573","exception":false,"start_time":"2020-12-27T08:05:00.329071","status":"completed"},"tags":[],"id":"kSCcqxf0c_O7","trusted":true},"cell_type":"code","source":"import numpy as np\nimport librosa as lb\nimport soundfile as sf\nimport pandas as pd\nfrom pathlib import Path\n\nimport torch\nfrom  torch.utils.data import Dataset, DataLoader\n\nfrom tqdm.notebook import tqdm\n\nfrom resnest.torch import resnest50\n\nimport time","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Configs"},{"metadata":{"execution":{"iopub.execute_input":"2020-12-27T08:05:03.664229Z","iopub.status.busy":"2020-12-27T08:05:03.663369Z","iopub.status.idle":"2020-12-27T08:05:03.665921Z","shell.execute_reply":"2020-12-27T08:05:03.665517Z"},"papermill":{"duration":0.024061,"end_time":"2020-12-27T08:05:03.666018","exception":false,"start_time":"2020-12-27T08:05:03.641957","status":"completed"},"tags":[],"id":"rJhYZVIDc_O9","trusted":true},"cell_type":"code","source":"# Data Loader\nNUM_CLASSES = 24\nSR = 32_000\nDURATION =  10\nSTRIDE = 5\n\n\n# Neural Net\nTEST_BATCH_SIZE = 30\nTEST_NUM_WORKERS = 2\n\nUSE_PRE_COMPUTED_MFCC = True\n\n# DEVICE = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nDEVICE = xm.xla_device() # TPU\ntorch.set_default_tensor_type('torch.FloatTensor')\n\nTEST_AUDIO_ROOT = Path(\"../input/rfcx-species-audio-detection/test\")\n\nTEST_MFCC_ROOT = \"../input/kkiller-rfcx-test-mfcc-1-0400/test_mfcc_d10_s10_sr32000\"","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.016845,"end_time":"2020-12-27T08:05:03.699716","exception":false,"start_time":"2020-12-27T08:05:03.682871","status":"completed"},"tags":[],"id":"pbqOPUExc_O-","trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data"},{"metadata":{"execution":{"iopub.execute_input":"2020-12-27T08:05:03.740251Z","iopub.status.busy":"2020-12-27T08:05:03.73975Z","iopub.status.idle":"2020-12-27T08:05:03.743651Z","shell.execute_reply":"2020-12-27T08:05:03.743234Z"},"papermill":{"duration":0.02684,"end_time":"2020-12-27T08:05:03.743735","exception":false,"start_time":"2020-12-27T08:05:03.716895","status":"completed"},"tags":[],"id":"4fwNhdbJc_O-","trusted":true},"cell_type":"code","source":"class MelSpecComputer:\n    def __init__(self, sr, n_mels, fmin, fmax):\n        self.sr = sr\n        self.n_mels = n_mels\n        self.fmin = fmin\n        self.fmax = fmax\n\n    def __call__(self, y):\n\n        melspec = lb.feature.melspectrogram(\n            y, sr=self.sr, n_mels=self.n_mels, fmin=self.fmin, fmax=self.fmax,\n        )\n\n        melspec = lb.power_to_db(melspec).astype(np.float32)\n        return melspec","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-12-27T08:05:03.791803Z","iopub.status.busy":"2020-12-27T08:05:03.790616Z","iopub.status.idle":"2020-12-27T08:05:03.793062Z","shell.execute_reply":"2020-12-27T08:05:03.793525Z"},"papermill":{"duration":0.033047,"end_time":"2020-12-27T08:05:03.79362","exception":false,"start_time":"2020-12-27T08:05:03.760573","status":"completed"},"tags":[],"id":"Nk0haSTIc_O_","trusted":true},"cell_type":"code","source":"def mono_to_color(X, eps=1e-6, mean=None, std=None):\n    X = np.stack([X, X, X], axis=-1)\n\n    # Standardize\n    mean = mean or X.mean()\n    std = std or X.std()\n    X = (X - mean) / (std + eps)\n\n    # Normalize to [0, 255]\n    _min, _max = X.min(), X.max()\n\n    if (_max - _min) > eps:\n        V = np.clip(X, _min, _max)\n        V = 255 * (V - _min) / (_max - _min)\n        V = V.astype(np.uint8)\n    else:\n        V = np.zeros_like(X, dtype=np.uint8)\n\n    return V\n\n\ndef normalize(image, mean=None, std=None):\n    image = image / 255.0\n    if mean is not None and std is not None:\n        image = (image - mean) / std\n    return np.moveaxis(image, 2, 0).astype(np.float32)\n\n\ndef crop_or_pad(y, length, sr, is_train=True):\n    if len(y) < length:\n        y = np.concatenate([y, np.zeros(length - len(y))])\n    elif len(y) > length:\n        if not is_train:\n            start = 0\n        else:\n            start = np.random.randint(len(y) - length)\n\n        y = y[start:start + length]\n\n    y = y.astype(np.float32, copy=False)\n\n    return y","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-12-27T08:05:03.882841Z","iopub.status.busy":"2020-12-27T08:05:03.879358Z","iopub.status.idle":"2020-12-27T08:05:03.884996Z","shell.execute_reply":"2020-12-27T08:05:03.885392Z"},"papermill":{"duration":0.035536,"end_time":"2020-12-27T08:05:03.88549","exception":false,"start_time":"2020-12-27T08:05:03.849954","status":"completed"},"tags":[],"id":"Q2MieafPc_PB","trusted":true},"cell_type":"code","source":"class RFCXDataset(Dataset):\n\n    def __init__(self, data, sr, n_mels=128, fmin=0, fmax=None, num_classes=NUM_CLASSES, duration=DURATION, stride=STRIDE, root=None):\n\n        self.data = data\n        \n        self.sr = sr\n        self.n_mels = n_mels\n        self.fmin = fmin\n        self.fmax = fmax or self.sr//2\n\n\n        self.num_classes = num_classes\n        self.duration = duration\n        self.stride = stride\n        self.audio_length = self.duration*self.sr\n        \n        self.root =  root or TEST_AUDIO_ROOT\n\n        self.mel_spec_computer = MelSpecComputer(sr=self.sr, n_mels=self.n_mels, fmin=self.fmin, fmax=self.fmax)\n        \n        self.res_type = \"kaiser_best\"\n\n\n    def __len__(self):\n        return len(self.data)\n    \n    def load(self, record):\n        y, _ = lb.load(self.root.joinpath(record).with_suffix(\".flac\").as_posix(), sr=self.sr, res_type=self.res_type)\n        return y\n    \n    def load2(self, record):\n        y, orig_sr = sf.read(self.root.joinpath(record).with_suffix(\".flac\").as_posix())\n        y = lb.resample(y, orig_sr=orig_sr, target_sr=self.sr, res_type=self.res_type)\n        return y\n    \n    def read_index(self, idx):\n        d = self.data.iloc[idx]\n        record = d[\"recording_id\"]\n        \n        y = self.load2(record)\n        \n        window = self.duration*self.sr\n        stride = self.stride*self.sr\n            \n        y = np.stack([y[i:i+window] for i in range(0, 60*self.sr+stride-window, stride)])\n\n        return y\n            \n    def process(self, y):\n        melspec = self.mel_spec_computer(y) \n        image = mono_to_color(melspec)\n        image = normalize(image, mean=None, std=None)\n        return image\n\n    def __getitem__(self, idx):\n\n        y = self.read_index(idx)\n        \n        image = np.stack([self.process(_y) for _y in y])\n\n        return image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class SimpleRFCXDataset(Dataset):\n    def __init__(self, data):\n        self.data = data\n        \n    def __getitem__(self, idx):\n        row = self.data.iloc[idx]\n        record_id_path = Path(row.mfcc_root).joinpath(row.recording_id).with_suffix(\".npy\")\n        image = np.load(record_id_path)\n        return image\n    \n    def __len__(self):\n        return len(self.data)","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-12-27T08:05:03.964403Z","iopub.status.busy":"2020-12-27T08:05:03.963877Z","iopub.status.idle":"2020-12-27T08:05:18.438105Z","shell.execute_reply":"2020-12-27T08:05:18.437604Z"},"papermill":{"duration":14.495358,"end_time":"2020-12-27T08:05:18.438221","exception":false,"start_time":"2020-12-27T08:05:03.942863","status":"completed"},"tags":[],"id":"rVwlOrbxc_PD","outputId":"9b53ec99-634a-4b30-f9b5-75036184482b","trusted":true},"cell_type":"code","source":"%%time\n\ndata = pd.DataFrame({\n    \"recording_id\": [path.stem for path in Path(TEST_AUDIO_ROOT).glob(\"*.flac\")],\n})\ndata[\"mfcc_root\"] = TEST_MFCC_ROOT\nprint(data.shape)\ndata.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"TEST_MFCC_ROOTs = [\n    \"../input/kkiller-rfcx-test-mfcc-0000-0400/test_mfcc_d10_s2_sr32000_0000_0400\",\n    \"../input/kkiller-rfcx-test-mfcc-0400-0800/test_mfcc_d10_s2_sr32000_0400_0800\",\n    \"../input/kkiller-rfcx-test-mfcc-0800-1200/test_mfcc_d10_s2_sr32000_0800_1200\",\n    \"../input/kkiller-rfcx-test-mfcc-1200-1600/test_mfcc_d10_s2_sr32000_1200_1600\",\n    \"../input/kkiller-rfcx-test-mfcc-1600-2000/test_mfcc_d10_s2_sr32000_1600_2000\",\n]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mfccs = []\nfor mfcc_root in TEST_MFCC_ROOTs:\n    mfccs += [(mfcc.stem, mfcc.parent.as_posix()) for mfcc in Path(mfcc_root).glob(\"*.npy\")]\nmfccs = pd.DataFrame(mfccs, columns = [\"recording_id\", 'mfcc_root'])\n\ndata = data[[\"recording_id\"]].merge(mfccs, on=\"recording_id\")\nprint(data.shape)\ndata.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-12-27T08:05:18.78594Z","iopub.status.busy":"2020-12-27T08:05:18.785444Z","iopub.status.idle":"2020-12-27T08:05:18.789579Z","shell.execute_reply":"2020-12-27T08:05:18.788896Z"},"papermill":{"duration":0.026055,"end_time":"2020-12-27T08:05:18.789711","exception":false,"start_time":"2020-12-27T08:05:18.763656","status":"completed"},"tags":[],"id":"DajVt4q7c_PF","trusted":true},"cell_type":"code","source":"ds = RFCXDataset(data=data, sr=SR)","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-12-27T08:05:18.831385Z","iopub.status.busy":"2020-12-27T08:05:18.830615Z","iopub.status.idle":"2020-12-27T08:05:22.359206Z","shell.execute_reply":"2020-12-27T08:05:22.358244Z"},"papermill":{"duration":3.551127,"end_time":"2020-12-27T08:05:22.359324","exception":false,"start_time":"2020-12-27T08:05:18.808197","status":"completed"},"tags":[],"id":"a4z6e5v8c_PF","outputId":"6bb5a3ef-c74e-4ded-d9f9-a861a2c0ae04","trusted":true},"cell_type":"code","source":"%%time\n\nx = ds[1]\nprint(x.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.019612,"end_time":"2020-12-27T08:05:22.444719","exception":false,"start_time":"2020-12-27T08:05:22.425107","status":"completed"},"tags":[],"id":"BNsivZZZc_PG"},"cell_type":"markdown","source":"# Inference"},{"metadata":{"execution":{"iopub.execute_input":"2020-12-27T08:05:22.574655Z","iopub.status.busy":"2020-12-27T08:05:22.573929Z","iopub.status.idle":"2020-12-27T08:05:22.576796Z","shell.execute_reply":"2020-12-27T08:05:22.57628Z"},"papermill":{"duration":0.027254,"end_time":"2020-12-27T08:05:22.576874","exception":false,"start_time":"2020-12-27T08:05:22.54962","status":"completed"},"tags":[],"id":"AzlOErOmc_PH","trusted":true},"cell_type":"code","source":"test_data = SimpleRFCXDataset(data) if (USE_PRE_COMPUTED_MFCC and TEST_MFCC_ROOT) else RFCXDataset(data=data, sr=SR)\ntest_loader = DataLoader(test_data, batch_size=TEST_BATCH_SIZE, num_workers=TEST_NUM_WORKERS)","execution_count":null,"outputs":[]},{"metadata":{"id":"vfU4QJxJfwsD","trusted":true},"cell_type":"code","source":"def load_net(checkpoint_path):\n    net = resnest50(pretrained=True)#.to(DEVICE)\n    n_features = net.fc.in_features\n    net.fc = torch.nn.Linear(n_features, NUM_CLASSES)\n    dummy_device = torch.device(\"cpu\")\n    net.load_state_dict(torch.load(checkpoint_path, map_location=dummy_device))\n    net = net.to(DEVICE)\n    net = net.eval()\n    return net","execution_count":null,"outputs":[]},{"metadata":{"id":"qoALJpJFgmAH","trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"id":"-QrZqdBsf-0n","trusted":true},"cell_type":"code","source":"checkpoint_paths = [\n    \"../input/kkiller-rfcx-species-detection-public-checkpoints/rfcx_resnest50/rfcx_resnest_50_fold0.pth\",\n    \"../input/kkiller-rfcx-species-detection-public-checkpoints/rfcx_resnest50/rfcx_resnest_50_fold1.pth\",\n    \"../input/kkiller-rfcx-species-detection-public-checkpoints/rfcx_resnest50/rfcx_resnest_50_fold2.pth\",\n    \"../input/kkiller-rfcx-species-detection-public-checkpoints/rfcx_resnest50/rfcx_resnest_50_fold3.pth\",\n    \"../input/kkiller-rfcx-species-detection-public-checkpoints/rfcx_resnest50/rfcx_resnest_50_fold4.pth\",\n]\n\nnets = [\n        load_net(checkpoint_path) for checkpoint_path in checkpoint_paths\n]","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.02159,"end_time":"2020-12-27T08:05:37.132215","exception":false,"start_time":"2020-12-27T08:05:37.110625","status":"completed"},"tags":[],"id":"ayVLRTzLc_PI","trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-12-27T08:05:37.184124Z","iopub.status.busy":"2020-12-27T08:05:37.183132Z","iopub.status.idle":"2020-12-27T09:22:14.433199Z","shell.execute_reply":"2020-12-27T09:22:14.432749Z"},"papermill":{"duration":4597.280114,"end_time":"2020-12-27T09:22:14.433289","exception":false,"start_time":"2020-12-27T08:05:37.153175","status":"completed"},"tags":[],"id":"QzIEBYcEc_PI","outputId":"ca43738a-138c-4fc6-b440-738441e13806","trusted":true},"cell_type":"code","source":"preds = []\n# net.eval()\nwith torch.no_grad():\n    for xb in  tqdm(test_loader):\n        bsize, nframes = xb.shape[:2]\n        xb = xb.to(DEVICE).view(bsize*nframes, *xb.shape[2:])\n\n        pred = 0.\n        for net in nets:\n            o = net(xb)\n            o = torch.sigmoid(o)\n            o = o.view(bsize, nframes, *o.shape[1:]).max(1).values\n            o = o.detach().cpu().numpy()\n\n            pred += o\n        \n        pred /= len(nets)\n        \n        preds.append(pred)\npreds = np.vstack(preds)\npreds.shape","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-12-27T09:22:14.607435Z","iopub.status.busy":"2020-12-27T09:22:14.597569Z","iopub.status.idle":"2020-12-27T09:22:14.613147Z","shell.execute_reply":"2020-12-27T09:22:14.612704Z"},"papermill":{"duration":0.062126,"end_time":"2020-12-27T09:22:14.613229","exception":false,"start_time":"2020-12-27T09:22:14.551103","status":"completed"},"tags":[],"id":"Mv3loFuuc_PJ","outputId":"144fe7e4-7900-463d-8404-47537153ede8","trusted":true},"cell_type":"code","source":"sub = pd.DataFrame(preds, columns=[f\"s{i}\" for i in range(24)])\nsub[\"recording_id\"] = data[\"recording_id\"].values[:len(sub)]\nsub = sub[[\"recording_id\"] + [f\"s{i}\" for i in range(24)]]\nprint(sub.shape)\nsub.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"id":"BsyQMZhNlgEB","trusted":true},"cell_type":"code","source":"sub.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}