{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Objective: \n \n

Audio competitions are a joy to work with, as there are a plenty of opportunities to learn from. This notebook is a overview on one of the state of art models for audio representation which learns audio representation as part of its pretraining process and the learnt representation can be used on the downstream tasks similar to language models in NLP world

"},{"metadata":{},"cell_type":"markdown","source":"Note:\n

We will start with few basic things about audio and gradually move on to the recent advancements in audio representations.

"},{"metadata":{},"cell_type":"markdown","source":"Buckle up
"},{"metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"!wget https://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_large.pt\n!pip install soundfile \n\n!pip -q install timm\n!pip -q install torchlibrosa\n!pip -q install audiomentations\n!pip install -q transformers==4.3.0\n\n!pip uninstall -y typing\n!pip install soundfile git+git://github.com/pytorch/fairseq.git@b8ea8a9b72c82192da07e3377adf4ebbde16716d\n ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os, glob, random\nimport numpy as np, pandas as pd\nfrom sklearn.model_selection import StratifiedKFold\nimport random, glob\nimport soundfile as sf\n\nimport torch\nfrom torch.utils.data import Dataset\nfrom albumentations.pytorch.functional import img_to_tensor\n\nfrom functools import partial\n\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.nn.modules.dropout import Dropout\nfrom torch.nn.modules.linear import Linear\nfrom torch.nn.modules.pooling import AdaptiveAvgPool2d, AdaptiveMaxPool2d\n\nimport timm\nfrom timm.models.efficientnet import tf_efficientnet_b4_ns, tf_efficientnet_b3_ns, \\\n tf_efficientnet_b5_ns, tf_efficientnet_b2_ns, tf_efficientnet_b6_ns, tf_efficientnet_b7_ns, tf_efficientnet_b0_ns\n\nfrom torchlibrosa.stft import Spectrogram, LogmelFilterBank\nfrom torchlibrosa.augmentation import SpecAugmentation\n\nfrom torch.nn import BCEWithLogitsLoss, CrossEntropyLoss\nimport audiomentations as A\nfrom sklearn import metrics\nfrom sklearn.metrics import log_loss\nimport wandb\nfrom transformers import get_linear_schedule_with_warmup\nimport time\nimport fairseq\n\nimport numpy as np\nfrom functools import partial\n\nimport sys\nsys.path.append('/kaggle/input/pytorch-utils/')\n\nfrom pytorch_utils import do_mixup, interpolate, pad_framewise_output\nfrom torch.nn import BCEWithLogitsLoss, CrossEntropyLoss\n\nfrom torch.utils.data import Dataset\nfrom albumentations.pytorch.functional import img_to_tensor\nfrom tqdm import tqdm\nimport librosa\nimport librosa.display\nimport matplotlib.pyplot as plt\nfrom transformers import Wav2Vec2Tokenizer,Wav2Vec2Model\n\n\n#needs to be a secret one\nos.environ[\"WANDB_API_KEY\"]='b8a047fb4d3f65c5f4616be42830b539e03c6a42'\n\n#wandb.login()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Table of Contents\n\n- 1. Audio representations\n - 1.1 Fourier transform\n - 1.2 Spectrograms\n - 1.3 Mel spectrograms\n\n- 2. Wav2vec overview\n - 2.1 Architecture\n - 2.2 Pipeline view\n \n- 3. Wav2vec family\n\n- 4. Modeling based on wav2vec representation\n - 4.1 Preparing dataset\n - 4.2 Dataset class\n - 4.3 Model definition\n - 4.4 Utils\n - 4.5 Engine\n - 4.6 Fold run\n \n- 5. Acknowledgements"},{"metadata":{},"cell_type":"markdown","source":"# 1. Audio representations"},{"metadata":{},"cell_type":"markdown","source":"

A signal is a variation in a certain quantity over time. For audio, the quantity that varies is air pressure. How do we capture this information digitally? We can take samples of the air pressure over time. The rate at which we sample the data can vary, but is most commonly 44.1kHz, or 44,100 samples per second. What we have captured is a waveform for the signal, and this can be interpreted, modified, and analyzed with computer software.

"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_file='../input/rfcx-species-audio-detection/train/00204008d.flac'\ny, sr = librosa.load(test_file)\nplt.plot(y);\nplt.title('Signal');\nplt.xlabel('Time (samples)');\nplt.ylabel('Amplitude');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 1.1 Fourier transform"},{"metadata":{},"cell_type":"markdown","source":"

The Fourier transform is a mathematical formula that allows us to decompose a signal into it’s individual frequencies and the frequency’s amplitude. In other words, it converts the signal from the time domain into the frequency domain. The result is called a spectrum.\n\nThe fast Fourier transform (FFT) is an algorithm that can efficiently compute the Fourier transform. It is widely used in signal processing

"},{"metadata":{},"cell_type":"markdown","source":"\n
Source: Google
"},{"metadata":{"trusted":true},"cell_type":"code","source":"n_fft = 2048\nft = np.abs(librosa.stft(y[:n_fft], hop_length = n_fft+1))\nplt.plot(ft);\nplt.title('Spectrum');\nplt.xlabel('Frequency Bin');\nplt.ylabel('Amplitude');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 1.2 Spectrograms"},{"metadata":{},"cell_type":"markdown","source":"The fast Fourier transform is a powerful tool that allows us to analyze the frequency content of a signal, but what if our signal’s frequency content varies over time?\n\nWhen FFT is computed on overlapping windowed segments of the signal, and we get what is called the spectrogram"},{"metadata":{},"cell_type":"markdown","source":"\n
Source: Google
"},{"metadata":{"trusted":true},"cell_type":"code","source":"spec = np.abs(librosa.stft(y, hop_length=512))\nspec = librosa.amplitude_to_db(spec, ref=np.max)\nlibrosa.display.specshow(spec, sr=sr, x_axis='time', y_axis='log');\nplt.colorbar(format='%+2.0f dB');\nplt.title('Spectrogram');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 1.3 Mel spectrograms"},{"metadata":{},"cell_type":"markdown","source":"

Studies have shown that humans do not perceive frequencies on a linear scale. We are better at detecting differences in lower frequencies than higher frequencies. For example, we can easily tell the difference between 500 and 1000 Hz, but we will hardly be able to tell a difference between 10,000 and 10,500 Hz, even though the distance between the two pairs are the same.

\n\n

A mel spectrogram is a spectrogram where the frequencies are converted to the mel scale.

"},{"metadata":{"trusted":true},"cell_type":"code","source":"mel_spect = librosa.feature.melspectrogram(y=y, sr=sr, n_fft=2048, hop_length=1024)\nmel_spect = librosa.power_to_db(mel_spect, ref=np.max)\nlibrosa.display.specshow(mel_spect, y_axis='mel', fmax=8000, x_axis='time');\nplt.title('Mel Spectrogram');\nplt.colorbar(format='%+2.0f dB');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

Now that we know the basics of traditional signal processing, we can get into the latest advancement in ASR models, one among them is wav2vec released by facebook AI

"},{"metadata":{},"cell_type":"markdown","source":"## 2. Wav2vec overview"},{"metadata":{},"cell_type":"markdown","source":"

Wav2vec uses semi-supervised learning to learn vector representations for preprocessed sound frames. This is similar to what word2vec does to learn word embeddings a text corpus. In the case of wav2vec it samples random parts of the sound file and learns to predict if a given part is in the near future from a current offset position. This is somewhat similar to the masked word task used to train transformers such as BERT.

"},{"metadata":{},"cell_type":"markdown","source":"

The algorithm is trained on unlabeled data since it uses the temporal structure of the data to produce labels and it uses random sampling to produce contrasting negative examples. It is a binary classification task (is the proposed processed sound frame in the near future of the current offset or not) and it uses contrastive loss as its objective. In training for this binary classification task, it learns vector representations of sound frames (one 512 dim vector for each 10ms of sound).

\n "},{"metadata":{},"cell_type":"markdown","source":"**These vector representations are useful features because they concentrate information relevant to predicting speech. These vectors can then be used instead of spectrogram vectors**"},{"metadata":{},"cell_type":"markdown","source":"\n
Source: Google
\n"},{"metadata":{},"cell_type":"markdown","source":"## 2.2 Pipeline overview for general wav2vec model"},{"metadata":{},"cell_type":"markdown","source":"* Pretraining\n* Acoustic model\n* Decoding\n\n

Wav2vec(after self training stage) is used as an input to an acoustic model. The vector supposedly carries more representation information than other types of features. It can be used as an input in ASR model.

"},{"metadata":{},"cell_type":"markdown","source":"## 3. Wav2vec family"},{"metadata":{},"cell_type":"markdown","source":"There are 4 core papers of wav2vec series, all of them coming from Facebook AI. \n\nAll these papers are building blocks of what could be a great innovation in speech recognition but also a lot of other downstream tasks related to speech:\n\n* Wav2vec paper\n* VQ - wav2vec\n* Wav2vec2.0 paper\n* Self-training and Pre-training are Complementary for Speech Recognition"},{"metadata":{},"cell_type":"markdown","source":"

I am no expert in speech domain, but these are some of the recent advancements in speech recognition systems. We will try to extract features through Wav2vec 2 model and finetune it on our usecase. For more detailed reading please go through the links attached in acknowledgement section

"},{"metadata":{},"cell_type":"markdown","source":"## 4. Modeling based on wav2vec representation"},{"metadata":{},"cell_type":"markdown","source":"

We will take the convolutional feature extractor from the pretrained model of wav2vec and used it to fintune it for our downstream task here

"},{"metadata":{},"cell_type":"markdown","source":"The code is heavily inspired from this [notebook](https://www.kaggle.com/gopidurgaprasad/rfcs-audio-detection-pytorch-stater)"},{"metadata":{},"cell_type":"markdown","source":"## 4.1 Preparing dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"root_dir='../input/rfcx-species-audio-detection/'\n\ntrain = pd.read_csv(root_dir+\"train_tp.csv\").sort_values(\"recording_id\")\nss = pd.read_csv(root_dir+\"sample_submission.csv\")\n\nFOLDS = 5\nSEED = 42\n\ntrain_gby = train.groupby(\"recording_id\")[[\"species_id\"]].first().reset_index()\ntrain_gby = train_gby.sample(frac=1, random_state=SEED).reset_index(drop=True)\ntrain_gby.loc[:, 'kfold'] = -1\n\nX = train_gby[\"recording_id\"].values\ny = train_gby[\"species_id\"].values\n\nkfold = StratifiedKFold(n_splits=FOLDS)\nfor fold, (t_idx, v_idx) in enumerate(kfold.split(X, y)):\n train_gby.loc[v_idx, \"kfold\"] = fold\n\ntrain = train.merge(train_gby[['recording_id', 'kfold']], on=\"recording_id\", how=\"left\")\nprint(train.kfold.value_counts())\ntrain.to_csv(\"train_folds.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head(2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.2 Dataset class"},{"metadata":{"trusted":true},"cell_type":"code","source":"class AudioDataset(Dataset):\n def __init__(self, df, period=10, transforms=None, data_path=\"train\", train=True):\n self.period = period\n self.transforms = transforms\n self.data_path = data_path\n self.train = train\n \n if train: \n dfgby = df.groupby(\"recording_id\").agg(lambda x: list(x)).reset_index()\n self.recording_ids = dfgby[\"recording_id\"].values\n self.species_ids = dfgby[\"species_id\"].values\n self.t_mins = dfgby[\"t_min\"].values\n self.t_maxs = dfgby[\"t_max\"].values\n else:\n self.recording_ids = df[\"recording_id\"].values\n\n \n def __len__(self):\n return len(self.recording_ids)\n \n def __getitem__(self, idx):\n\n recording_id = self.recording_ids[idx]\n\n if self.train:\n species_id = self.species_ids[idx]\n t_min, t_max = self.t_mins[idx], self.t_maxs[idx]\n else:\n species_id = [0]\n t_min, t_max = [0], [0]\n\n y, sr = sf.read(f\"{self.data_path}/{recording_id}.flac\")\n #y, sr = librosa.load(f\"{self.data_path}/{recording_id}.flac\", sr=16000) \n\n len_y = len(y)\n effective_length = sr * self.period\n rint = np.random.randint(len(t_min))\n tmin, tmax = round(sr * t_min[rint]), round(sr * t_max[rint])\n\n if len_y < effective_length:\n start = np.random.randint(effective_length - len_y)\n new_y = np.zeros(effective_length, dtype=y.dtype)\n new_y[start:start+len_y] = y\n y = new_y.astype(np.float32)\n\n elif len_y > effective_length:\n center = round((tmin + tmax) / 2)\n big = center - effective_length\n if big < 0:\n big = 0\n\n start = np.random.randint(big, center)\n y = y[start:start+effective_length]\n if len(y) < effective_length:\n new_y = np.zeros(effective_length, dtype=y.dtype)\n start1 = np.random.randint(effective_length - len(y))\n new_y[start1:start1+len(y)] = y\n y = new_y.astype(np.float32)\n else:\n y = y.astype(np.float32)\n else:\n y = y.astype(np.float32)\n start = 0\n \n if self.transforms:\n y = self.transforms(samples=y, sample_rate=sr)\n \n start_time = start / sr\n end_time = (start + effective_length) / sr\n\n label = np.zeros(24, dtype='f')\n\n if self.train:\n for i in range(len(t_min)):\n if (t_min[i] >= start_time) & (t_max[i] <= end_time):\n label[species_id[i]] = 1\n elif start_time <= ((t_min[i] + t_max[i]) / 2) <= end_time:\n label[species_id[i]] = 1\n \n return {\n \"waveform\" : y,\n \"target\" : torch.tensor(label, dtype=torch.float),\n \"id\" : recording_id\n }\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"a=AudioDataset(train,data_path=root_dir+'train')\na[0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.3 Model definition"},{"metadata":{},"cell_type":"markdown","source":"

Hugging face has released wav2vec2 model in their recent version. This makes it lot more easier to use those pretrained models for downstream finetuning

"},{"metadata":{"trusted":true},"cell_type":"code","source":"class AudioClassifier(nn.Module):\n def __init__(self,path,classes_num,hfmodel=False):\n super().__init__()\n\n if not hfmodel:\n self.model, self.cfg,self.task = fairseq.checkpoint_utils.load_model_ensemble_and_task([path])\n self.model = self.model[0]\n self.fc = Linear(512, classes_num)\n else:\n self.model = Wav2Vec2Model.from_pretrained(path)\n self.fc = Linear(256, classes_num)\n \n self.dropout = Dropout(0.3)\n self.hfmodel=hfmodel\n \n \n def forward(self, input, spec_aug=False, mixup_lambda=None):\n\n if not self.hfmodel:\n wav2feature = self.model.feature_extractor(input)\n x1 = F.max_pool1d(wav2feature, kernel_size=3, stride=1)\n x2 = F.avg_pool1d(wav2feature, kernel_size=3, stride=1)\n x = x1 + x2\n x = F.dropout(x, p=0.5, training=self.training)\n x = self.fc(torch.sum(x,axis=2))\n \n else:\n wav2feature = self.model(input).last_hidden_state\n x1 = F.max_pool1d(wav2feature, kernel_size=3)\n x2 = F.avg_pool1d(wav2feature, kernel_size=3) \n x = x1 + x2\n x = F.dropout(x, p=0.5, training=self.training)\n x = self.fc(torch.sum(x,axis=1))\n \n \n return x","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"m=AudioClassifier('./wav2vec_large.pt',24)\n\nres=m(torch.tensor(a[0]['waveform']).reshape(1,-1))\nres.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.4 Utils"},{"metadata":{"trusted":true},"cell_type":"code","source":"# https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/198418#1086063\ndef _one_sample_positive_class_precisions(scores, truth):\n num_classes = scores.shape[0]\n pos_class_indices = np.flatnonzero(truth > 0)\n\n if not len(pos_class_indices):\n return pos_class_indices, np.zeros(0)\n\n retrieved_classes = np.argsort(scores)[::-1]\n\n class_rankings = np.zeros(num_classes, dtype=np.int)\n class_rankings[retrieved_classes] = range(num_classes)\n\n retrieved_class_true = np.zeros(num_classes, dtype=np.bool)\n retrieved_class_true[class_rankings[pos_class_indices]] = True\n\n retrieved_cumulative_hits = np.cumsum(retrieved_class_true)\n\n precision_at_hits = (\n retrieved_cumulative_hits[class_rankings[pos_class_indices]] /\n (1 + class_rankings[pos_class_indices].astype(np.float)))\n return pos_class_indices, precision_at_hits\n\ndef lwlrap(truth, scores):\n assert truth.shape == scores.shape\n num_samples, num_classes = scores.shape\n precisions_for_samples_by_classes = np.zeros((num_samples, num_classes))\n for sample_num in range(num_samples):\n pos_class_indices, precision_at_hits = _one_sample_positive_class_precisions(scores[sample_num, :], truth[sample_num, :])\n precisions_for_samples_by_classes[sample_num, pos_class_indices] = precision_at_hits\n\n labels_per_class = np.sum(truth > 0, axis=0)\n weight_per_class = labels_per_class / float(np.sum(labels_per_class))\n\n per_class_lwlrap = (np.sum(precisions_for_samples_by_classes, axis=0) /\n np.maximum(1, labels_per_class))\n return per_class_lwlrap, weight_per_class\n\nclass AverageMeter(object):\n \"\"\"Computes and stores the average and current value\"\"\"\n\n def __init__(self):\n self.reset()\n\n def reset(self):\n self.val = 0\n self.avg = 0\n self.sum = 0\n self.count = 0\n\n def update(self, val, n=1):\n self.val = val\n self.sum += val * n\n self.count += n\n self.avg = self.sum / self.count\n\nclass MetricMeter(object):\n def __init__(self):\n self.reset()\n \n def reset(self):\n self.y_true = []\n self.y_pred = []\n \n def update(self, y_true, y_pred):\n self.y_true.extend(y_true.cpu().detach().numpy().tolist())\n self.y_pred.extend(torch.sigmoid(y_pred).cpu().detach().numpy().tolist())\n\n @property\n def avg(self):\n \n score_class, weight = lwlrap(np.array(self.y_true), np.array(self.y_pred))\n self.score = (score_class * weight).sum()\n\n return {\n \"lwlrap\" : self.score\n }","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.5 Engine"},{"metadata":{"trusted":true},"cell_type":"code","source":"def train_epoch(config, model, loader, criterion, optimizer, scheduler, epoch,tokenizer):\n losses = AverageMeter()\n scores = MetricMeter()\n\n model.train()\n t = tqdm(loader)\n \n\n for i, sample in enumerate(t):\n optimizer.zero_grad()\n \n waveform_list=sample['waveform'].tolist()\n input_values =tokenizer(waveform_list, return_tensors = \"pt\").input_values\n \n input = input_values.to(config['device'])\n target = sample['target'].to(config['device'])\n output = model(input)\n loss = criterion(output, target)\n loss.backward()\n optimizer.step()\n if scheduler and config['step_scheduler']:\n scheduler.step()\n\n bs = input.size(0)\n #scores.update(target, torch.sigmoid(torch.max(output['framewise_output'], dim=1)[0]))\n scores.update(target, output)\n losses.update(loss.item(), bs)\n\n t.set_description(f\"Train E:{epoch} - Loss{losses.avg:0.4f}\")\n t.close()\n return scores.avg, losses.avg\n \ndef valid_epoch(config, model, loader, criterion, epoch,tokenizer):\n losses = AverageMeter()\n scores = MetricMeter()\n \n model.eval()\n \n with torch.no_grad():\n t = tqdm(loader)\n for i, sample in enumerate(t):\n \n waveform_list=sample['waveform'].tolist()\n input_values =tokenizer(waveform_list, return_tensors = \"pt\").input_values\n \n input = input_values.to(config['device'])\n target = sample['target'].to(config['device'])\n output = model(input)\n loss = criterion(output, target)\n\n bs = input.size(0)\n scores.update(target,output)\n losses.update(loss.item(), bs)\n t.set_description(f\"Valid E:{epoch} - Loss:{losses.avg:0.4f}\")\n t.close()\n return scores.avg, losses.avg\n\ndef test_epoch(config, model, loader,tokenizer):\n\n model.eval()\n pred_list = []\n id_list = []\n with torch.no_grad():\n t = tqdm(loader)\n for i, sample in enumerate(t):\n waveform_list=sample['waveform'].tolist()\n input_values =tokenizer(waveform_list, return_tensors = \"pt\").input_values\n \n input = input_values.to(config['device'])\n id = sample[\"id\"]\n output = torch.sigmoid(model(input)).cpu().detach().numpy().tolist()\n pred_list.extend(output)\n id_list.extend(id)\n\n return pred_list, id_list","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.6 Fold run"},{"metadata":{},"cell_type":"markdown","source":"

Let's train it on few epochs

"},{"metadata":{"trusted":true},"cell_type":"code","source":"\ndef seed_everything(seed):\n # Setting seed\n random.seed(seed)\n os.environ['PYTHONHASHSEED'] = str(seed)\n np.random.seed(seed)\n torch.manual_seed(seed)\n torch.cuda.manual_seed(seed)\n torch.backends.cudnn.deterministic = True\n\n\ndef main(fold,config):\n seed_everything(config['seed'])\n\n config['fold'] = fold\n config['save_path'] = os.path.join(config['output_dir'],config['exp_name'])\n os.makedirs(config['save_path'], exist_ok=True)\n\n train_df = pd.read_csv(config['train_csv'])\n sub_df = pd.read_csv(config['sub_csv'])\n \n if config['DEBUG']:\n train_df = train_df.sample(200)\n\n train_fold = train_df[train_df.kfold != fold]\n valid_fold = train_df[train_df.kfold == fold]\n\n train_dataset = AudioDataset(\n df = train_fold,\n period=config['period'],\n transforms=None,\n data_path=root_dir+'train',\n train=True\n )\n\n valid_dataset = AudioDataset(\n df = valid_fold,\n period=config['period'],\n #stride=5,\n transforms=None,\n data_path=root_dir+'train',\n train=True\n )\n \n\n test_dataset = AudioDataset(\n df = sub_df,\n period=60,\n #stride=5,\n transforms=None,\n data_path=root_dir+'test',\n train=False\n )\n\n train_loader = torch.utils.data.DataLoader(\n train_dataset,\n batch_size=config['batch_size'],\n shuffle=True,\n drop_last=True,\n num_workers=config['num_workers']\n )\n\n valid_loader = torch.utils.data.DataLoader(\n valid_dataset,\n batch_size=config['batch_size'],\n shuffle=False,\n drop_last=False,\n num_workers=config['num_workers']\n )\n\n test_loader = torch.utils.data.DataLoader(\n test_dataset,\n batch_size=config['batch_size']//2,\n shuffle=False,\n drop_last=False,\n num_workers=config['num_workers']\n )\n\n tokenizer = Wav2Vec2Tokenizer.from_pretrained(config.hf_path)\n \n #model = AudioClassifier('./wav2vec_large.pt',24,True)\n model = AudioClassifier(config.hf_path,24,True)\n \n model = model.to(config['device'])\n\n if config['pretrain_weights']:\n model.load_state_dict(torch.load(config['pretrain_weights'], map_location=config['device']), strict=False)\n model = model.to(config['device'])\n\n criterion = BCEWithLogitsLoss() #PANNsLoss() #MaskedBCEWithLogitsLoss() #BCEWithLogitsLoss()\n optimizer = torch.optim.Adam(model.parameters(), lr=config['lr'])\n num_train_steps = int(len(train_loader) * config['epochs'])\n num_warmup_steps = int(0.1 * config['epochs'] * len(train_loader))\n scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=num_warmup_steps, num_training_steps=num_train_steps)\n\n best_lwlrap = -np.inf\n early_stop_count = 0\n wandb.watch(model, criterion, log=\"all\", log_freq=10)\n\n for epoch in range(config['start_epoch'], config['epochs']):\n train_avg, train_loss = train_epoch(config, model, train_loader, criterion, optimizer, scheduler, epoch,tokenizer)\n valid_avg, valid_loss = valid_epoch(config, model, valid_loader, criterion, epoch,tokenizer)\n\n if config['epoch_scheduler']:\n scheduler.step()\n \n content = f\"\"\"\n {time.ctime()} \\n\n Fold:{config['fold']}, Epoch:{epoch}, lr:{optimizer.param_groups[0]['lr']:.7}\\n\n Train Loss:{train_loss:0.4f} - LWLRAP:{train_avg['lwlrap']:0.4f}\\n\n Valid Loss:{valid_loss:0.4f} - LWLRAP:{valid_avg['lwlrap']:0.4f}\\n\n \"\"\"\n print(content)\n \n wandb.log({\"LWLRAP_train\": train_avg['lwlrap']})\n wandb.log({\"LWLRAP_valid\": valid_avg['lwlrap']})\n\n with open(config['save_path']+'/log_'+config['exp_name']+'.txt', 'a') as appender:\n appender.write(content+'\\n')\n \n if valid_avg['lwlrap'] > best_lwlrap:\n print(f\"########## >>>>>>>> Model Improved From {best_lwlrap} ----> {valid_avg['lwlrap']}\")\n torch.save(model.state_dict(), os.path.join(config['save_path'],'fold-'+str(config['fold'])+'.bin'))\n best_lwlrap = valid_avg['lwlrap']\n early_stop_count = 0\n else:\n early_stop_count += 1\n \n if config['early_stop'] == early_stop_count:\n print(\"\\n we reached early stoping count :\", early_stop_count)\n break\n \n model.load_state_dict(torch.load(os.path.join(config['save_path'],'fold-'+str(config['fold'])+'.bin'), map_location=config['device']))\n model = model.to(config['device'])\n\n target_cols = sub_df.columns[1:].values.tolist()\n test_pred, ids = test_epoch(config, model, test_loader,tokenizer)\n print(np.array(test_pred).shape)\n\n test_pred_df = pd.DataFrame({\n \"recording_id\" : sub_df.recording_id.values\n })\n test_pred_df[target_cols] = test_pred\n test_pred_df.to_csv(os.path.join(config['save_path'], f\"fold-{config['fold']}-submission.csv\"), index=False)\n print(os.path.join(config['save_path'], f\"fold-{config['fold']}-submission.csv\"))\n\n\nif __name__ == \"__main__\":\n \n for fold in range(1):\n config = dict(\n DEBUG = False,\n wandb = False,\n exp_name = \"W2V_V1\",\n network = \"AudioClassifier\",\n pretrain_weights = None,\n lr = 1e-3,\n step_scheduler = True,\n epoch_scheduler = False,\n period = 10,\n seed = 42,\n start_epoch = 0,\n epochs = 1,\n batch_size = 2,\n num_workers = 2,\n early_stop = 10,\n\n device = ('cuda' if torch.cuda.is_available() else 'cpu'),\n train_csv = \"./train_folds.csv\",\n test_csv = \"test_df.csv\",\n sub_csv = root_dir+\"sample_submission.csv\",\n output_dir = \"weights\",\n hf_path=\"facebook/wav2vec2-base-960h\"\n )\n\n with wandb.init(project=f\"pytorch_audio_{config['exp_name']}\"+f\"_wand_{fold}\", config=config):\n \n # access all HPs through wandb.config, so logging matches execution!\n config = wandb.config\n \n main(fold,config)\n \n print(f\"***********Fold:{fold} done*******\")\n print(\"\\n\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6. Acknowledgements"},{"metadata":{},"cell_type":"markdown","source":"1. https://medium.com/analytics-vidhya/understanding-the-mel-spectrogram-fca2afa2ce53\n2. https://maelfabien.github.io/machinelearning/wav2vec/#e-end-to-end-model\n3. https://www.kaggle.com/gopidurgaprasad/rfcs-audio-detection-pytorch-stater "},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}