![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F13651743%2F0d18919aee04af8f54bfe0880472543d%2Fphoto_2023-05-25%2023.34.07.jpeg?generation=1685046878803412&alt=media\")
efficientnet_b2 from torchvision and rexnet150 from timmopenvinorkl prefix is indicator)Our five models from 19 with rkl-rx-wdfbn-rx-ema-200-full prefix achieve 0.7471 on private test set and 0.83194 on public test set, rx - rexnet150, distillated rexnet150 models from rexnet150 trained before
Let's imagine you're at some party right now and you're surrounded by talking people. It's not hard for you to pay attention to a certain group to hear their conversation and you can freely change your target. So the task - to cut specific speaker from audio - is natural. What we can say about audio in which - one person cries and another one laughs? In such audio there are two actions - laughing and crying, and this is true for birds too. With this consideration we've come up to sumup idea:
def sumup(waves: torch.Tensor, : torch.Tensor):\n batch_size = len()\n perm = torch.randperm(batch_size)\n\n waves = waves + waves[perm]\n\n {\n : waves,\n : torch.clip( + [perm], =, =)\n }\nWhat if you have an audio where one bird is more louder than another? This audio is still with two different birds, but if signal from second bird is tiny, it's okay to assume, that maybe probability of second bird is small or even zero.
\ndef sumix(waves: torch.Tensor, : torch.Tensor, max_percent: = , min_percent: = ):\n batch_size = len()\n perm = torch.randperm(batch_size)\n coeffs_1 = torch.rand(batch_size, device=waves.device).(-, ) * (\n max_percent - min_percent\n ) + min_percent\n coeffs_2 = torch.rand(batch_size, device=waves.device).(-, ) * (\n max_percent - min_percent\n ) + min_percent\n label_coeffs_1 = torch.where(coeffs_1 >= , , - * ( - coeffs_1))\n label_coeffs_2 = torch.where(coeffs_2 >= , , - * ( - coeffs_2))\n = label_coeffs_1 * + label_coeffs_2 * [perm]\n\n waves = coeffs_1 * waves + coeffs_2 * waves[perm]\n {\n : waves,\n : torch.clip(, , )\n }\nWe propose yet another customization of data mix specific for audio domain - sumix augmentation. As you can see, we used linear slope in our experiments, but it's open question what's slope for labels better to use, maybe min_percent = 0.3 doesn't need slope at all. This augmentation is applicable to any audio classification task. sumix is better than sumup because it also affects loudness, i. e. implies another audio augmentation.
When we decided to check solutions from past competitions, we tried to add background noise in our pipeline, but noise didn't show any improvement. Our thoughts - incorporating new background noises for audios is already in sumix or sumup. For gaussian/colored noises we have similar explanation, why those augmentations didn't work - competition's data is not excellent and consists of noisy samples, so sumix and sumup propagate noise and make noise diverse by themselves.
concatmix with probability 0.5 - implementationsumix(min_percent=0.3, max_percent=1.0) with probability 1mixup on mel-spectrograms with Beta(1.5, 1.5) with probability 1cutmix on mel-spectrograms with Beta(1.5, 1.5) with probability 0.5AdamW with constant LR 1e-3 and weight_decay 1e-2 for all modelsbinary_cross_entropy for every class, labels are primary labels plus secondary labelsweight_decay for all normalization layersEMA with decay 0.999After training a model, we use it as teacher for another model with the same architecture, augmentations, AdamW, EMA and etc., except loss. At second stage with distillation we use $$L = 0.33 * BceFromTargets + 0.34 * KL(probs_{student} || probs_{teacher}) + 0.33 * KL(probs_{teacher} || probs_{student}) $$
rexnet150 with dropout 0.3 and drop_path 0.2rexnet150-time - same rexnet150 but with attention head, distilled from rexnet150 without time-aware headefficientnet-b2 with dropout 0.3 and drop_path 0.2efficientnet-b2-time - analogously, distilled from efficientnet-b2Already mentioned hacks allow to reach score 0.7471 with 5 rexnet150 trained on full data with batch_size=336 200 epochs and knowledge distillation on a single A100 (40GB). To boost performance we ensemble rexnet150 with efficientnet-b2 from torchvision. To boost performance yet another time we use attention head - copy-past self-attention operation from transformers. All experiments were conducted in mix-precision with bfloat16 type.
Ensemble of 5 models with prefix rkl-rx150-wdfbn-rx150-time-ema-ep200-full-{5seeds}, i.e rexnet150 with attention head for time awareness distilled from simple rexnet150, achieves 0.75151 on private set and 0.83261 on public set.
eca_nfnet from timmtf_efficientnet_b2_ns from timmSwinV2/ConvNeXt from torchvisionfocal_loss, soft_focal_lossSpecAugmentWe measured inference time with random input in kaggle kernel, I don't remember which one of intel's processors was, often in our kaggle notebook we got Intel Xeon 2.2 Gz.
Our thoughts - ONNX by itself utilizes 2 cpus, OpenVino is faster cause it additionally fuses some layers for inference and uses compilation specific to a certain CPU in submission notebook to take advantage of processor's type
MultiHeadAttentionClassifier's implementation. All hyperparameters are preservedefficientnet_b2 from torchvision and rexnet150 from timmopenvinorkl prefix is indicator)Our five models from 19 with rkl-rx-wdfbn-rx-ema-200-full prefix achieve 0.7471 on private test set and 0.83194 on public test set, rx - rexnet150, distillated rexnet150 models from rexnet150 trained before
Let's imagine you're at some party right now and you're surrounded by talking people. It's not hard for you to pay attention to a certain group to hear their conversation and you can freely change your target. So the task - to cut specific speaker from audio - is natural. What we can say about audio in which - one person cries and another one laughs? In such audio there are two actions - laughing and crying, and this is true for birds too. With this consideration we've come up to sumup idea:
def sumup(waves: torch.Tensor, : torch.Tensor):\n batch_size = len()\n perm = torch.randperm(batch_size)\n\n waves = waves + waves[perm]\n\n {\n : waves,\n : torch.clip( + [perm], =, =)\n }\nWhat if you have an audio where one bird is more louder than another? This audio is still with two different birds, but if signal from second bird is tiny, it's okay to assume, that maybe probability of second bird is small or even zero.
\ndef sumix(waves: torch.Tensor, : torch.Tensor, max_percent: = , min_percent: = ):\n batch_size = len()\n perm = torch.randperm(batch_size)\n coeffs_1 = torch.rand(batch_size, device=waves.device).(-, ) * (\n max_percent - min_percent\n ) + min_percent\n coeffs_2 = torch.rand(batch_size, device=waves.device).(-, ) * (\n max_percent - min_percent\n ) + min_percent\n label_coeffs_1 = torch.where(coeffs_1 >= , , - * ( - coeffs_1))\n label_coeffs_2 = torch.where(coeffs_2 >= , , - * ( - coeffs_2))\n = label_coeffs_1 * + label_coeffs_2 * [perm]\n\n waves = coeffs_1 * waves + coeffs_2 * waves[perm]\n {\n : waves,\n : torch.clip(, , )\n }\nWe propose yet another customization of data mix specific for audio domain - sumix augmentation. As you can see, we used linear slope in our experiments, but it's open question what's slope for labels better to use, maybe min_percent = 0.3 doesn't need slope at all. This augmentation is applicable to any audio classification task. sumix is better than sumup because it also affects loudness, i. e. implies another audio augmentation.
When we decided to check solutions from past competitions, we tried to add background noise in our pipeline, but noise didn't show any improvement. Our thoughts - incorporating new background noises for audios is already in sumix or sumup. For gaussian/colored noises we have similar explanation, why those augmentations didn't work - competition's data is not excellent and consists of noisy samples, so sumix and sumup propagate noise and make noise diverse by themselves.
concatmix with probability 0.5 - implementationsumix(min_percent=0.3, max_percent=1.0) with probability 1mixup on mel-spectrograms with Beta(1.5, 1.5) with probability 1cutmix on mel-spectrograms with Beta(1.5, 1.5) with probability 0.5AdamW with constant LR 1e-3 and weight_decay 1e-2 for all modelsbinary_cross_entropy for every class, labels are primary labels plus secondary labelsweight_decay for all normalization layersEMA with decay 0.999After training a model, we use it as teacher for another model with the same architecture, augmentations, AdamW, EMA and etc., except loss. At second stage with distillation we use $$L = 0.33 * BceFromTargets + 0.34 * KL(probs_{student} || probs_{teacher}) + 0.33 * KL(probs_{teacher} || probs_{student}) $$
rexnet150 with dropout 0.3 and drop_path 0.2rexnet150-time - same rexnet150 but with attention head, distilled from rexnet150 without time-aware headefficientnet-b2 with dropout 0.3 and drop_path 0.2efficientnet-b2-time - analogously, distilled from efficientnet-b2Already mentioned hacks allow to reach score 0.7471 with 5 rexnet150 trained on full data with batch_size=336 200 epochs and knowledge distillation on a single A100 (40GB). To boost performance we ensemble rexnet150 with efficientnet-b2 from torchvision. To boost performance yet another time we use attention head - copy-past self-attention operation from transformers. All experiments were conducted in mix-precision with bfloat16 type.
Ensemble of 5 models with prefix rkl-rx150-wdfbn-rx150-time-ema-ep200-full-{5seeds}, i.e rexnet150 with attention head for time awareness distilled from simple rexnet150, achieves 0.75151 on private set and 0.83261 on public set.
eca_nfnet from timmtf_efficientnet_b2_ns from timmSwinV2/ConvNeXt from torchvisionfocal_loss, soft_focal_lossSpecAugmentWe measured inference time with random input in kaggle kernel, I don't remember which one of intel's processors was, often in our kaggle notebook we got Intel Xeon 2.2 Gz.
Our thoughts - ONNX by itself utilizes 2 cpus, OpenVino is faster cause it additionally fuses some layers for inference and uses compilation specific to a certain CPU in submission notebook to take advantage of processor's type
MultiHeadAttentionClassifier's implementation. All hyperparameters are preservedCongrats for landing in the gold zone and thanks for sharing the writeup
", "rawMarkdown": "Congrats for landing in the gold zone and thanks for sharing the writeup", "votes": null } ], "comments": [ { "id": 2286252, "author_name": "pardeep19singh", "author_url": "", "post_date": "06/03/2023 10:31:07", "content": "Congrats for landing in the gold zone and thanks for sharing the writeup
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2274329": "#### What's important in [our best inference notebook](https://www.kaggle.com/code/loonypenguin/ensemble-openvino-models-inference-notebook) ?\n\n- Ensemble of 19 models\n- Ensemble of `efficientnet_b2` from `torchvision` and `rexnet150` from `timm`\n- Speedup with `openvino`\n- For submission we trained all models with knowledge distillation on full dataset - (`rkl` prefix is indicator)\n\n\nOur five models from 19 with `rkl-rx-wdfbn-rx-ema-200-full` prefix achieve 0.7471 on private test set and 0.83194 on public test set, `rx` - `rexnet150`, distillated `rexnet150` models from `rexnet150` trained before\n\n#### Custom Augmentations\n\nLet's imagine you're at some party right now and you're surrounded by talking people. It's not hard for you to pay attention to a certain group to hear their conversation and you can freely change your target. So the task - to cut specific speaker from audio - is natural. What we can say about audio in which - one person cries and another one laughs? In such audio there are two actions - laughing and crying, and this is true for birds too. With this consideration we've come up to `sumup` idea:\n\n```python3\ndef sumup(waves: torch.Tensor, labels: torch.Tensor):\n batch_size = len(labels)\n perm = torch.randperm(batch_size)\n\n waves = waves + waves[perm]\n\n return {\n \"waves\": waves,\n \"labels\": torch.clip(labels + labels[perm], min=0, max=1)\n }\n```\n\n###### Boosting score\nWhat if you have an audio where one bird is more louder than another? This audio is still with two different birds, but if signal from second bird is tiny, it's okay to assume, that maybe probability of second bird is small or even zero. \n\n```python3\ndef sumix(waves: torch.Tensor, labels: torch.Tensor, max_percent: float = 1.0, min_percent: float = 0.3):\n batch_size = len(labels)\n perm = torch.randperm(batch_size)\n coeffs_1 = torch.rand(batch_size, device=waves.device).view(-1, 1) * (\n max_percent - min_percent\n ) + min_percent\n coeffs_2 = torch.rand(batch_size, device=waves.device).view(-1, 1) * (\n max_percent - min_percent\n ) + min_percent\n label_coeffs_1 = torch.where(coeffs_1 >= 0.5, 1, 1 - 2 * (0.5 - coeffs_1))\n label_coeffs_2 = torch.where(coeffs_2 >= 0.5, 1, 1 - 2 * (0.5 - coeffs_2))\n labels = label_coeffs_1 * labels + label_coeffs_2 * labels[perm]\n\n waves = coeffs_1 * waves + coeffs_2 * waves[perm]\n return {\n \"waves\": waves,\n \"labels\": torch.clip(labels, 0, 1)\n }\n```\n\nWe propose yet another customization of data mix specific for audio domain - `sumix` augmentation. As you can see, we used linear slope in our experiments, but it's open question what's slope for labels better to use, maybe `min_percent = 0.3` doesn't need slope at all. This augmentation is applicable to any audio classification task. `sumix` is better than `sumup` because it also affects loudness, i. e. implies another audio augmentation. \n\nWhen we decided to check solutions from past competitions, we tried to add background noise in our pipeline, but noise didn't show any improvement. Our thoughts - incorporating new background noises for audios is already in `sumix` or `sumup`. For gaussian/colored noises we have similar explanation, why those augmentations didn't work - competition's data is not excellent and consists of noisy samples, so `sumix` and `sumup` propagate noise and make noise diverse by themselves.\n\n#### Training pipeline\n- `concatmix` with probability 0.5 - [implementation](https://github.com/dazzle-me/birdclef-2022-3rd-place-solution/blob/main/experiments/new-metrics-exp-17/fold_4/saved_src/models/layers/concat_mix.py)\n- `sumix(min_percent=0.3, max_percent=1.0)` with probability 1\n- `mixup` on mel-spectrograms with `Beta(1.5, 1.5)` with probability 1\n- `cutmix` on mel-spectrograms with `Beta(1.5, 1.5)` with probability 0.5\n- `AdamW` with constant `LR` 1e-3 and `weight_decay` 1e-2 for all models\n- `binary_cross_entropy` for every class, labels are primary labels plus secondary labels\n- Random 5 continuous seconds from audio at training, first 5 seconds at validation\n- We turned off `weight_decay` for all normalization layers\n- `EMA` with `decay` 0.999\n- Almost total correlation between LB and our CV, ensemble of 20 models gathered as best in CV achieves 0.75021 on private test set and 0.83328 on public test set, this ensemble is our top-3 submission on public test set\n\n#### Knowledge distillation\n\nAfter training a model, we use it as teacher for another model with the same architecture, augmentations, `AdamW`, `EMA` and etc., except `loss`. At second stage with distillation we use $$L = 0.33 * BceFromTargets + 0.34 * KL(probs_{student} || probs_{teacher}) + 0.33 * KL(probs_{teacher} || probs_{student}) $$\n\n\n#### Models\n- `rexnet150` with `dropout` 0.3 and `drop_path` 0.2\n- `rexnet150-time` - same `rexnet150` but with attention head, distilled from `rexnet150` without `time`-aware head\n- `efficientnet-b2` with `dropout` 0.3 and `drop_path` 0.2\n- `efficientnet-b2-time` - analogously, distilled from `efficientnet-b2`\n\nAlready mentioned hacks allow to reach score 0.7471 with 5 `rexnet150` trained on full data with `batch_size=336` 200 epochs and knowledge distillation on a single `A100 (40GB)`. To boost performance we ensemble `rexnet150` with `efficientnet-b2` from `torchvision`. To boost performance yet another time we use attention head - copy-past `self-attention` operation from `transformers`. All experiments were conducted in mix-precision with `bfloat16` type.\n\nEnsemble of 5 models with prefix `rkl-rx150-wdfbn-rx150-time-ema-ep200-full-{5seeds}`, i.e `rexnet150` with attention head for time awareness distilled from simple `rexnet150`, achieves 0.75151 on private set and 0.83261 on public set.\n\n#### We didn't try:\n- External data not from BirdClef2021/BirdClef2022\n- SSL\n- SED models\n- Utilization of BirdNet\n- `eca_nfnet` from `timm`\n\n#### We tried, but it didn't work:\n- Pretraining our models on BirdClef2021/BirdClef2022, we made few attempts, it didn't show any improvements in LB, experiments conduction became much slower, so we gave up the idea\n- Any background noise (even which was used in previous year's competition)\n- Any LR scheduler\n- Any Noise(Gauss, Colored)\n- `tf_efficientnet_b2_ns` from `timm`\n- `SwinV2`/`ConvNeXt` from `torchvision`\n- Manifold MixUp\n- `focal_loss`, `soft_focal_loss`\n- Weighted samplers\n- `SpecAugment`\n\n#### Speeding up inference time\n\nWe measured inference time with random input in kaggle kernel, I don't remember which one of intel's processors was, often in our kaggle notebook we got `Intel Xeon 2.2 Gz`. \n\n\n\nOur thoughts - `ONNX` by itself utilizes 2 cpus, `OpenVino` is faster cause it additionally fuses some layers for inference and uses compilation specific to a certain CPU in submission notebook to take advantage of processor's type\n\n\n#### Appendix: `MultiHeadAttentionClassifier`'s implementation. All hyperparameters are preserved\n```python3\nclass MultiHeadSelfAttention(nn.Module):\n def __init__(\n self,\n input_channel: int,\n head_size: int,\n num_heads: int,\n attention_dropout: float,\n ) -> None:\n super().__init__()\n hidden_dim = head_size * num_heads\n self.hidden_dim = hidden_dim\n\n self.head_size = head_size\n self.num_heads = num_heads\n\n self.key = nn.Linear(input_channel, hidden_dim)\n self.query = nn.Linear(input_channel, hidden_dim)\n\n self.attention_dropout = nn.Dropout(attention_dropout)\n self.sqrt_head_size = sqrt(head_size)\n\n self.value = nn.Linear(input_channel, hidden_dim)\n\n def tranpose_for_scores(self, x: Tensor) -> Tensor:\n # [BS; T; H] -> [BS; T; K, M]\n new_x_shape = x.size()[:-1] + (self.num_heads, self.head_size)\n x = x.view(new_x_shape)\n return x.permute(0, 2, 1, 3) # [BS; K; T; M]\n\n def get_key(self, x: Tensor) -> Tensor:\n return self.tranpose_for_scores(self.key(x))\n\n def get_query(self, x: Tensor) -> Tensor:\n return self.tranpose_for_scores(self.query(x))\n\n def get_value(self, x: Tensor) -> Tensor:\n return self.tranpose_for_scores(self.value(x))\n\n def forward(self, x: Tensor) -> Tensor:\n # [BS; L; C]\n key = self.get_key(x)\n query = self.get_query(x)\n value = self.get_value(x)\n\n attention_scores = torch.matmul(query, key.transpose(-1, -2)) \n attention_scores /= self.sqrt_head_size\n attention_scores = F.softmax(attention_scores, dim=-1)\n attention_scores = self.attention_dropout(attention_scores)\n x = torch.matmul(attention_scores, value) # [BS; K; N; M]\n x = x.permute(0, 2, 1, 3).contiguous()\n x = x.view(x.shape[:2] + (self.hidden_dim,))\n return x\n\nclass MultiHeadAttentionClassifier(MultiHeadSelfAttention):\n def __init__(\n self,\n input_channel: int,\n head_size: int = 32,\n num_heads: int = 24,\n attention_dropout: float = 0.3,\n num_classes: int = 264,\n dropout: float = 0.3\n ) -> None:\n super().__init__(input_channel, head_size, num_heads, attention_dropout)\n self.query = nn.Parameter(\n torch.empty(num_heads, num_classes, head_size),\n requires_grad=True\n )\n nn.init.normal_(self.query)\n self.classifier = nn.Sequential(\n nn.Dropout(dropout),\n nn.Conv1d(num_classes, num_classes, kernel_size=self.hidden_dim, groups=num_classes)\n )\n\n def get_query(self, x: Tensor) -> Tensor:\n return self.query\n\n def forward(self, x: Tensor) -> Tensor:\n # x.shape == [BS; C; F; T] // output from backbone\n x = vec.view(x.shape[:-2] + (-1,))\n # [BS; C; L]\n x = vec.permute(0, 2, 1)\n # [BS; L; C]\n \n out = super().forward(x)\n return self.classifier(out).squeeze()\n```", "2286252": "Congrats for landing in the gold zone and thanks for sharing the writeup" }, "source": "meta" }