![\"image.png\"](\"https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/264781/2d0ad5b8-f046-64da-a98a-96f0ae79ba2e.png\")
First of all, many thanks to Kaggler.
\n I got a lot of ideas from Kaggler in the discussion. And it was fun.
My solution summary is below.
\nI started from basic experiment with SED. Why SED? Because I think SED is strong in the multi label task.
\nI used log mel-spectrogram as the input of SED. Basic experiment involves data augmentation (Gaussian noise, SpecAugment and MixUP), backbone model choice and adjusting resolution of log mel-spectrogram. As a result, below condition was the best for me.
\nThe most important one is the resolution. Recently, in Kaggle computer vision solution, the higher the resolution of the image, the better the result. In spectrogram, the same phenomenon may happen. In mel-spectrogram, the resolution can be changed by adjusting \"hop_size\" and \"mel_bins\". Following result is changing the resolution with ResNest50(single model).
\n| Resolution(Width-Height) | \npublic LB | \n
|---|---|
| 1501-64(PANNs default) | \n0.692 | \n
| 3001-128 | \n0.805 | \n
| 6001-64 | \n0.725 | \n
| 1501-256 | \n0.761 | \n
| 751-512 | \n0.823 | \n
| 1501-512 | \n0.821 | \n
The resolution was critical! According to experimental result, good resolution was \"high\" and similar to the square. 751-512 looks good. As a result, I chose 858-850. This configuration is as follows.
\nmodel_config = {\n \"sample_rate\": 48000,\n \"window_size\": 1024,\n \"hop_size\": 560,\n \"mel_bins\": 850,\n \"fmin\": 50,\n \"fmax\": 14000,\n \"classes_num\": 24\n}\nI used Framewise output for submission. It contains time and classes information. But there is a lot of false positive information in framewise output. Because they are not processing by a long time information. Therefore a short event of framewise output should be deleted. I prepared post-processing for framewise output. It is a moving average.
\n
By taking a moving average in the time direction for each class, we can delete short events. This idea is based on the paper[1]. The sample code is as follows.
\ndef post_processing(data): # data.shape = (24, 600) # (classes, time)\n result = []\n for i in range(len(data)):\n result.append(cv2.blur(data[i],(1,31)))\n return result\nI improved LB by using moving average. The following result is comparing post-processing with EfficientNetB3(single model).
\n| \n | public LB | \n
|---|---|
| w/o post-processing | \n0.785 | \n
| w/ post-processing | \n0.840 | \n
Then I got 0.916 public LB with EfficientNetB0(5-folds average ensemble).
\nI reported discovering missing labels and re-train. And It didn't work. After that, I thought about missing labels again. My answer is that the model is not correct for discovering missing labels. There are a lot of missing labels around tp. Therefore the model is not correct.
\n![\"train.png\"](\"https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/264781/0e6009db-c1cf-7297-3ac4-df92dd3420c5.png\")
To solve this issue, I used teacher-student model.
\n![\"geretation.png\"](\"https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/264781/79d8e1c8-6814-68c6-0a9d-d41d830fcdfb.png\")
1st generation is similar to 1st stage. I gradually increased model prediction ratio. By using teacher-student model, I could discover missing labels. Specially, in strong label training, teacher-student model was effective. Following result is teacher-student model score with EfficientNetB0.
\n![\"image.png\"](\"https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/264781/ddded273-c2b1-92c1-3272-1e19377a5b77.png\")
\"MixUp rate\" is probabilistic MixUp. This method is based on the paper[2].
\nFinally, I made the ensemble of 1st stage model and 2nd stage model. Ensemble procedure is simple average. Then I got 0.924 public LB.
\n[1] Teck Kai Chan, Cheng Siong Chin1 and Ye Li, \"SEMI-SUPERVISED NMF-CNN FOR SOUND EVENT DETECTION\".
\n[2] Yuan Gong, Yu-An Chung, and James Glass, \"PSLA: Improving Audio Event Classification with
\nPretraining, Sampling, Labeling, and Aggregation\".
Finally, I show interesting result. It is relationship between EfficientNet and the resolution. The following result is public LB(5-folds average ensemble).
\n| Resolution(W-H) | \n751-512 | \n751-751 | \n858-850 | \n
|---|---|---|---|
| EfficientNetB0 | \n0.893 | \n0.904 | \n0.916 | \n
| EfficientNetB3 | \n0.913 | \n0.912 | \n0.900 | \n
In B0, the higher resolution, the better result. But B3 was vice versa. Usually, the larger EfficientNet, the better at high resolution it is. But the above is reverse. Why?
\nMaybe domain shift(train: noisy sound -> test: clean sound) is concerned. B3 has learned about train domain features(noisy sound). On the other hand, B0 has less representational ability than B3. Therefore B0 has learned the common features of the train and test domain with high resolution. Without domain shift, B3 would have also shown good results with high resolution.
", "messages": [ { "id": "1207769", "postDate": "02/18/2021 03:05:33", "content": "First of all, many thanks to Kaggler.
\n I got a lot of ideas from Kaggler in the discussion. And it was fun.
My solution summary is below.
\nI started from basic experiment with SED. Why SED? Because I think SED is strong in the multi label task.
\nI used log mel-spectrogram as the input of SED. Basic experiment involves data augmentation (Gaussian noise, SpecAugment and MixUP), backbone model choice and adjusting resolution of log mel-spectrogram. As a result, below condition was the best for me.
\nThe most important one is the resolution. Recently, in Kaggle computer vision solution, the higher the resolution of the image, the better the result. In spectrogram, the same phenomenon may happen. In mel-spectrogram, the resolution can be changed by adjusting \"hop_size\" and \"mel_bins\". Following result is changing the resolution with ResNest50(single model).
\n| Resolution(Width-Height) | \npublic LB | \n
|---|---|
| 1501-64(PANNs default) | \n0.692 | \n
| 3001-128 | \n0.805 | \n
| 6001-64 | \n0.725 | \n
| 1501-256 | \n0.761 | \n
| 751-512 | \n0.823 | \n
| 1501-512 | \n0.821 | \n
The resolution was critical! According to experimental result, good resolution was \"high\" and similar to the square. 751-512 looks good. As a result, I chose 858-850. This configuration is as follows.
\nmodel_config = {\n \"sample_rate\": 48000,\n \"window_size\": 1024,\n \"hop_size\": 560,\n \"mel_bins\": 850,\n \"fmin\": 50,\n \"fmax\": 14000,\n \"classes_num\": 24\n}\nI used Framewise output for submission. It contains time and classes information. But there is a lot of false positive information in framewise output. Because they are not processing by a long time information. Therefore a short event of framewise output should be deleted. I prepared post-processing for framewise output. It is a moving average.
\n
By taking a moving average in the time direction for each class, we can delete short events. This idea is based on the paper[1]. The sample code is as follows.
\ndef post_processing(data): # data.shape = (24, 600) # (classes, time)\n result = []\n for i in range(len(data)):\n result.append(cv2.blur(data[i],(1,31)))\n return result\nI improved LB by using moving average. The following result is comparing post-processing with EfficientNetB3(single model).
\n| \n | public LB | \n
|---|---|
| w/o post-processing | \n0.785 | \n
| w/ post-processing | \n0.840 | \n
Then I got 0.916 public LB with EfficientNetB0(5-folds average ensemble).
\nI reported discovering missing labels and re-train. And It didn't work. After that, I thought about missing labels again. My answer is that the model is not correct for discovering missing labels. There are a lot of missing labels around tp. Therefore the model is not correct.
\n
To solve this issue, I used teacher-student model.
\n
1st generation is similar to 1st stage. I gradually increased model prediction ratio. By using teacher-student model, I could discover missing labels. Specially, in strong label training, teacher-student model was effective. Following result is teacher-student model score with EfficientNetB0.
\n
\"MixUp rate\" is probabilistic MixUp. This method is based on the paper[2].
\nFinally, I made the ensemble of 1st stage model and 2nd stage model. Ensemble procedure is simple average. Then I got 0.924 public LB.
\n[1] Teck Kai Chan, Cheng Siong Chin1 and Ye Li, \"SEMI-SUPERVISED NMF-CNN FOR SOUND EVENT DETECTION\".
\n[2] Yuan Gong, Yu-An Chung, and James Glass, \"PSLA: Improving Audio Event Classification with
\nPretraining, Sampling, Labeling, and Aggregation\".
Finally, I show interesting result. It is relationship between EfficientNet and the resolution. The following result is public LB(5-folds average ensemble).
\n| Resolution(W-H) | \n751-512 | \n751-751 | \n858-850 | \n
|---|---|---|---|
| EfficientNetB0 | \n0.893 | \n0.904 | \n0.916 | \n
| EfficientNetB3 | \n0.913 | \n0.912 | \n0.900 | \n
In B0, the higher resolution, the better result. But B3 was vice versa. Usually, the larger EfficientNet, the better at high resolution it is. But the above is reverse. Why?
\nMaybe domain shift(train: noisy sound -> test: clean sound) is concerned. B3 has learned about train domain features(noisy sound). On the other hand, B0 has less representational ability than B3. Therefore B0 has learned the common features of the train and test domain with high resolution. Without domain shift, B3 would have also shown good results with high resolution.
", "rawMarkdown": "First of all, many thanks to Kaggler.\n I got a lot of ideas from Kaggler in the discussion. And it was fun.\n\nMy solution summary is below.\n+ The high resolution of spectrogram\n+ Post-processing with moving average\n+ Teacher-student model for missing labels\n\n#1st stage: SED\nI started from basic experiment with [SED](https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/211007). Why SED? Because I think SED is strong in the multi label task. \n\nI used log mel-spectrogram as the input of SED. Basic experiment involves data augmentation (Gaussian noise, SpecAugment and MixUP), backbone model choice and adjusting resolution of log mel-spectrogram. As a result, below condition was the best for me.\n+ No noise injection\n+ MixUp\n+ The best model architecture is EfficientNet\n+ **The higher the resolution** of log mel-spectrogram, the better the result.\n\n### The resolution\nThe most important one is the resolution. Recently, in Kaggle computer vision solution, the higher the resolution of the image, the better the result. In spectrogram, the same phenomenon may happen. In mel-spectrogram, the resolution can be changed by adjusting \"hop_size\" and \"mel_bins\". Following result is changing the resolution with ResNest50(single model).\n\n|Resolution(Width-Height)|public LB|\n|---|---|\n|1501-64(PANNs default)|0.692\n|3001-128|0.805\n|6001-64|0.725\n|1501-256|0.761\n|751-512|**0.823**\n|1501-512|0.821\n\nThe resolution was critical! According to experimental result, good resolution was \"high\" and similar to the square. 751-512 looks good. As a result, I chose 858-850. This configuration is as follows.\n\n```\nmodel_config = {\n \"sample_rate\": 48000,\n \"window_size\": 1024,\n \"hop_size\": 560,\n \"mel_bins\": 850,\n \"fmin\": 50,\n \"fmax\": 14000,\n \"classes_num\": 24\n}\n```\n\n### Post-processing\nI used [Framewise output](https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/211007) for submission. It contains time and classes information. But there is a lot of false positive information in framewise output. Because they are not processing by a long time information. Therefore **a short event of framewise output should be deleted.** I prepared post-processing for framewise output. It is a moving average.\n\n\n\nBy taking a moving average in the time direction for each class, we can delete short events. This idea is based on [the paper](http://dcase.community/documents/challenge2020/technical_reports/DCASE2020_Chan_6.pdf)[1]. The sample code is as follows.\n```\ndef post_processing(data): # data.shape = (24, 600) # (classes, time)\n result = []\n for i in range(len(data)):\n result.append(cv2.blur(data[i],(1,31)))\n return result\n```\n\nI improved LB by using moving average. The following result is comparing post-processing with EfficientNetB3(single model).\n\n||public LB|\n|---|---|\n|w/o post-processing|0.785|\n|w/ post-processing|**0.840**|\n \n###Summary\n+ MixUp(alpha=0.1)\n+ Epoch 30\n+ Adam(lr=0.001) + CosineAnnealing(T=10)\n+ Batchsize 6\n+ Use only tp label\n+ Get random tp clip 10 sec\n+ The resolution of log mel-spectrogram: 858-850\n+ Loss function: BCE\n+ [Weak label training](https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/211007)\n+ Post-processing: moving average\n\nThen I got **0.916 public LB** with EfficientNetB0(5-folds average ensemble).\n\n#2nd stage: missing labels and the ensemble\nI reported [discovering missing labels and re-train](https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/208830#1139171). And It didn't work. After that, I thought about missing labels again. My answer is that the model is not correct for discovering missing labels. There are a lot of missing labels around tp. Therefore the model is not correct. \n\n\n\nTo solve this issue, I used teacher-student model. \n\n\n\n1st generation is similar to 1st stage. I gradually increased model prediction ratio. By using teacher-student model, I could discover missing labels. Specially, in strong label training, teacher-student model was effective. Following result is teacher-student model score with EfficientNetB0.\n\n\n\n\"MixUp rate\" is probabilistic MixUp. This method is based on [the paper](https://arxiv.org/abs/2102.01243)[2]. \n\nFinally, I made the ensemble of 1st stage model and 2nd stage model. Ensemble procedure is simple average. Then I got **0.924 public LB.**\n\n#References\n[1] Teck Kai Chan, Cheng Siong Chin1 and Ye Li, \"SEMI-SUPERVISED NMF-CNN FOR SOUND EVENT DETECTION\".\n[2] Yuan Gong, Yu-An Chung, and James Glass, \"PSLA: Improving Audio Event Classification with\nPretraining, Sampling, Labeling, and Aggregation\".\n\n#Appendix: the resolution and EfficientNet\nFinally, I show interesting result. It is relationship between EfficientNet and the resolution. The following result is public LB(5-folds average ensemble).\n\n|Resolution(W-H)|751-512|751-751|858-850|\n|---|---|---|---|\n|EfficientNetB0|0.893|0.904|**0.916**|\n|EfficientNetB3|**0.913**|0.912|0.900|\n\nIn B0, the higher resolution, the better result. But B3 was vice versa. Usually, the larger EfficientNet, the better at [high resolution](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/154683) it is. But the above is reverse. Why?\n\nMaybe **domain shift**(train: noisy sound -> test: clean sound) is concerned. B3 has learned about **train domain features**(noisy sound). On the other hand, B0 has less representational ability than B3. Therefore B0 has learned the **common features** of the train and test domain with high resolution. Without domain shift, B3 would have also shown good results with high resolution.", "votes": null }, { "id": "1207777", "postDate": "02/18/2021 03:18:49", "content": "Congrats on results and thanks for the writeup @shinmurashinmura
", "rawMarkdown": "Congrats on results and thanks for the writeup @shinmurashinmura", "votes": null }, { "id": "1208018", "postDate": "02/18/2021 05:52:26", "content": "Interesting that the resolution had that large of an impact. Do you think that it was actually a more detail representation to learn from or do you think maybe the imagenet weights are more tuned to have square images and features and not good when the features are quite small and spread out far across the time axis?
", "rawMarkdown": "Interesting that the resolution had that large of an impact. Do you think that it was actually a more detail representation to learn from or do you think maybe the imagenet weights are more tuned to have square images and features and not good when the features are quite small and spread out far across the time axis?", "votes": null }, { "id": "1208257", "postDate": "02/18/2021 08:18:20", "content": "Maybe small feature is more critical. In the paper[2](state-of-the-art in audioset), EfficientNet (pretrained by ImageNet) is used. But resolution is not square.
", "rawMarkdown": "Maybe small feature is more critical. In the paper[2](state-of-the-art in audioset), EfficientNet (pretrained by ImageNet) is used. But resolution is not square.", "votes": null }, { "id": "1208394", "postDate": "02/18/2021 09:29:50", "content": "Congratz !
\nYour sharings durign the competition helped a lot of people, so thanks a lot for that.
", "rawMarkdown": "Congratz ! \n\nYour sharings durign the competition helped a lot of people, so thanks a lot for that.", "votes": null }, { "id": "1215006", "postDate": "02/23/2021 09:03:42", "content": "Thanks @shinmurashinmura for the write up. I would like to ask when you use a large number of mel bands (850), but n_fft =1024, would you end up with some empty bands?
", "rawMarkdown": "Thanks @shinmurashinmura for the write up. I would like to ask when you use a large number of mel bands (850), but n_fft =1024, would you end up with some empty bands?", "votes": null }, { "id": "1216353", "postDate": "02/24/2021 08:16:43", "content": "Thank you for your comment.
\n\n\nwould you end up with some empty bands?
\n
Yes. In 850 mel bands, empty bands is exist. But 850 is better than 512 or 751.
\nIt is strange. I cannot understand this reason.
Congrats on results and thanks for the writeup @shinmurashinmura
", "votes": null, "replies": [] }, { "id": 1208018, "author_name": "ryches", "author_url": "", "post_date": "02/18/2021 05:52:26", "content": "Interesting that the resolution had that large of an impact. Do you think that it was actually a more detail representation to learn from or do you think maybe the imagenet weights are more tuned to have square images and features and not good when the features are quite small and spread out far across the time axis?
", "votes": null, "replies": [ { "id": 1208257, "author_name": "shinmurashinmura", "author_url": "", "post_date": "02/18/2021 08:18:20", "content": "Maybe small feature is more critical. In the paper[2](state-of-the-art in audioset), EfficientNet (pretrained by ImageNet) is used. But resolution is not square.
", "votes": null, "replies": [] } ] }, { "id": 1208394, "author_name": "theoviel", "author_url": "", "post_date": "02/18/2021 09:29:50", "content": "Congratz !
\nYour sharings durign the competition helped a lot of people, so thanks a lot for that.
", "votes": null, "replies": [] }, { "id": 1215006, "author_name": "thomeou", "author_url": "", "post_date": "02/23/2021 09:03:42", "content": "Thanks @shinmurashinmura for the write up. I would like to ask when you use a large number of mel bands (850), but n_fft =1024, would you end up with some empty bands?
", "votes": null, "replies": [ { "id": 1216353, "author_name": "shinmurashinmura", "author_url": "", "post_date": "02/24/2021 08:16:43", "content": "Thank you for your comment.
\n\n\nwould you end up with some empty bands?
\n
Yes. In 850 mel bands, empty bands is exist. But 850 is better than 512 or 751.
\nIt is strange. I cannot understand this reason.