![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4704212%2Fe4f92bf08c7e326c579321a67e449a8e%2Ffig.png?generation=1609853933525114&alt=media\")
My main approach is SED.
\nThe following methods worked well(LB score is 0.9 over).
But the following methods didn't work.
\nCOLA has an optimal batch size of 1024 in the paper. However, in my environment, the limit of batch size was 64 and therefore it did not work.
\nAlso, as many participants may have noticed, there are many missing labels in the tp file sources. I discovered the missing labels and retrained using them, but it did not improve.
\nNote: I showed missing label example in this topic.
", "messages": [ { "id": 1139171, "postDate": "2021-01-05T08:17:38.030Z", "content": "My main approach is SED.
\nThe following methods worked well(LB score is 0.9 over).
But the following methods didn't work.
\nCOLA has an optimal batch size of 1024 in the paper. However, in my environment, the limit of batch size was 64 and therefore it did not work.
\nAlso, as many participants may have noticed, there are many missing labels in the tp file sources. I discovered the missing labels and retrained using them, but it did not improve.
\nNote: I showed missing label example in this topic.
", "rawMarkdown": "My main approach is [SED](https://www.kaggle.com/hidehisaarai1213/introduction-to-sound-event-detection).\nThe following methods worked well(LB score is 0.9 over).\n+ PANNs SED architecture\n+ BCE Loss\n+ Random 10sec clip\n+ Using only tp file\n+ MixUp\n+ The base model is EfficientNet\n\nBut the following methods didn't work.\n+ Dice Loss\n+ SpecAugment\n+ Using **fp file** as background noise sound(For example, mix 0.9tp + 0.1fp)\n+ Using **fp file** as no label sound (all zero label)\n+ Self Supervised Learning ([COLA](https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/197805))\n+ Discover missing labels and retrain\n\nCOLA has an optimal batch size of 1024 in the paper. However, in my environment, the limit of batch size was 64 and therefore it did not work.\n\nAlso, as many participants may have noticed, there are many **missing labels** in the tp file sources. I discovered the missing labels and retrained using them, but it did not improve.\n\nNote: I showed missing label example in [this topic](https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/209040).", "votes": 101 }, { "id": 1161714, "postDate": "2021-01-20T17:50:54.753Z", "content": "\n\nCOLA has an optimal batch size of 1024 in the paper. However, in my environment, the limit of batch size was 64 and therefore it did not work.
\n
If you use pytorch then you can simulate larger batch size by one of these, or by combining them:
\nVery thanks.
\nGradient accumulation is good. It looks like you could theoretically make the batch size infinite. I'd like to try it.
", "rawMarkdown": "Very thanks.\n\nGradient accumulation is good. It looks like you could theoretically make the batch size infinite. I'd like to try it.", "votes": 1 }, { "id": 1162318, "postDate": "2021-01-21T05:20:25.480Z", "content": "You may also try Cross-Batch Memory. Gradient accumulation has its own limitations (e.g. doesn't work on BN layers; doesn't work on contrastive/triplet loss without modifications)
", "rawMarkdown": "You may also try [Cross-Batch Memory](https://github.com/MalongTech/research-xbm). Gradient accumulation has its own limitations (e.g. doesn't work on BN layers; doesn't work on contrastive/triplet loss without modifications)", "votes": 3 }, { "id": 1162557, "postDate": "2021-01-21T08:15:19.923Z", "content": "\n\ndoesn't work on BN layers
\n
That's true! If one batch size is too small, Gradient accumulation might not work. That was helpful. Thank you.
", "rawMarkdown": "> doesn't work on BN layers\n\nThat's true! If one batch size is too small, Gradient accumulation might not work. That was helpful. Thank you.", "votes": 1 }, { "id": 1178202, "postDate": "2021-01-30T18:04:45.427Z", "content": "\nWere you able to try out COLA with larger batch size ? Was it helpful ?
", "rawMarkdown": "@shinmurashinmura \n\nWere you able to try out COLA with larger batch size ? Was it helpful ? " }, { "id": 1178265, "postDate": "2021-01-30T19:07:27.277Z", "content": "i have tried COLA with gradient accumalation but not successful. The test accuracy of COLA was around 93%.
", "rawMarkdown": "i have tried COLA with gradient accumalation but not successful. The test accuracy of COLA was around 93%.", "votes": 2 }, { "id": 1178322, "postDate": "2021-01-30T19:35:56.363Z", "content": "\"test accuracy of COLA was around 93%\". You mean multi-label accuracy not the LWLRAP metric right ? What LWLRAP score did you get ?
", "rawMarkdown": "\"test accuracy of COLA was around 93%\". You mean multi-label accuracy not the LWLRAP metric right ? What LWLRAP score did you get ? " }, { "id": 1178339, "postDate": "2021-01-30T19:54:43.707Z", "content": "If you train COLA, it reports a test accuracy. With bs=256, gradient_accum=8, epochs=100 and max_length=300, the final accuracy of COLA was around 94% (shorter max_length -> higher test accuracy up to 99%). After that I have trained a classifier on 6 second segments: 89% LWLRAP, full clip LWLRAP I don't remember. But the model was really bad on LB.
", "rawMarkdown": "If you train COLA, it reports a test accuracy. With bs=256, gradient_accum=8, epochs=100 and max_length=300, the final accuracy of COLA was around 94% (shorter max_length -> higher test accuracy up to 99%). After that I have trained a classifier on 6 second segments: 89% LWLRAP, full clip LWLRAP I don't remember. But the model was really bad on LB.", "votes": 2 }, { "id": 1178459, "postDate": "2021-01-30T22:17:48.500Z", "content": "I don't think COLA works with gradient accumulation because of the nature of contrastive loss. But CrossBatchMemory was proven to be useful in MoCo-style self-supervision as stated in this repo
\n(cannot post attachment right now, just ctrl+f and search for key words Using loss functions for unsupervised / self-supervised learning
", "rawMarkdown": "I don't think COLA works with gradient accumulation because of the nature of contrastive loss. But CrossBatchMemory was proven to be useful in MoCo-style self-supervision as stated in this [repo](https://github.com/KevinMusgrave/pytorch-metric-learning/tree/master) \n\n(cannot post attachment right now, just ctrl+f and search for key words **Using loss functions for unsupervised / self-supervised learning**" }, { "id": 1178582, "postDate": "2021-01-31T01:53:45.467Z", "content": "Thanks for sharing the details.
\nI just skim through the COLA but here is some preliminary thoughts on potentially tailoring it for our problem set.
\nThe original paper assumes same audio clip --> Similarity and different audio clip --> Dissimilarity. In our competition, most part of audio clip this similar/dissimilar relation do not exist ( except for a few tiny window where birds call).
\nMaybe the naive approach ( like proposed in the paper) of drawing positive and negative segment from a batch of audio is not ideal because the signal ( similarity / dissimialrity is just too sparse )
\nI am thinking along this line of customize it: ( I have not read the paper or the code in that much detail, so maybe my thought is incorrect )
\nFor bird specie X, we construct audio positive clips by randomly merging 10sec segments where X is calling ( tp ) and construct negative clips by merging 10 sec segments where X is not calling (fp). And then somehow modify the batching and sampling procedure to force the model to learn dissimilarity between tp and fp fro specie X. And for different batch, we could pass through these positive / negative pair for different specie.
\nWhat do you guys think ? @jihangz @tugstugi
", "rawMarkdown": "Thanks for sharing the details.\n\nI just skim through the COLA but here is some preliminary thoughts on potentially tailoring it for our problem set.\n\nThe original paper assumes same audio clip --> Similarity and different audio clip --> Dissimilarity. In our competition, most part of audio clip this similar/dissimilar relation do not exist ( except for a few tiny window where birds call). \n\nMaybe the naive approach ( like proposed in the paper) of drawing positive and negative segment from a batch of audio is not ideal because the signal ( similarity / dissimialrity is just too sparse )\n\nI am thinking along this line of customize it: ( I have not read the paper or the code in that much detail, so maybe my thought is incorrect ) \n\nFor bird specie X, we construct audio positive clips by randomly merging 10sec segments where X is calling ( tp ) and construct negative clips by merging 10 sec segments where X is not calling (fp). And then somehow modify the batching and sampling procedure to force the model to learn dissimilarity between tp and fp fro specie X. And for different batch, we could pass through these positive / negative pair for different specie.\n\n\nWhat do you guys think ? @jihangz @tugstugi " }, { "id": 1178622, "postDate": "2021-01-31T03:02:55.680Z", "content": "@jihangz I have tried also smaller max_length with bigger batch sizes on Quadro 48GB, but still no luck.
\n@garfieldchh I am doubting that the audios with TP and FP are enough.
Thanks for sharing!!! What are your validation strategies and metrics ? Do your CV results correlate with your LB score?
", "rawMarkdown": "Thanks for sharing!!! What are your validation strategies and metrics ? Do your CV results correlate with your LB score?", "votes": 3, "replies": [ { "id": 1140471, "postDate": "2021-01-06T03:06:36.860Z", "content": "CV and LB were roughly correlated, but I didn't trust CV because maybe there is a domain shift.
", "rawMarkdown": "CV and LB were roughly correlated, but I didn't trust CV because maybe there is a domain shift.", "votes": 3 }, { "id": 1140616, "postDate": "2021-01-06T06:06:09.417Z", "content": "Thanks for the info. If you dont trust CV results, could you give some hint on how you select models for LB? I'm using LWLRAP in 10-second crop windows for CV but the LB results sometimes seems very random and not well correlated with my CV results. I also try CV using LWLRAP on full clip but it does not improve the correlation.
", "rawMarkdown": "Thanks for the info. If you dont trust CV results, could you give some hint on how you select models for LB? I'm using LWLRAP in 10-second crop windows for CV but the LB results sometimes seems very random and not well correlated with my CV results. I also try CV using LWLRAP on full clip but it does not improve the correlation.", "votes": 1 }, { "id": 1140723, "postDate": "2021-01-06T08:16:21.540Z", "content": "Do you use SED with weak label?
", "rawMarkdown": "Do you use SED with weak label? " }, { "id": 1140814, "postDate": "2021-01-06T09:38:03.103Z", "content": "Yes. I'm using SED (either resnet or efficient net as base model) and weak label.
", "rawMarkdown": "Yes. I'm using SED (either resnet or efficient net as base model) and weak label." }, { "id": 1141063, "postDate": "2021-01-06T13:15:10.070Z", "content": "OK. Why do CV and LBs different? I think three reasons. However, it is a long text and I want to write a new topic later. Please wait.
", "rawMarkdown": "OK. Why do CV and LBs different? I think three reasons. However, it is a long text and I want to write a new topic later. Please wait.", "votes": 3 } ] }, { "id": 1139533, "postDate": "2021-01-05T13:15:35.620Z", "content": "its interesting to see MIXUP improve the soce. for me, the mixup code provided in the PANN was not improving my score.
\nI also found with a larger base model(resnext50) the performance reduced.
\npsudolabeling for the missing label did not help me too.
\nlonger training did help me after correcting the learning rate.
Thanks for sharing. I found that a larger base model had bad performance too.
\nEfficientNetB0 and MobileNetV2 was good performance.
| \n | LB | \n
|---|---|
| EfficientNetB0 | \n0.893 | \n
| MobileNetV2 | \n0.893 | \n
| ResNest50 | \n0.868 | \n
@shinmurashinmura Are these results from single models or k-folds?
", "rawMarkdown": "@shinmurashinmura Are these results from single models or k-folds?", "votes": 1 }, { "id": 1142236, "postDate": "2021-01-07T08:45:42.767Z", "content": "It's 5folds ensemble result. A single model is down about 0.05 in LB.
", "rawMarkdown": "It's 5folds ensemble result. A single model is down about 0.05 in LB.", "votes": 4 }, { "id": 1143257, "postDate": "2021-01-07T20:11:38.427Z", "content": "@shinmurashinmura thanks for sharing it!
\nI am the beginner and little bit confused on how you ensemble 5 fold models in SED. Is it averaging the framewise outputs or something else?
I used framewise_outputs for prediction.
And I made submission per a fold(5times).
\nFinally, I averaged these submissions.
Thank you so much. One more thing…
\nIs it better to keep PERIOD=30 or just use all the data once? which is leading to how can I get a score from several chunks predicted on one record_id?
I experimented period10 vs period30. The winner was period30. I don't recommend long period.
\nBecause the longer period it takes, the more missing labels it contains.
Thank you for the answer!
\nHow do you aggregate predictions during the inference time.
\nFor example,
\nPERIOD=30;
\nprediction shape of 000316da7.flac is (3, 24) How should I reduce it to (1, 24)
@yuvaramsingh Is there something wrong with the mixup approach in PANN ? they are mixing two spectrogram with a different coefficient right ? Isn't it what the mixup paper presented ?
", "rawMarkdown": "@yuvaramsingh Is there something wrong with the mixup approach in PANN ? they are mixing two spectrogram with a different coefficient right ? Isn't it what the mixup paper presented ?" }, { "id": 1149630, "postDate": "2021-01-12T03:02:49.783Z", "content": "@bayartsogtya
\nMy prediction is np.max((3,24), axis=time)
@ludovick
\nI used MixUp with wave style. I random select 10sec clip(A(4800010) and B(4800010)).
\nAnd I mix A with B. For example 0.9A+0.1B. Finally, label is mixed.
@shinmurashinmura, thanks for this.
\nso for each class, you are getting frame predictions in time..
\nand you get max of these frame predictions.
\nWhat exactly is 'a frame prediction'. Is it a 10 sec of each 60sec recording in sequence?
\nSo 6 frame predictions per record in sequence?
\n(sorry for my newbie question)
I used 10sec clip weak label training. And I split 60sec test clip per 10 sec.
\nAnd I get mini prediction(max(framewise_output, axis=time)) per 10 sec.
\nFinal prediction is max of mini prediction(=np.max((6,24), axis=0)).
@ludovick Hi! I was wondering what do you mean by \"they are mixing two spectrogram with a different coefficient\"? Thanks!
", "rawMarkdown": "@ludovick Hi! I was wondering what do you mean by \"they are mixing two spectrogram with a different coefficient\"? Thanks! " }, { "id": 1154694, "postDate": "2021-01-15T19:41:02.320Z", "content": "@tonychenxyz Hi, the idea of mixup is to combine two images such as new_image = alpha * image_A + (1-alpha) * image_B. (A and B are two different images from your training data) The labels are then also mixed : new_label = alpha * label_A + (1-alpha) * label_B . In our competition, we can mix raw signal or spectrogram. For now I saw some drop using mixup on spectrogram, and weird results with mixup on raw signal.
@shinmurashinmura Thanks for sharing this. It is very useful info. Do we have any special pre-processing for using MobileNetV2? My CV dips by half if I switch from Resnet to Mobilenet. This is my first shot at CNN problems…just wondering if I am making rookie mistake somewhere. I have resized to 224, 224 which is what the model expects and done the preprocessing returned by MobileNetV2. Any guidance or suggestions?
", "rawMarkdown": "@shinmurashinmura Thanks for sharing this. It is very useful info. Do we have any special pre-processing for using MobileNetV2? My CV dips by half if I switch from Resnet to Mobilenet. This is my first shot at CNN problems...just wondering if I am making rookie mistake somewhere. I have resized to 224, 224 which is what the model expects and done the preprocessing returned by MobileNetV2. Any guidance or suggestions?" } ] }, { "id": 1154880, "postDate": "2021-01-16T02:36:46.313Z", "content": "@shinmura0
\nThanks for sharing the approach. I have 2 questions.
You mentioned that training instance is obtained from \"Random 10sec clip(contain tp sound)\". I understand that random crop is obtained by some random shift of the tmin tmax window given in the training data, is this correct ? If yes, do you generate multiple training instances (multiple random crops with different offsets ) from a single audio file ? Also, do you mind sharing random cropping parameters ( IE: the range of possible offset from the ground truth window).
\nI am sorry if these data preprocessing questions sound very basic. I am new to audio competition and SED in general .
I am a bit confused about how long a testing segment is. Are we using 10 sec or 30 sec testing segments ?
\nQuoting discussion:
\n\"I experimented period10 vs period30. The winner was period30.\"
\n\"And I split 60sec test clip per 10 sec.\"
\nIf we are not using 10 seconds long test segment, what is the reason for training on 1 segment length and testing on another segment length ? The SED notebook you referenced seems to be doing different training / testing segment length too ( 5 period training segment and 30 period testing segment ). I am wondering what is the rationale for doing this ?
\n\nI understand that random crop is obtained by some random shift of the tmin tmax window given in the training data, is this correct ?
\n
Yes.
\n\n\nIf yes, do you generate multiple training instances (multiple random crops with different offsets ) from a single audio file ?
\n
For example,
\ntp segment: 11sec ~ 13sec,
\nTraining clip is following.
Cases 1 to 3 are all possible. And more diverse training clips are also possible. It is random.
\n\n\nAre we using 10 sec or 30 sec testing segments ?
\n
I used 10 sec testing segments.
\n\n\nThe SED notebook you referenced seems to be doing different training
\n
Clip time can be changed between training and test. However, since it changes the prediction a bit, we experimented with it(test clip:10sec vs 30sec).
", "rawMarkdown": "> I understand that random crop is obtained by some random shift of the tmin tmax window given in the training data, is this correct ?\n\nYes.\n\n> If yes, do you generate multiple training instances (multiple random crops with different offsets ) from a single audio file ? \n\nFor example,\ntp segment: 11sec ~ 13sec,\nTraining clip is following.\n+ case1: 3sec ~ 13sec\n+ case2: 5sec ~ 15sec\n+ case3: 11sec ~ 21sec\n\nCases 1 to 3 are all possible. And more diverse training clips are also possible. It is random.\n\n> Are we using 10 sec or 30 sec testing segments ?\n\nI used 10 sec testing segments.\n\n> The SED notebook you referenced seems to be doing different training\n\nClip time can be changed between training and test. However, since it changes the prediction a bit, we experimented with it(test clip:10sec vs 30sec).", "votes": 4 }, { "id": 1159984, "postDate": "2021-01-19T15:39:18.177Z", "content": "@shinmurashinmura Got it. Thanks for the detailed answers. Really appreciate it
", "rawMarkdown": "@shinmurashinmura Got it. Thanks for the detailed answers. Really appreciate it " } ] }, { "id": 1140375, "postDate": "2021-01-06T00:55:07.680Z", "content": "Hi
\nI use the SED too. But it is not stable, could you tell me what wrong in my notebook?
https://www.kaggle.com/truonghoang/rfcx-pannscnn14att-train-inference
\nNote: I used the Multi-label + LWLRAP
\nThanks.
Do you use pannscnn14? In my experiment, pannscnn14 had LB:0.762(single model).
\nThis is not strong model. EfficientNet is recommended.
EfficientB0 is good for me too.
\nbut only in train pipeline. When i submit it, it's so worse.
Why do CV and LBs different? I think three reasons. However, it is a long text and I want to write a new topic later. Please wait.
", "rawMarkdown": "Why do CV and LBs different? I think three reasons. However, it is a long text and I want to write a new topic later. Please wait.", "votes": 3 }, { "id": 1147676, "postDate": "2021-01-10T16:35:02.750Z", "content": "same as @doanquanvietnamca I did not suceed to have results as high as yours with SED model with Eff architecture
", "rawMarkdown": "same as @doanquanvietnamca I did not suceed to have results as high as yours with SED model with Eff architecture" } ] }, { "id": 1139401, "postDate": "2021-01-05T11:14:31.723Z", "content": "So SED is once again showing great results, interesting !
\n(I say once again because it won the last audio competition)
", "rawMarkdown": "So SED is once again showing great results, interesting !\n\n(I say once again because it won the last audio competition)", "votes": 1, "replies": [ { "id": 1139522, "postDate": "2021-01-05T13:00:04.570Z", "content": "It's Cornell Birdcall Identification!
\nYou are excellent performance in that competition.
I think SED is a strong method in multi label task.Cornell Birdcall Identification and this competition is multi label task. Therefore SED is good performance.
", "rawMarkdown": "It's Cornell Birdcall Identification!\nYou are excellent performance in that competition.\n\nI think SED is a strong method in **multi label task.**Cornell Birdcall Identification and this competition is multi label task. Therefore SED is good performance.", "votes": 1 } ] }, { "id": 1143495, "postDate": "2021-01-07T23:40:26.367Z", "content": "@shinmurashinmura Thanks you for sharing . Have you try to use the clipwise output ? if it's the case, how good they are compared to segmentwise output ?
", "rawMarkdown": "@shinmurashinmura Thanks you for sharing . Have you try to use the clipwise output ? if it's the case, how good they are compared to segmentwise output ?", "votes": 2, "replies": [ { "id": 1143721, "postDate": "2021-01-08T03:03:50.423Z", "content": "I don't try clipwise_output prediction. Sorry.
Does anyone know of a Tensorflow Notebook that implements SED? All the examples I've seen are in PyTorch.
", "rawMarkdown": "Does anyone know of a Tensorflow Notebook that implements SED? All the examples I've seen are in PyTorch.", "votes": 2 }, { "id": 1140569, "postDate": "2021-01-06T05:38:04.827Z", "content": "Tnank you for your great discussion!
\nI have a question.
\nDo you have any reason that you don't use strong label when training SED.
\nIntuitively, I think the score will improve more, but do you have any insights?
The reason for not using strong label is that missing labels exist.
\nAs shown here, there are missing labels in many sound sources.
If I get random 10sec clip, It might contain missing labels. Robust training(for missing labels) is possible by using weak labels.However, I have not experimented with strong label.it may be better result with using strong label.
", "rawMarkdown": "The reason for not using strong label is that missing labels exist.\nAs shown [here](https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/209040), there are missing labels in many sound sources.\n\nIf I get random 10sec clip, It might contain missing labels. Robust training(for missing labels) is possible by using weak labels.However, I have not experimented with strong label.it may be better result with using strong label.", "votes": 4 }, { "id": 1140801, "postDate": "2021-01-06T09:24:38.450Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1140800, "postDate": "2021-01-06T09:24:38.450Z", "content": "Tnank you for your answering!
\nIt is very convincing.
Thanks for kindly sharing your approach in post and your replies really helped. I am waiting for your topic on \"How to use SED\".
", "rawMarkdown": "Thanks for kindly sharing your approach in post and your replies really helped. I am waiting for your topic on \"How to use SED\". ", "votes": 1 }, { "id": 1176821, "postDate": "2021-01-29T20:10:50.770Z", "content": "Thanks! I'm joining this competition now, hope it wont be too late :)
", "rawMarkdown": "Thanks! I'm joining this competition now, hope it wont be too late :)" }, { "id": 1168300, "postDate": "2021-01-24T20:59:55.140Z", "content": "@shinmurashinmura nice write-up. I have a question about the strong labels. How do you define them? From what I understand, lets say you take 0-10 sec segment of the whole clip, and there an event at 1-2 sec and the cnn returns a B,10,C tensor after avg the frequency dimension, do you just say the second segment has an event as a binary label?
", "rawMarkdown": "@shinmurashinmura nice write-up. I have a question about the strong labels. How do you define them? From what I understand, lets say you take 0-10 sec segment of the whole clip, and there an event at 1-2 sec and the cnn returns a B,10,C tensor after avg the frequency dimension, do you just say the second segment has an event as a binary label? ", "replies": [ { "id": 1170158, "postDate": "2021-01-26T03:21:23.017Z", "content": "\n\ndo you just say the second segment has an event as a binary label?
\n
Yes. Binary label(second segment) is needed.
\nhttps://www.kaggle.com/c/rfcx-species-audio-detection/discussion/211007#1152329
I see. You mentioned you're using clip labels in the other post; from what I understand clip labels give you the best results? I also tried using frame labels (labeling each column after interpolate), but results are worse
", "rawMarkdown": "I see. You mentioned you're using clip labels in the other post; from what I understand clip labels give you the best results? I also tried using frame labels (labeling each column after interpolate), but results are worse", "votes": 1 }, { "id": 1173669, "postDate": "2021-01-28T03:34:37.917Z", "content": "I tried clipwise vs framewise for prediction. Clipwise have improved a bit. You are right.
\nI also will use clipwise for prediction. Thank you.
Interesting to see SED working so well!
", "rawMarkdown": "Interesting to see SED working so well!" }, { "id": 1140878, "postDate": "2021-01-06T10:39:10.847Z", "content": "I'm new to this compeition after the NFL has just finished, you share a lot on the disscusion which is great for a starter.
\nMaybe you could try to treat FP label as TP label, but in a weak prob way?
Hi, what are weak labels? What does it mean to work with weak labels?
", "rawMarkdown": "Hi, what are weak labels? What does it mean to work with weak labels?", "replies": [ { "id": 1141029, "postDate": "2021-01-06T12:46:54.547Z", "content": "What is weak label? Let's check link.
\nhttps://www.kaggle.com/hidehisaarai1213/introduction-to-sound-event-detection
And \"Train SED model with only weak supervision\" help you.
", "rawMarkdown": "What is weak label? Let's check link.\nhttps://www.kaggle.com/hidehisaarai1213/introduction-to-sound-event-detection\n\nAnd \"Train SED model with only weak supervision\" help you.", "votes": 1 }, { "id": 1141074, "postDate": "2021-01-06T13:24:20.450Z", "content": "Thank you, I read that. It's just that in this problem, in both approaches, we cut a segment out of the audio and claim there is a target class. I don't understand what the difference is? Or is the weak label just the name of the SED approach?
", "rawMarkdown": "Thank you, I read that. It's just that in this problem, in both approaches, we cut a segment out of the audio and claim there is a target class. I don't understand what the difference is? Or is the weak label just the name of the SED approach? " }, { "id": 1142232, "postDate": "2021-01-07T08:42:05.230Z", "content": "Now, I'm writing new topic \"How to use SED\". Please wait.
", "rawMarkdown": "Now, I'm writing new topic \"How to use SED\". Please wait.", "votes": 1 } ] }, { "id": 1139338, "postDate": "2021-01-05T10:27:08.297Z", "content": "Thanks for sharing! Is your SED using strong supervision and producing clipwise output as final prediction?
", "rawMarkdown": "Thanks for sharing! Is your SED using strong supervision and producing clipwise output as final prediction?", "replies": [ { "id": 1139552, "postDate": "2021-01-05T13:31:51.947Z", "content": "My SED training is
\nFinal prediction is
\n
the visualization looks good . i did not think about viz frame prediction as a 2d image. this will help me understand my prediction even better . thanks
", "rawMarkdown": "the visualization looks good . i did not think about viz frame prediction as a 2d image. this will help me understand my prediction even better . thanks " }, { "id": 1141293, "postDate": "2021-01-06T15:54:37.770Z", "content": "What image size did you use?
", "rawMarkdown": "What image size did you use?" }, { "id": 1144492, "postDate": "2021-01-08T13:41:23.640Z", "content": "@shinmurashinmura Thanks for the clear graphical explaination. My understanding is that you train with clip-wise label and conduct inference with frame-wise label. Is my understanding correct?
", "rawMarkdown": "@shinmurashinmura Thanks for the clear graphical explaination. My understanding is that you train with clip-wise label and conduct inference with frame-wise label. Is my understanding correct?" }, { "id": 1149631, "postDate": "2021-01-12T03:03:15.690Z", "content": "\n\n\nMy understanding is that you train with clip-wise label and conduct inference with frame-wise label.
\n
Yes, I used this style. And for your reference (3. A gap training and prediction)
", "rawMarkdown": "@tonychenxyz \n\n> My understanding is that you train with clip-wise label and conduct inference with frame-wise label.\n\nYes, I used this style. And for your [reference](https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/209684) (3. A gap training and prediction)", "votes": 1 }, { "id": 1149973, "postDate": "2021-01-12T09:17:07.373Z", "content": "Thanks for clarifying!
", "rawMarkdown": "Thanks for clarifying!" } ] }, { "id": 1139224, "postDate": "2021-01-05T08:59:24.270Z", "content": "Using SED? Interesting…
\nHow high is your single model score?
I forget single model score.
\nBut EfficientNetB3(5fold ensemble) had 0.913 score.
I tried replicating this but failed probably missing a processing step. Do you mind @shinmurashinmura sharing your notebook that got you 0.913? Thanks in advance. :)
", "rawMarkdown": "I tried replicating this but failed probably missing a processing step. Do you mind @shinmurashinmura sharing your notebook that got you 0.913? Thanks in advance. :)" } ] }, { "id": 1143330, "postDate": "2021-01-07T21:04:22.937Z", "content": "Thank you!
", "rawMarkdown": "Thank you!" } ], "comments": [ { "id": 1161714, "author_name": "CPMP", "author_url": "", "post_date": "2021-01-20T17:50:54.753000", "content": "\n\nCOLA has an optimal batch size of 1024 in the paper. However, in my environment, the limit of batch size was 64 and therefore it did not work.
\n
If you use pytorch then you can simulate larger batch size by one of these, or by combining them:
\nVery thanks.
\nGradient accumulation is good. It looks like you could theoretically make the batch size infinite. I'd like to try it.
", "votes": 1, "replies": [] }, { "id": 1162318, "author_name": "Darkate", "author_url": "", "post_date": "2021-01-21T05:20:25.480000", "content": "You may also try Cross-Batch Memory. Gradient accumulation has its own limitations (e.g. doesn't work on BN layers; doesn't work on contrastive/triplet loss without modifications)
", "votes": 3, "replies": [] }, { "id": 1162557, "author_name": "shinmura0", "author_url": "", "post_date": "2021-01-21T08:15:19.923000", "content": "\n\ndoesn't work on BN layers
\n
That's true! If one batch size is too small, Gradient accumulation might not work. That was helpful. Thank you.
", "votes": 1, "replies": [] }, { "id": 1178202, "author_name": "NakedKoala", "author_url": "", "post_date": "2021-01-30T18:04:45.427000", "content": "\nWere you able to try out COLA with larger batch size ? Was it helpful ?
", "votes": 0, "replies": [] }, { "id": 1178265, "author_name": "tugstugi", "author_url": "", "post_date": "2021-01-30T19:07:27.277000", "content": "i have tried COLA with gradient accumalation but not successful. The test accuracy of COLA was around 93%.
", "votes": 2, "replies": [] }, { "id": 1178322, "author_name": "NakedKoala", "author_url": "", "post_date": "2021-01-30T19:35:56.363000", "content": "\"test accuracy of COLA was around 93%\". You mean multi-label accuracy not the LWLRAP metric right ? What LWLRAP score did you get ?
", "votes": 0, "replies": [] }, { "id": 1178339, "author_name": "tugstugi", "author_url": "", "post_date": "2021-01-30T19:54:43.707000", "content": "If you train COLA, it reports a test accuracy. With bs=256, gradient_accum=8, epochs=100 and max_length=300, the final accuracy of COLA was around 94% (shorter max_length -> higher test accuracy up to 99%). After that I have trained a classifier on 6 second segments: 89% LWLRAP, full clip LWLRAP I don't remember. But the model was really bad on LB.
", "votes": 2, "replies": [] }, { "id": 1178459, "author_name": "Darkate", "author_url": "", "post_date": "2021-01-30T22:17:48.500000", "content": "I don't think COLA works with gradient accumulation because of the nature of contrastive loss. But CrossBatchMemory was proven to be useful in MoCo-style self-supervision as stated in this repo
\n(cannot post attachment right now, just ctrl+f and search for key words Using loss functions for unsupervised / self-supervised learning
", "votes": 0, "replies": [] }, { "id": 1178582, "author_name": "NakedKoala", "author_url": "", "post_date": "2021-01-31T01:53:45.467000", "content": "Thanks for sharing the details.
\nI just skim through the COLA but here is some preliminary thoughts on potentially tailoring it for our problem set.
\nThe original paper assumes same audio clip --> Similarity and different audio clip --> Dissimilarity. In our competition, most part of audio clip this similar/dissimilar relation do not exist ( except for a few tiny window where birds call).
\nMaybe the naive approach ( like proposed in the paper) of drawing positive and negative segment from a batch of audio is not ideal because the signal ( similarity / dissimialrity is just too sparse )
\nI am thinking along this line of customize it: ( I have not read the paper or the code in that much detail, so maybe my thought is incorrect )
\nFor bird specie X, we construct audio positive clips by randomly merging 10sec segments where X is calling ( tp ) and construct negative clips by merging 10 sec segments where X is not calling (fp). And then somehow modify the batching and sampling procedure to force the model to learn dissimilarity between tp and fp fro specie X. And for different batch, we could pass through these positive / negative pair for different specie.
\nWhat do you guys think ? @jihangz @tugstugi
", "votes": 0, "replies": [] }, { "id": 1178622, "author_name": "tugstugi", "author_url": "", "post_date": "2021-01-31T03:02:55.680000", "content": "@jihangz I have tried also smaller max_length with bigger batch sizes on Quadro 48GB, but still no luck.
\n@garfieldchh I am doubting that the audios with TP and FP are enough.
Thanks for sharing!!! What are your validation strategies and metrics ? Do your CV results correlate with your LB score?
", "votes": 3, "replies": [ { "id": 1140471, "author_name": "shinmura0", "author_url": "", "post_date": "2021-01-06T03:06:36.860000", "content": "CV and LB were roughly correlated, but I didn't trust CV because maybe there is a domain shift.
", "votes": 3, "replies": [] }, { "id": 1140616, "author_name": "mangeption", "author_url": "", "post_date": "2021-01-06T06:06:09.417000", "content": "Thanks for the info. If you dont trust CV results, could you give some hint on how you select models for LB? I'm using LWLRAP in 10-second crop windows for CV but the LB results sometimes seems very random and not well correlated with my CV results. I also try CV using LWLRAP on full clip but it does not improve the correlation.
", "votes": 1, "replies": [] }, { "id": 1140723, "author_name": "shinmura0", "author_url": "", "post_date": "2021-01-06T08:16:21.540000", "content": "Do you use SED with weak label?
", "votes": 0, "replies": [] }, { "id": 1140814, "author_name": "mangeption", "author_url": "", "post_date": "2021-01-06T09:38:03.103000", "content": "Yes. I'm using SED (either resnet or efficient net as base model) and weak label.
", "votes": 0, "replies": [] }, { "id": 1141063, "author_name": "shinmura0", "author_url": "", "post_date": "2021-01-06T13:15:10.070000", "content": "OK. Why do CV and LBs different? I think three reasons. However, it is a long text and I want to write a new topic later. Please wait.
", "votes": 3, "replies": [] } ] }, { "id": 1139533, "author_name": "yuvaramsingh", "author_url": "", "post_date": "2021-01-05T13:15:35.620000", "content": "its interesting to see MIXUP improve the soce. for me, the mixup code provided in the PANN was not improving my score.
\nI also found with a larger base model(resnext50) the performance reduced.
\npsudolabeling for the missing label did not help me too.
\nlonger training did help me after correcting the learning rate.
Thanks for sharing. I found that a larger base model had bad performance too.
\nEfficientNetB0 and MobileNetV2 was good performance.
| \n | LB | \n
|---|---|
| EfficientNetB0 | \n0.893 | \n
| MobileNetV2 | \n0.893 | \n
| ResNest50 | \n0.868 | \n
@shinmurashinmura Are these results from single models or k-folds?
", "votes": 1, "replies": [] }, { "id": 1142236, "author_name": "shinmura0", "author_url": "", "post_date": "2021-01-07T08:45:42.767000", "content": "It's 5folds ensemble result. A single model is down about 0.05 in LB.
", "votes": 4, "replies": [] }, { "id": 1143257, "author_name": "Bayartsogt Yadamsuren", "author_url": "", "post_date": "2021-01-07T20:11:38.427000", "content": "@shinmurashinmura thanks for sharing it!
\nI am the beginner and little bit confused on how you ensemble 5 fold models in SED. Is it averaging the framewise outputs or something else?
I used framewise_outputs for prediction.
And I made submission per a fold(5times).
\nFinally, I averaged these submissions.
Thank you so much. One more thing…
\nIs it better to keep PERIOD=30 or just use all the data once? which is leading to how can I get a score from several chunks predicted on one record_id?
I experimented period10 vs period30. The winner was period30. I don't recommend long period.
\nBecause the longer period it takes, the more missing labels it contains.
Thank you for the answer!
\nHow do you aggregate predictions during the inference time.
\nFor example,
\nPERIOD=30;
\nprediction shape of 000316da7.flac is (3, 24) How should I reduce it to (1, 24)
@yuvaramsingh Is there something wrong with the mixup approach in PANN ? they are mixing two spectrogram with a different coefficient right ? Isn't it what the mixup paper presented ?
", "votes": 0, "replies": [] }, { "id": 1149630, "author_name": "shinmura0", "author_url": "", "post_date": "2021-01-12T03:02:49.783000", "content": "@bayartsogtya
\nMy prediction is np.max((3,24), axis=time)
@ludovick
\nI used MixUp with wave style. I random select 10sec clip(A(4800010) and B(4800010)).
\nAnd I mix A with B. For example 0.9A+0.1B. Finally, label is mixed.
@shinmurashinmura, thanks for this.
\nso for each class, you are getting frame predictions in time..
\nand you get max of these frame predictions.
\nWhat exactly is 'a frame prediction'. Is it a 10 sec of each 60sec recording in sequence?
\nSo 6 frame predictions per record in sequence?
\n(sorry for my newbie question)
I used 10sec clip weak label training. And I split 60sec test clip per 10 sec.
\nAnd I get mini prediction(max(framewise_output, axis=time)) per 10 sec.
\nFinal prediction is max of mini prediction(=np.max((6,24), axis=0)).
@ludovick Hi! I was wondering what do you mean by \"they are mixing two spectrogram with a different coefficient\"? Thanks!
", "votes": 0, "replies": [] }, { "id": 1154694, "author_name": "Shiro", "author_url": "", "post_date": "2021-01-15T19:41:02.320000", "content": "@tonychenxyz Hi, the idea of mixup is to combine two images such as new_image = alpha * image_A + (1-alpha) * image_B. (A and B are two different images from your training data) The labels are then also mixed : new_label = alpha * label_A + (1-alpha) * label_B . In our competition, we can mix raw signal or spectrogram. For now I saw some drop using mixup on spectrogram, and weird results with mixup on raw signal.
@shinmurashinmura Thanks for sharing this. It is very useful info. Do we have any special pre-processing for using MobileNetV2? My CV dips by half if I switch from Resnet to Mobilenet. This is my first shot at CNN problems…just wondering if I am making rookie mistake somewhere. I have resized to 224, 224 which is what the model expects and done the preprocessing returned by MobileNetV2. Any guidance or suggestions?
", "votes": 0, "replies": [] } ] }, { "id": 1154880, "author_name": "NakedKoala", "author_url": "", "post_date": "2021-01-16T02:36:46.313000", "content": "@shinmura0
\nThanks for sharing the approach. I have 2 questions.
You mentioned that training instance is obtained from \"Random 10sec clip(contain tp sound)\". I understand that random crop is obtained by some random shift of the tmin tmax window given in the training data, is this correct ? If yes, do you generate multiple training instances (multiple random crops with different offsets ) from a single audio file ? Also, do you mind sharing random cropping parameters ( IE: the range of possible offset from the ground truth window).
\nI am sorry if these data preprocessing questions sound very basic. I am new to audio competition and SED in general .
I am a bit confused about how long a testing segment is. Are we using 10 sec or 30 sec testing segments ?
\nQuoting discussion:
\n\"I experimented period10 vs period30. The winner was period30.\"
\n\"And I split 60sec test clip per 10 sec.\"
\nIf we are not using 10 seconds long test segment, what is the reason for training on 1 segment length and testing on another segment length ? The SED notebook you referenced seems to be doing different training / testing segment length too ( 5 period training segment and 30 period testing segment ). I am wondering what is the rationale for doing this ?
\n\nI understand that random crop is obtained by some random shift of the tmin tmax window given in the training data, is this correct ?
\n
Yes.
\n\n\nIf yes, do you generate multiple training instances (multiple random crops with different offsets ) from a single audio file ?
\n
For example,
\ntp segment: 11sec ~ 13sec,
\nTraining clip is following.
Cases 1 to 3 are all possible. And more diverse training clips are also possible. It is random.
\n\n\nAre we using 10 sec or 30 sec testing segments ?
\n
I used 10 sec testing segments.
\n\n\nThe SED notebook you referenced seems to be doing different training
\n
Clip time can be changed between training and test. However, since it changes the prediction a bit, we experimented with it(test clip:10sec vs 30sec).
", "votes": 4, "replies": [] }, { "id": 1159984, "author_name": "NakedKoala", "author_url": "", "post_date": "2021-01-19T15:39:18.177000", "content": "@shinmurashinmura Got it. Thanks for the detailed answers. Really appreciate it
", "votes": 0, "replies": [] } ] }, { "id": 1140375, "author_name": "( ͡° ͜ʖ ͡°)", "author_url": "", "post_date": "2021-01-06T00:55:07.680000", "content": "Hi
\nI use the SED too. But it is not stable, could you tell me what wrong in my notebook?
https://www.kaggle.com/truonghoang/rfcx-pannscnn14att-train-inference
\nNote: I used the Multi-label + LWLRAP
\nThanks.
Do you use pannscnn14? In my experiment, pannscnn14 had LB:0.762(single model).
\nThis is not strong model. EfficientNet is recommended.
EfficientB0 is good for me too.
\nbut only in train pipeline. When i submit it, it's so worse.
Why do CV and LBs different? I think three reasons. However, it is a long text and I want to write a new topic later. Please wait.
", "votes": 3, "replies": [] }, { "id": 1147676, "author_name": "Shiro", "author_url": "", "post_date": "2021-01-10T16:35:02.750000", "content": "same as @doanquanvietnamca I did not suceed to have results as high as yours with SED model with Eff architecture
", "votes": 0, "replies": [] } ] }, { "id": 1139401, "author_name": "Theo Viel", "author_url": "", "post_date": "2021-01-05T11:14:31.723000", "content": "So SED is once again showing great results, interesting !
\n(I say once again because it won the last audio competition)
", "votes": 1, "replies": [ { "id": 1139522, "author_name": "shinmura0", "author_url": "", "post_date": "2021-01-05T13:00:04.570000", "content": "It's Cornell Birdcall Identification!
\nYou are excellent performance in that competition.
I think SED is a strong method in multi label task.Cornell Birdcall Identification and this competition is multi label task. Therefore SED is good performance.
", "votes": 1, "replies": [] } ] }, { "id": 1143495, "author_name": "Shiro", "author_url": "", "post_date": "2021-01-07T23:40:26.367000", "content": "@shinmurashinmura Thanks you for sharing . Have you try to use the clipwise output ? if it's the case, how good they are compared to segmentwise output ?
", "votes": 2, "replies": [ { "id": 1143721, "author_name": "shinmura0", "author_url": "", "post_date": "2021-01-08T03:03:50.423000", "content": "I don't try clipwise_output prediction. Sorry.
Does anyone know of a Tensorflow Notebook that implements SED? All the examples I've seen are in PyTorch.
", "votes": 2, "replies": [] }, { "id": 1140569, "author_name": "ibuki", "author_url": "", "post_date": "2021-01-06T05:38:04.827000", "content": "Tnank you for your great discussion!
\nI have a question.
\nDo you have any reason that you don't use strong label when training SED.
\nIntuitively, I think the score will improve more, but do you have any insights?
The reason for not using strong label is that missing labels exist.
\nAs shown here, there are missing labels in many sound sources.
If I get random 10sec clip, It might contain missing labels. Robust training(for missing labels) is possible by using weak labels.However, I have not experimented with strong label.it may be better result with using strong label.
", "votes": 4, "replies": [] }, { "id": 1140801, "author_name": "", "author_url": "", "post_date": "2021-01-06T09:24:38.450000", "content": "", "votes": 0, "replies": [] }, { "id": 1140800, "author_name": "ibuki", "author_url": "", "post_date": "2021-01-06T09:24:38.450000", "content": "Tnank you for your answering!
\nIt is very convincing.
Thanks for kindly sharing your approach in post and your replies really helped. I am waiting for your topic on \"How to use SED\".
", "votes": 1, "replies": [] }, { "id": 1176821, "author_name": "Enricovoltan", "author_url": "", "post_date": "2021-01-29T20:10:50.770000", "content": "Thanks! I'm joining this competition now, hope it wont be too late :)
", "votes": 0, "replies": [] }, { "id": 1168300, "author_name": "Shujun", "author_url": "", "post_date": "2021-01-24T20:59:55.140000", "content": "@shinmurashinmura nice write-up. I have a question about the strong labels. How do you define them? From what I understand, lets say you take 0-10 sec segment of the whole clip, and there an event at 1-2 sec and the cnn returns a B,10,C tensor after avg the frequency dimension, do you just say the second segment has an event as a binary label?
", "votes": 0, "replies": [ { "id": 1170158, "author_name": "shinmura0", "author_url": "", "post_date": "2021-01-26T03:21:23.017000", "content": "\n\ndo you just say the second segment has an event as a binary label?
\n
Yes. Binary label(second segment) is needed.
\nhttps://www.kaggle.com/c/rfcx-species-audio-detection/discussion/211007#1152329
I see. You mentioned you're using clip labels in the other post; from what I understand clip labels give you the best results? I also tried using frame labels (labeling each column after interpolate), but results are worse
", "votes": 1, "replies": [] }, { "id": 1173669, "author_name": "shinmura0", "author_url": "", "post_date": "2021-01-28T03:34:37.917000", "content": "I tried clipwise vs framewise for prediction. Clipwise have improved a bit. You are right.
\nI also will use clipwise for prediction. Thank you.
Interesting to see SED working so well!
", "votes": 0, "replies": [] }, { "id": 1140878, "author_name": "steaphan", "author_url": "", "post_date": "2021-01-06T10:39:10.847000", "content": "I'm new to this compeition after the NFL has just finished, you share a lot on the disscusion which is great for a starter.
\nMaybe you could try to treat FP label as TP label, but in a weak prob way?
Hi, what are weak labels? What does it mean to work with weak labels?
", "votes": 0, "replies": [ { "id": 1141029, "author_name": "shinmura0", "author_url": "", "post_date": "2021-01-06T12:46:54.547000", "content": "What is weak label? Let's check link.
\nhttps://www.kaggle.com/hidehisaarai1213/introduction-to-sound-event-detection
And \"Train SED model with only weak supervision\" help you.
", "votes": 1, "replies": [] }, { "id": 1141074, "author_name": "Kramarenko Vladislav", "author_url": "", "post_date": "2021-01-06T13:24:20.450000", "content": "Thank you, I read that. It's just that in this problem, in both approaches, we cut a segment out of the audio and claim there is a target class. I don't understand what the difference is? Or is the weak label just the name of the SED approach?
", "votes": 0, "replies": [] }, { "id": 1142232, "author_name": "shinmura0", "author_url": "", "post_date": "2021-01-07T08:42:05.230000", "content": "Now, I'm writing new topic \"How to use SED\". Please wait.
", "votes": 1, "replies": [] } ] }, { "id": 1139338, "author_name": "Darkate", "author_url": "", "post_date": "2021-01-05T10:27:08.297000", "content": "Thanks for sharing! Is your SED using strong supervision and producing clipwise output as final prediction?
", "votes": 0, "replies": [ { "id": 1139552, "author_name": "shinmura0", "author_url": "", "post_date": "2021-01-05T13:31:51.947000", "content": "My SED training is
\nFinal prediction is
\n
the visualization looks good . i did not think about viz frame prediction as a 2d image. this will help me understand my prediction even better . thanks
", "votes": 0, "replies": [] }, { "id": 1141293, "author_name": "Svitlana Tarasenko ", "author_url": "", "post_date": "2021-01-06T15:54:37.770000", "content": "What image size did you use?
", "votes": 0, "replies": [] }, { "id": 1144492, "author_name": "Gold Retriever", "author_url": "", "post_date": "2021-01-08T13:41:23.640000", "content": "@shinmurashinmura Thanks for the clear graphical explaination. My understanding is that you train with clip-wise label and conduct inference with frame-wise label. Is my understanding correct?
", "votes": 0, "replies": [] }, { "id": 1149631, "author_name": "shinmura0", "author_url": "", "post_date": "2021-01-12T03:03:15.690000", "content": "\n\n\nMy understanding is that you train with clip-wise label and conduct inference with frame-wise label.
\n
Yes, I used this style. And for your reference (3. A gap training and prediction)
", "votes": 1, "replies": [] }, { "id": 1149973, "author_name": "Gold Retriever", "author_url": "", "post_date": "2021-01-12T09:17:07.373000", "content": "Thanks for clarifying!
", "votes": 0, "replies": [] } ] }, { "id": 1139224, "author_name": "resistance0108", "author_url": "", "post_date": "2021-01-05T08:59:24.270000", "content": "Using SED? Interesting…
\nHow high is your single model score?
I forget single model score.
\nBut EfficientNetB3(5fold ensemble) had 0.913 score.
I tried replicating this but failed probably missing a processing step. Do you mind @shinmurashinmura sharing your notebook that got you 0.913? Thanks in advance. :)
", "votes": 0, "replies": [] } ] }, { "id": 1143330, "author_name": "Yier", "author_url": "", "post_date": "2021-01-07T21:04:22.937000", "content": "Thank you!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1139171": "My main approach is [SED](https://www.kaggle.com/hidehisaarai1213/introduction-to-sound-event-detection).\nThe following methods worked well(LB score is 0.9 over).\n+ PANNs SED architecture\n+ BCE Loss\n+ Random 10sec clip\n+ Using only tp file\n+ MixUp\n+ The base model is EfficientNet\n\nBut the following methods didn't work.\n+ Dice Loss\n+ SpecAugment\n+ Using **fp file** as background noise sound(For example, mix 0.9tp + 0.1fp)\n+ Using **fp file** as no label sound (all zero label)\n+ Self Supervised Learning ([COLA](https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/197805))\n+ Discover missing labels and retrain\n\nCOLA has an optimal batch size of 1024 in the paper. However, in my environment, the limit of batch size was 64 and therefore it did not work.\n\nAlso, as many participants may have noticed, there are many **missing labels** in the tp file sources. I discovered the missing labels and retrained using them, but it did not improve.\n\nNote: I showed missing label example in [this topic](https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/209040).", "1161714": "> COLA has an optimal batch size of 1024 in the paper. However, in my environment, the limit of batch size was 64 and therefore it did not work.\n\nIf you use pytorch then you can simulate larger batch size by one of these, or by combining them:\n- gradient accumulation: only call optimizer step every n batches is almost the same as using batch size n times larger. I say almost because batch norm is updated at each mini batch, not every n mini batches.\n- use distributed data parallel, if you have several GPUs.", "1139769": "Thanks for sharing!!! What are your validation strategies and metrics ? Do your CV results correlate with your LB score?", "1139533": "its interesting to see MIXUP improve the soce. for me, the mixup code provided in the PANN was not improving my score. \nI also found with a larger base model(resnext50) the performance reduced. \npsudolabeling for the missing label did not help me too.\nlonger training did help me after correcting the learning rate.", "1154880": "\n@shinmura0 \n\nThanks for sharing the approach. I have 2 questions.\n\n### Random Cropping:\n\nYou mentioned that training instance is obtained from \"Random 10sec clip(contain tp sound)\". I understand that random crop is obtained by some random shift of the tmin tmax window given in the training data, is this correct ? If yes, do you generate multiple training instances (multiple random crops with different offsets ) from a single audio file ? Also, do you mind sharing random cropping parameters ( IE: the range of possible offset from the ground truth window).\n\nI am sorry if these data preprocessing questions sound very basic. I am new to audio competition and SED in general .\n\n### Testing clip length.\n\nI am a bit confused about how long a testing segment is. Are we using 10 sec or 30 sec testing segments ? \n\nQuoting discussion:\n\n\"I experimented period10 vs period30. The winner was period30.\"\n\n\"And I split 60sec test clip per 10 sec.\"\n\nIf we are not using 10 seconds long test segment, what is the reason for training on 1 segment length and testing on another segment length ? The SED notebook you referenced seems to be doing different training / testing segment length too ( 5 period training segment and 30 period testing segment ). I am wondering what is the rationale for doing this ? \n\n\n\n\n", "1140375": "Hi\nI use the SED too. But it is not stable, could you tell me what wrong in my notebook?\n\n[https://www.kaggle.com/truonghoang/rfcx-pannscnn14att-train-inference](https://www.kaggle.com/truonghoang/rfcx-pannscnn14att-train-inference)\n\nNote: I used the Multi-label + LWLRAP\nThanks.", "1139401": "So SED is once again showing great results, interesting !\n\n(I say once again because it won the last audio competition)", "1143495": "@shinmurashinmura Thanks you for sharing . Have you try to use the clipwise output ? if it's the case, how good they are compared to segmentwise output ?", "1141258": "Does anyone know of a Tensorflow Notebook that implements SED? All the examples I've seen are in PyTorch.", "1140569": "Tnank you for your great discussion!\nI have a question.\nDo you have any reason that you don't use strong label when training SED.\nIntuitively, I think the score will improve more, but do you have any insights?", "1146530": "Thanks for kindly sharing your approach in post and your replies really helped. I am waiting for your topic on \"How to use SED\". ", "1176821": "Thanks! I'm joining this competition now, hope it wont be too late :)", "1168300": "@shinmurashinmura nice write-up. I have a question about the strong labels. How do you define them? From what I understand, lets say you take 0-10 sec segment of the whole clip, and there an event at 1-2 sec and the cnn returns a B,10,C tensor after avg the frequency dimension, do you just say the second segment has an event as a binary label? ", "1159132": "Interesting to see SED working so well!", "1140878": "I'm new to this compeition after the NFL has just finished, you share a lot on the disscusion which is great for a starter.\nMaybe you could try to treat FP label as TP label, but in a weak prob way?", "1140829": "Hi, what are weak labels? What does it mean to work with weak labels?", "1139338": "Thanks for sharing! Is your SED using strong supervision and producing clipwise output as final prediction?", "1139224": "Using SED? Interesting...\nHow high is your single model score?", "1143330": "Thank you!" } }