{ "id": 173144, "title": "Your CV vs. LB comparison", "url": "/competitions/birdsong-recognition/discussion/173144", "author_name": "", "post_date": "2020-08-08T04:23:38.383665Z", "votes": 27, "comment_count": 59, "views": 0, "content": "

Didn't see this post in this competition. Thought might be important.

", "messages": [ { "id": "962348", "postDate": "08/08/2020 04:23:38", "content": "

Didn't see this post in this competition. Thought might be important.

", "rawMarkdown": "Didn't see this post in this competition. Thought might be important.", "votes": null }, { "id": "962444", "postDate": "08/08/2020 06:12:31", "content": "

Yep. If we have some reasonable validation policy

", "rawMarkdown": "Yep. If we have some reasonable validation policy", "votes": null }, { "id": "964256", "postDate": "08/09/2020 17:34:48", "content": "

Single EfficientNet B1 CV5 model (*S*tate *O*f the *A*rt model with 5 seconds slices followed by MEL-spectrogram to feed CNN):\nCV=0.672/0.675/0.667/0.668/0.685 \nLB=0.572

", "rawMarkdown": "Single EfficientNet B1 CV5 model (**S**tate **O**f the **A**rt model with 5 seconds slices followed by MEL-spectrogram to feed CNN):\nCV=0.672/0.675/0.667/0.668/0.685 \nLB=0.572", "votes": null }, { "id": "964537", "postDate": "08/10/2020 01:36:02", "content": "

Thanks for sharing. What does CV5 mean? Also, could you share your CV strategy? @mpware

", "rawMarkdown": "Thanks for sharing. What does CV5 mean? Also, could you share your CV strategy? @mpware", "votes": null }, { "id": "964879", "postDate": "08/10/2020 08:33:47", "content": "

Cross Validation with 5 folds. I'm using StratifiedKFold (K=5) to balance target among each folds. Nothing more than a regular CV strategy. CosineAnnealingLR is used too (1 cycle). I've also removed too short (to have enough data) or too long clips (to speed up training). Here is one fold example:

\n\n

\"\"

", "rawMarkdown": "Cross Validation with 5 folds. I'm using `StratifiedKFold` (K=5) to balance target among each folds. Nothing more than a regular CV strategy. `CosineAnnealingLR` is used too (1 cycle). I've also removed too short (to have enough data) or too long clips (to speed up training). Here is one fold example:\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F698363%2F90316b6aa7f2a8271044563c12568e93%2Ftrain.png?generation=1597049654653056&alt=media)", "votes": null }, { "id": "965023", "postDate": "08/10/2020 10:33:48", "content": "

@mpware That's amazing. Thank you for replying. Are you using sklearn for f1 calculation? or something else.

", "rawMarkdown": "mpware That's amazing. Thank you for replying. Are you using sklearn for f1 calculation? or something else.", "votes": null }, { "id": "965032", "postDate": "08/10/2020 10:39:45", "content": "

Yes,

\n\n

```\nAVG = 'samples'

\n\n

def f1(y_true, y_pred):\n return metrics.f1_score(y_true, y_pred, average=AVG)
\n```

", "rawMarkdown": "Yes,\n\n```\nAVG = 'samples'\n\ndef f1(y_true, y_pred):\n return metrics.f1_score(y_true, y_pred, average=AVG) \n```", "votes": null }, { "id": "966190", "postDate": "08/11/2020 08:17:21", "content": "

@mpware Thanks!

", "rawMarkdown": "mpware Thanks!", "votes": null }, { "id": "967778", "postDate": "08/12/2020 13:56:28", "content": "

Thanks for your insight @mpware . Glad to see you in this comp .

", "rawMarkdown": "Thanks for your insight @mpware . Glad to see you in this comp .", "votes": null }, { "id": "968350", "postDate": "08/12/2020 23:16:55", "content": "

Thanks for sharing I have also tried Efficient Net B1 with 5 fold CV but it did not reflect on my LB. Can you share the batch size which was possible through your system.

", "rawMarkdown": "Thanks for sharing I have also tried Efficient Net B1 with 5 fold CV but it did not reflect on my LB. Can you share the batch size which was possible through your system.", "votes": null }, { "id": "968729", "postDate": "08/13/2020 07:55:09", "content": "

Batch size = 48

", "rawMarkdown": "Batch size = 48", "votes": null }, { "id": "970312", "postDate": "08/14/2020 10:56:26", "content": "

Thanks for sharing! That's good, that even with roughly speaking constant train loss, you get increasing train metrics.

", "rawMarkdown": "Thanks for sharing! That's good, that even with roughly speaking constant train loss, you get increasing train metrics.", "votes": null }, { "id": "970409", "postDate": "08/14/2020 12:39:19", "content": "

We tried Efficient Net b5 with various combination of batch sizes and resolution, but we always ended up with a poor LB score, while during CV, our model performed really great. What resolution of image did u opt for these results?

", "rawMarkdown": "We tried Efficient Net b5 with various combination of batch sizes and resolution, but we always ended up with a poor LB score, while during CV, our model performed really great. What resolution of image did u opt for these results?", "votes": null }, { "id": "970422", "postDate": "08/14/2020 12:52:36", "content": "

Image size is 240x445. Training procedure is important, you need to add some augmentations (add noise, backgrounds, …). What is your CV?

\n

I've also trained a ResneSt50 with the same procedure and I got same LB even if CV was a bit better CV: 0.703/0.706/0.685/0.708/0.703.

\n

\"\"

\n

Also, main difficulty is that test set is soundscape with a lot background noise that differs from the train set (which is cleaner).

", "rawMarkdown": "Image size is 240x445. Training procedure is important, you need to add some augmentations (add noise, backgrounds, ...). What is your CV? \n\nI've also trained a ResneSt50 with the same procedure and I got same LB even if CV was a bit better CV: 0.703/0.706/0.685/0.708/0.703.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F698363%2Fd1742929a2dc2be03cb747dd1d1991a6%2Ftrain.png?generation=1597409518166977&alt=media)\n\nAlso, main difficulty is that test set is soundscape with a lot background noise that differs from the train set (which is cleaner).", "votes": null }, { "id": "970564", "postDate": "08/14/2020 15:06:01", "content": "

Seems like going with a shallower model and increasing batches helped u a lot. My best so far has been 0.62 (organic), we have tried a couple of noise augmentations but we are still loitering around that score.
\nThanx for the hint, kind sir. Cheers!!

\n

\"\"

\n

\"\"

\n

\"\"

\n

\"\"

", "rawMarkdown": "Seems like going with a shallower model and increasing batches helped u a lot. My best so far has been 0.62 (organic), we have tried a couple of noise augmentations but we are still loitering around that score.\nThanx for the hint, kind sir. Cheers!!\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1604687%2Fef0384238ad2650804f1b090f6390834%2Fscreenshot-app.wandb.ai-2020.08.14-20_31_00.png?generation=1597417293673418&alt=media)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1604687%2Ffc70d306a57e5148b091dbf48ef6db89%2Fscreenshot-app.wandb.ai-2020.08.14-20_30_47.png?generation=1597417347235344&alt=media)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1604687%2F6c02b010dd032b2a717e9642fd8aafd3%2Fscreenshot-app.wandb.ai-2020.08.14-20_30_13.png?generation=1597417374634914&alt=media)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1604687%2F929b9555cd0d5f10ec70a261a7579f4d%2Fscreenshot-app.wandb.ai-2020.08.14-20_29_50.png?generation=1597417425828374&alt=media)", "votes": null }, { "id": "970830", "postDate": "08/14/2020 21:09:48", "content": "

I have also tried adding background clips but I cannot figure with the noise to data ratio. If you can give a hint regarding augmentations and how you go about it will be very very helpful. Thanks

", "rawMarkdown": "I have also tried adding background clips but I cannot figure with the noise to data ratio. If you can give a hint regarding augmentations and how you go about it will be very very helpful. Thanks", "votes": null }, { "id": "981555", "postDate": "08/22/2020 14:33:57", "content": "

@mpware Can u share how many epochs does it take for you to reach 0.70+ f1 score with Resnest50.

", "rawMarkdown": "mpware Can u share how many epochs does it take for you to reach 0.70+ f1 score with Resnest50.", "votes": null }, { "id": "981657", "postDate": "08/22/2020 15:57:45", "content": "

What do you mean by 'organic'?

", "rawMarkdown": "What do you mean by 'organic'?", "votes": null }, { "id": "981682", "postDate": "08/22/2020 16:26:15", "content": "

Epochs = 64

", "rawMarkdown": "Epochs = 64", "votes": null }, { "id": "983333", "postDate": "08/24/2020 07:42:54", "content": "

@mpware was your choice of batch size due to memory constraints or because you found that to work best in your model?

", "rawMarkdown": "mpware was your choice of batch size due to memory constraints or because you found that to work best in your model?", "votes": null }, { "id": "983911", "postDate": "08/24/2020 17:21:54", "content": "

@brianfeeny Yes memory constraints, I'm trying to use the highest possible but with P100 GPU I'm limited. I've another model trained with Batch Size = 36 that scores similarly so I don't think there is a big difference with batch sizes within [36 - 48]

", "rawMarkdown": "brianfeeny Yes memory constraints, I'm trying to use the highest possible but with P100 GPU I'm limited. I've another model trained with Batch Size = 36 that scores similarly so I don't think there is a big difference with batch sizes within [36 - 48]", "votes": null }, { "id": "983941", "postDate": "08/24/2020 17:44:23", "content": "

Thanks for sharing @mpware , I would be greatful if you answer the following questions

\n", "rawMarkdown": "Thanks for sharing @mpware , I would be greatful if you answer the following questions\n* Did you try PCEN mel spectograms?\n* Did you try SED ?\nWhat do you think about the large Cv/LB gap?", "votes": null }, { "id": "983983", "postDate": "08/24/2020 18:30:56", "content": "

No PCEN. I've tried SED but it does not work for me. I mean it does not work better than a model with regular average/max pooling. SED just adds an attention pooling, it generates a lot of false positive so I guess training procedure is super important to benefit from it but I didn't figure out how. I have not tried other pooling solutions yet.

", "rawMarkdown": "No PCEN. I've tried SED but it does not work for me. I mean it does not work better than a model with regular average/max pooling. SED just adds an attention pooling, it generates a lot of false positive so I guess training procedure is super important to benefit from it but I didn't figure out how. I have not tried other pooling solutions yet.", "votes": null }, { "id": "984123", "postDate": "08/24/2020 21:01:53", "content": "

@MPWARE have you tried AMP? It should give you plenty of memory head room and likely speed up training considerably.

", "rawMarkdown": "MPWARE have you tried AMP? It should give you plenty of memory head room and likely speed up training considerably.", "votes": null }, { "id": "984131", "postDate": "08/24/2020 21:24:37", "content": "

Not tried yet, just wondering if it would work on inference kernel as it requires PyTorch 1.6, correct? Current Kaggle docker comes with PyTorch 1.5.1 and I've read that Apex installation was a pain for offline kernel.

", "rawMarkdown": "Not tried yet, just wondering if it would work on inference kernel as it requires [PyTorch 1.6](https://pytorch.org/blog/accelerating-training-on-nvidia-gpus-with-pytorch-automatic-mixed-precision/), correct? Current Kaggle docker comes with PyTorch 1.5.1 and I've read that Apex installation was a pain for offline kernel.", "votes": null }, { "id": "984157", "postDate": "08/24/2020 22:18:58", "content": "

@MPWARE good point I was not aware of that. I am not sure if you can train in fp16 and then do prediction in fp32. The main benefit of AMP is in training. With inference, we can use large batch sizes, with no negative effect, since we are not needing to store gradients/backprop.

", "rawMarkdown": "MPWARE good point I was not aware of that. I am not sure if you can train in fp16 and then do prediction in fp32. The main benefit of AMP is in training. With inference, we can use large batch sizes, with no negative effect, since we are not needing to store gradients/backprop.", "votes": null }, { "id": "984659", "postDate": "08/25/2020 08:01:53", "content": "

Single Resnest50, single fold CV(f1_score)=0.6877, LB:0.570. (Thanks to @mpware )

\n
\n

[UPDATE]: CV(5folds): 0.687/0.684/0.683/0.695/0.691 , LB: 0.572

\n
", "rawMarkdown": "Single Resnest50, single fold CV(f1_score)=0.6877, LB:0.570. (Thanks to @mpware )\n\n> [UPDATE]: CV(5folds): 0.687/0.684/0.683/0.695/0.691 , LB: 0.572", "votes": null }, { "id": "984677", "postDate": "08/25/2020 08:14:15", "content": "

CV against what? Just the regular sounds you are training on? Or against BirdCheck or what? What metric are you measuring for CV?

", "rawMarkdown": "CV against what? Just the regular sounds you are training on? Or against BirdCheck or what? What metric are you measuring for CV?", "votes": null }, { "id": "984736", "postDate": "08/25/2020 08:50:38", "content": "

@brianfeeny everything is almost same as what @mpware mentioned.

", "rawMarkdown": "brianfeeny everything is almost same as what @mpware mentioned.", "votes": null }, { "id": "984789", "postDate": "08/25/2020 09:34:29", "content": "

Thanks I didn't see what you all are measuring though to see how your model is converging

", "rawMarkdown": "Thanks I didn't see what you all are measuring though to see how your model is converging", "votes": null }, { "id": "984806", "postDate": "08/25/2020 09:52:38", "content": "

@rohitsingh9990 Can you please tell if you added background noise etc , or you have got this just by changing hyperparameters??

", "rawMarkdown": "rohitsingh9990 Can you please tell if you added background noise etc , or you have got this just by changing hyperparameters??", "votes": null }, { "id": "984808", "postDate": "08/25/2020 09:56:01", "content": "

@brianfeeny we all measuring f1_score. as already mentioned above by mpware

\n
AVG = 'samples'\n\ndef f1(y_true, y_pred):\n    return metrics.f1_score(y_true, y_pred, average=AVG)\n
", "rawMarkdown": "brianfeeny we all measuring f1_score. as already mentioned above by mpware\n\n```\nAVG = 'samples'\n\ndef f1(y_true, y_pred):\n return metrics.f1_score(y_true, y_pred, average=AVG)\n```", "votes": null }, { "id": "984811", "postDate": "08/25/2020 09:58:56", "content": "

@tanulsingh077 yes, i added a lot of augmentations, and one of them is noise addition. Augmentations are really helpful in this task, if you apply them correctly.

", "rawMarkdown": "tanulsingh077 yes, i added a lot of augmentations, and one of them is noise addition. Augmentations are really helpful in this task, if you apply them correctly.", "votes": null }, { "id": "985025", "postDate": "08/25/2020 12:53:56", "content": "

Organic as in without using public kernels

", "rawMarkdown": "Organic as in without using public kernels", "votes": null }, { "id": "985095", "postDate": "08/25/2020 13:57:10", "content": "

@mpware could you please give some insights or ressources to do augmentations ?

", "rawMarkdown": "mpware could you please give some insights or ressources to do augmentations ?", "votes": null }, { "id": "985503", "postDate": "08/25/2020 19:01:01", "content": "

So when you all do your multiple folds, are you combining/averaging your class probabilities and then predicting classes or do you let each fold predict classes and then do some sort of \"vote\" such as 2 of 4, or 3 of 4 folds agree etc?

", "rawMarkdown": "So when you all do your multiple folds, are you combining/averaging your class probabilities and then predicting classes or do you let each fold predict classes and then do some sort of \"vote\" such as 2 of 4, or 3 of 4 folds agree etc?", "votes": null }, { "id": "986917", "postDate": "08/26/2020 21:47:43", "content": "

Did you apply a threshold on prediction and use the 'samples' F1 score?

", "rawMarkdown": "Did you apply a threshold on prediction and use the 'samples' F1 score?", "votes": null }, { "id": "987415", "postDate": "08/27/2020 08:33:55", "content": "

@mpware Thanks for your generous sharing. If you don't mind disclosing:

\n
    \n
  1. Do you validate using 5s clips randomly cropped from each audio? (If I understand correctly, you are also using clips of this format to train).
    2. \n
  2. For calculation of f1, do you use threshold of .5?
    3. \n
  3. For calculation of f1, how do you account for \"nocall\" class? Or just calculate f1 without \"nocall\" considered?
    4. \n
\n

By the way, for installation of apex, maybe you can do this:

\n
%%bash\ngit clone https://github.com/NVIDIA/apex (replace it with .zip you download from github)\ncd apex\npip install -v --no-cache-dir --global-option=\"--cpp_ext\" --global-option=\"--cuda_ext\" ./\n
\n

Thanks in advance

", "rawMarkdown": "mpware Thanks for your generous sharing. If you don't mind disclosing:\n1. Do you validate using 5s clips randomly cropped from each audio? (If I understand correctly, you are also using clips of this format to train).\n2. For calculation of f1, do you use threshold of .5?\n3. For calculation of f1, how do you account for \"nocall\" class? Or just calculate f1 without \"nocall\" considered?\n\nBy the way, for installation of apex, maybe you can do this:\n```\n%%bash\ngit clone https://github.com/NVIDIA/apex (replace it with .zip you download from github)\ncd apex\npip install -v --no-cache-dir --global-option=\"--cpp_ext\" --global-option=\"--cuda_ext\" ./\n```\nThanks in advance", "votes": null }, { "id": "987444", "postDate": "08/27/2020 09:09:10", "content": "

@rguo97 not in current implementation, but i am planning to do that and will share the results.

", "rawMarkdown": "rguo97 not in current implementation, but i am planning to do that and will share the results.", "votes": null }, { "id": "987775", "postDate": "08/27/2020 14:03:09", "content": "

@roguekk007

\n
    \n
  1. I pick 5s randomly for train, but for eval I used the first 5s only.
    2. \n
  2. For F1, yes, threshold=0.5. I'm also monitoring mAP.
    3. \n
  3. I've some models with 264 classes and some with 265 including nocall class (with data including noise). I would expect 265 classes models to work quite better than 264 ones but it's quite similar.
    4. \n
", "rawMarkdown": "roguekk007 \n1. I pick 5s randomly for train, but for eval I used the first 5s only. \n2. For F1, yes, threshold=0.5. I'm also monitoring mAP.\n3. I've some models with 264 classes and some with 265 including nocall class (with data including noise). I would expect 265 classes models to work quite better than 264 ones but it's quite similar.", "votes": null }, { "id": "988003", "postDate": "08/27/2020 17:33:18", "content": "

@mpware , Did you notice higher F1 in validation when you lowering the threshold? In my case, 0.3 threshold is 0.02-0.03 higher than 0.5 threshold. But this is inverse on LB.

", "rawMarkdown": "mpware , Did you notice higher F1 in validation when you lowering the threshold? In my case, 0.3 threshold is 0.02-0.03 higher than 0.5 threshold. But this is inverse on LB.", "votes": null }, { "id": "988007", "postDate": "08/27/2020 17:34:44", "content": "

@rguo97 you are right, i noticed the same.

", "rawMarkdown": "rguo97 you are right, i noticed the same.", "votes": null }, { "id": "988048", "postDate": "08/27/2020 18:18:37", "content": "

Are you also monitoring mAP, is it correlated to what you describe?

", "rawMarkdown": "Are you also monitoring mAP, is it correlated to what you describe?", "votes": null }, { "id": "988392", "postDate": "08/28/2020 02:40:12", "content": "

mAP do not need threshold. I got around 0.73 mAP in my models, but I cannot get a stable LB…

", "rawMarkdown": "mAP do not need threshold. I got around 0.73 mAP in my models, but I cannot get a stable LB...", "votes": null }, { "id": "988607", "postDate": "08/28/2020 06:41:59", "content": "

@mpware Thanks for sharing this.

\n
\n

I've some models with 264 classes and some with 265 including nocall class (with data including noise). I would expect 265 classes models to work quite better than 264 ones but it's quite similar.

\n
\n

I was wondering what are you referreing to by data including noise?

\n

Thanks!

", "rawMarkdown": "mpware Thanks for sharing this.\n\n> I've some models with 264 classes and some with 265 including nocall class (with data including noise). I would expect 265 classes models to work quite better than 264 ones but it's quite similar.\n\nI was wondering what are you referreing to by data including noise?\n\nThanks!", "votes": null }, { "id": "988613", "postDate": "08/28/2020 06:46:47", "content": "

You can try to split data to isolate noise/background, then you can use it as another \"nocall\" class.

", "rawMarkdown": "You can try to split data to isolate noise/background, then you can use it as another \"nocall\" class.", "votes": null }, { "id": "988615", "postDate": "08/28/2020 06:49:30", "content": "

@rguo97 Yes, but when F1 score is high with 0.3 threshold, is mAP score also high for the same epoch?

", "rawMarkdown": "rguo97 Yes, but when F1 score is high with 0.3 threshold, is mAP score also high for the same epoch?", "votes": null }, { "id": "988637", "postDate": "08/28/2020 07:12:26", "content": "

I'm seeing a lot of f1 scores in the cv here, which is great, but I'm wondering how you guys are choosing your thresholds for the f1 score…

", "rawMarkdown": "I'm seeing a lot of f1 scores in the cv here, which is great, but I'm wondering how you guys are choosing your thresholds for the f1 score...", "votes": null }, { "id": "989348", "postDate": "08/28/2020 18:21:26", "content": "

Yes. They are correlated.

", "rawMarkdown": "Yes. They are correlated.", "votes": null }, { "id": "990208", "postDate": "08/29/2020 12:41:38", "content": "

@mpware , how much training time did one epoch took after adding augmentations(noise, pitching etc) ?. I am trying this but with kaggle GPU its taking 1 hr for 1 epoch.

", "rawMarkdown": "mpware , how much training time did one epoch took after adding augmentations(noise, pitching etc) ?. I am trying this but with kaggle GPU its taking 1 hr for 1 epoch.", "votes": null }, { "id": "990224", "postDate": "08/29/2020 12:52:37", "content": "

Felt like sharing our current LB CV scores, although they don't really mean anything.

\n

I use stratified 5-folds as everyone else, but when making the prediction I crop the 5 first second of the clip (I prefered this instead of randomly chosing a 5s crop, for reproducibility), which is why my CV scores are higher than those reported by everyone else.

\n
Fold 1 :   micro_f1=0.809      samples_f1=0.757\nFold 2 :  micro_f1=0.807      samples_f1=0.762\nFold 3 :  micro_f1=0.809      samples_f1=0.755\nFold 4 :  micro_f1=0.808      samples_f1=0.757\nFold 5 :  micro_f1=0.800      samples_f1=0.747\n
\n
\n

LB : 0.617

\n
", "rawMarkdown": "Felt like sharing our current LB CV scores, although they don't really mean anything.\n\nI use stratified 5-folds as everyone else, but when making the prediction I crop the 5 first second of the clip (I prefered this instead of randomly chosing a 5s crop, for reproducibility), which is why my CV scores are higher than those reported by everyone else.\n\n\n```\nFold 1 : micro_f1=0.809 \t samples_f1=0.757\nFold 2 : micro_f1=0.807 \t samples_f1=0.762\nFold 3 : micro_f1=0.809 \t samples_f1=0.755\nFold 4 : micro_f1=0.808 \t samples_f1=0.757\nFold 5 : micro_f1=0.800 \t samples_f1=0.747\n```\n> LB : 0.617", "votes": null }, { "id": "990297", "postDate": "08/29/2020 14:03:26", "content": "

Did the 0.617 of LB score come from 5fold ensemble?

", "rawMarkdown": "Did the 0.617 of LB score come from 5fold ensemble?", "votes": null }, { "id": "990417", "postDate": "08/29/2020 15:31:42", "content": "

Yep, that's the 5 folds described above

", "rawMarkdown": "Yep, that's the 5 folds described above", "votes": null }, { "id": "990475", "postDate": "08/29/2020 16:23:29", "content": "

It's around 8-10 hours for 64 epochs. I've an optimized pipeline running on 12 CPUs + 1 GPU.

", "rawMarkdown": "It's around 8-10 hours for 64 epochs. I've an optimized pipeline running on 12 CPUs + 1 GPU.", "votes": null }, { "id": "990484", "postDate": "08/29/2020 16:32:05", "content": "

@hidehisaarai1213 Is your 0.618 LB sub a single fold, 5 folds, or did you start some blending ?

", "rawMarkdown": "hidehisaarai1213 Is your 0.618 LB sub a single fold, 5 folds, or did you start some blending ?", "votes": null }, { "id": "990542", "postDate": "08/29/2020 17:20:51", "content": "

Thanks , I suppose , since you are also using P100 its a lot faster

", "rawMarkdown": "Thanks , I suppose , since you are also using P100 its a lot faster", "votes": null }, { "id": "990566", "postDate": "08/29/2020 17:39:55", "content": "

Yes, P100.

", "rawMarkdown": "Yes, P100.", "votes": null }, { "id": "990588", "postDate": "08/29/2020 17:56:50", "content": "

Also , It would be helpfull if you can share how you optimized your pipeline, when competition ends. It would help in future.

", "rawMarkdown": "Also , It would be helpfull if you can share how you optimized your pipeline, **when competition ends**. It would help in future.", "votes": null }, { "id": "990693", "postDate": "08/29/2020 19:30:13", "content": "

This competition and best-scored public kernel are heavily limited by CPU preprocessing.

\n

Reading mp3s every time you need a 5 second fragment - slowest (5-10% GPU usage)
\nReading zipped wavs every time you need a 5 second fragment - slow (20-30% GPU usage)
\nReading raw wavs every time you need a 5 second fragment - average (did not try it due to disk space requirement)

\n

Following steps require additional consideration with edge cases, data integrity and training behaviour and might negatively impact your score:
\nReading prepared spectrograms from disk - fast (50-90% GPU usage depending on format/compression, 1 channel bmps are fastest)
\nPlaying with prefetch settings/custom dataloaders/raw numpy arrays - fastest (should be able to 100% load your GPU, i did not go that far myself)

\n

For debugging, print elapsed time after each command in your getitem and after each completed batch, then consider if you really need to do all that stuff on the fly.

", "rawMarkdown": "This competition and best-scored public kernel are heavily limited by CPU preprocessing.\n\nReading mp3s every time you need a 5 second fragment - slowest (5-10% GPU usage)\nReading zipped wavs every time you need a 5 second fragment - slow (20-30% GPU usage)\nReading raw wavs every time you need a 5 second fragment - average (did not try it due to disk space requirement)\n\nFollowing steps require additional consideration with edge cases, data integrity and training behaviour and might negatively impact your score:\nReading prepared spectrograms from disk - fast (50-90% GPU usage depending on format/compression, 1 channel bmps are fastest)\nPlaying with prefetch settings/custom dataloaders/raw numpy arrays - fastest (should be able to 100% load your GPU, i did not go that far myself)\n\nFor debugging, print elapsed time after each command in your __getitem__ and after each completed batch, then consider if you _really_ need to do all that stuff on the fly.", "votes": null }, { "id": "990856", "postDate": "08/29/2020 22:55:22", "content": "

Thx for sharing!
\nMy 0.618 score is a blend of two single fold models (same arch, different settings).

", "rawMarkdown": "Thx for sharing!\nMy 0.618 score is a blend of two single fold models (same arch, different settings).", "votes": null } ], "comments": [ { "id": 984659, "author_name": "rohitsingh9990", "author_url": "", "post_date": "08/25/2020 08:01:53", "content": "

Single Resnest50, single fold CV(f1_score)=0.6877, LB:0.570. (Thanks to @mpware )

\n
\n

[UPDATE]: CV(5folds): 0.687/0.684/0.683/0.695/0.691 , LB: 0.572

\n
", "votes": null, "replies": [ { "id": 984677, "author_name": "brianfeeny", "author_url": "", "post_date": "08/25/2020 08:14:15", "content": "

CV against what? Just the regular sounds you are training on? Or against BirdCheck or what? What metric are you measuring for CV?

", "votes": null, "replies": [] }, { "id": 984736, "author_name": "rohitsingh9990", "author_url": "", "post_date": "08/25/2020 08:50:38", "content": "

@brianfeeny everything is almost same as what @mpware mentioned.

", "votes": null, "replies": [] }, { "id": 984789, "author_name": "brianfeeny", "author_url": "", "post_date": "08/25/2020 09:34:29", "content": "

Thanks I didn't see what you all are measuring though to see how your model is converging

", "votes": null, "replies": [] }, { "id": 984806, "author_name": "tanulsingh077", "author_url": "", "post_date": "08/25/2020 09:52:38", "content": "

@rohitsingh9990 Can you please tell if you added background noise etc , or you have got this just by changing hyperparameters??

", "votes": null, "replies": [] }, { "id": 984808, "author_name": "rohitsingh9990", "author_url": "", "post_date": "08/25/2020 09:56:01", "content": "

@brianfeeny we all measuring f1_score. as already mentioned above by mpware

\n
AVG = 'samples'\n\ndef f1(y_true, y_pred):\n    return metrics.f1_score(y_true, y_pred, average=AVG)\n
", "votes": null, "replies": [] }, { "id": 984811, "author_name": "rohitsingh9990", "author_url": "", "post_date": "08/25/2020 09:58:56", "content": "

@tanulsingh077 yes, i added a lot of augmentations, and one of them is noise addition. Augmentations are really helpful in this task, if you apply them correctly.

", "votes": null, "replies": [] }, { "id": 986917, "author_name": "rguo97", "author_url": "", "post_date": "08/26/2020 21:47:43", "content": "

Did you apply a threshold on prediction and use the 'samples' F1 score?

", "votes": null, "replies": [] }, { "id": 987444, "author_name": "rohitsingh9990", "author_url": "", "post_date": "08/27/2020 09:09:10", "content": "

@rguo97 not in current implementation, but i am planning to do that and will share the results.

", "votes": null, "replies": [] } ] }, { "id": 985503, "author_name": "brianfeeny", "author_url": "", "post_date": "08/25/2020 19:01:01", "content": "

So when you all do your multiple folds, are you combining/averaging your class probabilities and then predicting classes or do you let each fold predict classes and then do some sort of \"vote\" such as 2 of 4, or 3 of 4 folds agree etc?

", "votes": null, "replies": [] }, { "id": 988637, "author_name": "lewington", "author_url": "", "post_date": "08/28/2020 07:12:26", "content": "

I'm seeing a lot of f1 scores in the cv here, which is great, but I'm wondering how you guys are choosing your thresholds for the f1 score…

", "votes": null, "replies": [] }, { "id": 990224, "author_name": "theoviel", "author_url": "", "post_date": "08/29/2020 12:52:37", "content": "

Felt like sharing our current LB CV scores, although they don't really mean anything.

\n

I use stratified 5-folds as everyone else, but when making the prediction I crop the 5 first second of the clip (I prefered this instead of randomly chosing a 5s crop, for reproducibility), which is why my CV scores are higher than those reported by everyone else.

\n
Fold 1 :   micro_f1=0.809      samples_f1=0.757\nFold 2 :  micro_f1=0.807      samples_f1=0.762\nFold 3 :  micro_f1=0.809      samples_f1=0.755\nFold 4 :  micro_f1=0.808      samples_f1=0.757\nFold 5 :  micro_f1=0.800      samples_f1=0.747\n
\n
\n

LB : 0.617

\n
", "votes": null, "replies": [ { "id": 990297, "author_name": "hidehisaarai1213", "author_url": "", "post_date": "08/29/2020 14:03:26", "content": "

Did the 0.617 of LB score come from 5fold ensemble?

", "votes": null, "replies": [] }, { "id": 990417, "author_name": "theoviel", "author_url": "", "post_date": "08/29/2020 15:31:42", "content": "

Yep, that's the 5 folds described above

", "votes": null, "replies": [] }, { "id": 990484, "author_name": "theoviel", "author_url": "", "post_date": "08/29/2020 16:32:05", "content": "

@hidehisaarai1213 Is your 0.618 LB sub a single fold, 5 folds, or did you start some blending ?

", "votes": null, "replies": [] }, { "id": 990856, "author_name": "hidehisaarai1213", "author_url": "", "post_date": "08/29/2020 22:55:22", "content": "

Thx for sharing!
\nMy 0.618 score is a blend of two single fold models (same arch, different settings).

", "votes": null, "replies": [] } ] }, { "id": 962444, "author_name": "vladimirsydor", "author_url": "", "post_date": "08/08/2020 06:12:31", "content": "

Yep. If we have some reasonable validation policy

", "votes": null, "replies": [] }, { "id": 964256, "author_name": "mpware", "author_url": "", "post_date": "08/09/2020 17:34:48", "content": "

Single EfficientNet B1 CV5 model (*S*tate *O*f the *A*rt model with 5 seconds slices followed by MEL-spectrogram to feed CNN):\nCV=0.672/0.675/0.667/0.668/0.685 \nLB=0.572

", "votes": null, "replies": [ { "id": 964537, "author_name": "tonychenxyz", "author_url": "", "post_date": "08/10/2020 01:36:02", "content": "

Thanks for sharing. What does CV5 mean? Also, could you share your CV strategy? @mpware

", "votes": null, "replies": [] }, { "id": 964879, "author_name": "mpware", "author_url": "", "post_date": "08/10/2020 08:33:47", "content": "

Cross Validation with 5 folds. I'm using StratifiedKFold (K=5) to balance target among each folds. Nothing more than a regular CV strategy. CosineAnnealingLR is used too (1 cycle). I've also removed too short (to have enough data) or too long clips (to speed up training). Here is one fold example:

\n\n

\"\"

", "votes": null, "replies": [ { "id": 970312, "author_name": "koza4ukdmitrij", "author_url": "", "post_date": "08/14/2020 10:56:26", "content": "

Thanks for sharing! That's good, that even with roughly speaking constant train loss, you get increasing train metrics.

", "votes": null, "replies": [] } ] }, { "id": 965023, "author_name": "tonychenxyz", "author_url": "", "post_date": "08/10/2020 10:33:48", "content": "

@mpware That's amazing. Thank you for replying. Are you using sklearn for f1 calculation? or something else.

", "votes": null, "replies": [] }, { "id": 965032, "author_name": "mpware", "author_url": "", "post_date": "08/10/2020 10:39:45", "content": "

Yes,

\n\n

```\nAVG = 'samples'

\n\n

def f1(y_true, y_pred):\n return metrics.f1_score(y_true, y_pred, average=AVG)
\n```

", "votes": null, "replies": [] }, { "id": 966190, "author_name": "tonychenxyz", "author_url": "", "post_date": "08/11/2020 08:17:21", "content": "

@mpware Thanks!

", "votes": null, "replies": [] }, { "id": 967778, "author_name": "phoenix9032", "author_url": "", "post_date": "08/12/2020 13:56:28", "content": "

Thanks for your insight @mpware . Glad to see you in this comp .

", "votes": null, "replies": [] }, { "id": 968350, "author_name": "kuldeepvansadia", "author_url": "", "post_date": "08/12/2020 23:16:55", "content": "

Thanks for sharing I have also tried Efficient Net B1 with 5 fold CV but it did not reflect on my LB. Can you share the batch size which was possible through your system.

", "votes": null, "replies": [] }, { "id": 968729, "author_name": "mpware", "author_url": "", "post_date": "08/13/2020 07:55:09", "content": "

Batch size = 48

", "votes": null, "replies": [ { "id": 970409, "author_name": "humblediscipulus", "author_url": "", "post_date": "08/14/2020 12:39:19", "content": "

We tried Efficient Net b5 with various combination of batch sizes and resolution, but we always ended up with a poor LB score, while during CV, our model performed really great. What resolution of image did u opt for these results?

", "votes": null, "replies": [] }, { "id": 970422, "author_name": "mpware", "author_url": "", "post_date": "08/14/2020 12:52:36", "content": "

Image size is 240x445. Training procedure is important, you need to add some augmentations (add noise, backgrounds, …). What is your CV?

\n

I've also trained a ResneSt50 with the same procedure and I got same LB even if CV was a bit better CV: 0.703/0.706/0.685/0.708/0.703.

\n

\"\"

\n

Also, main difficulty is that test set is soundscape with a lot background noise that differs from the train set (which is cleaner).

", "votes": null, "replies": [ { "id": 970564, "author_name": "humblediscipulus", "author_url": "", "post_date": "08/14/2020 15:06:01", "content": "

Seems like going with a shallower model and increasing batches helped u a lot. My best so far has been 0.62 (organic), we have tried a couple of noise augmentations but we are still loitering around that score.
\nThanx for the hint, kind sir. Cheers!!

\n

\"\"

\n

\"\"

\n

\"\"

\n

\"\"

", "votes": null, "replies": [] }, { "id": 981657, "author_name": "marcogorelli", "author_url": "", "post_date": "08/22/2020 15:57:45", "content": "

What do you mean by 'organic'?

", "votes": null, "replies": [] }, { "id": 985025, "author_name": "humblediscipulus", "author_url": "", "post_date": "08/25/2020 12:53:56", "content": "

Organic as in without using public kernels

", "votes": null, "replies": [] } ] }, { "id": 970830, "author_name": "kuldeepvansadia", "author_url": "", "post_date": "08/14/2020 21:09:48", "content": "

I have also tried adding background clips but I cannot figure with the noise to data ratio. If you can give a hint regarding augmentations and how you go about it will be very very helpful. Thanks

", "votes": null, "replies": [] }, { "id": 981555, "author_name": "rohitsingh9990", "author_url": "", "post_date": "08/22/2020 14:33:57", "content": "

@mpware Can u share how many epochs does it take for you to reach 0.70+ f1 score with Resnest50.

", "votes": null, "replies": [] }, { "id": 981682, "author_name": "mpware", "author_url": "", "post_date": "08/22/2020 16:26:15", "content": "

Epochs = 64

", "votes": null, "replies": [] }, { "id": 990208, "author_name": "rsinda", "author_url": "", "post_date": "08/29/2020 12:41:38", "content": "

@mpware , how much training time did one epoch took after adding augmentations(noise, pitching etc) ?. I am trying this but with kaggle GPU its taking 1 hr for 1 epoch.

", "votes": null, "replies": [] }, { "id": 990475, "author_name": "mpware", "author_url": "", "post_date": "08/29/2020 16:23:29", "content": "

It's around 8-10 hours for 64 epochs. I've an optimized pipeline running on 12 CPUs + 1 GPU.

", "votes": null, "replies": [] }, { "id": 990542, "author_name": "rsinda", "author_url": "", "post_date": "08/29/2020 17:20:51", "content": "

Thanks , I suppose , since you are also using P100 its a lot faster

", "votes": null, "replies": [] }, { "id": 990566, "author_name": "mpware", "author_url": "", "post_date": "08/29/2020 17:39:55", "content": "

Yes, P100.

", "votes": null, "replies": [] }, { "id": 990588, "author_name": "rsinda", "author_url": "", "post_date": "08/29/2020 17:56:50", "content": "

Also , It would be helpfull if you can share how you optimized your pipeline, when competition ends. It would help in future.

", "votes": null, "replies": [] }, { "id": 990693, "author_name": "fffrrt", "author_url": "", "post_date": "08/29/2020 19:30:13", "content": "

This competition and best-scored public kernel are heavily limited by CPU preprocessing.

\n

Reading mp3s every time you need a 5 second fragment - slowest (5-10% GPU usage)
\nReading zipped wavs every time you need a 5 second fragment - slow (20-30% GPU usage)
\nReading raw wavs every time you need a 5 second fragment - average (did not try it due to disk space requirement)

\n

Following steps require additional consideration with edge cases, data integrity and training behaviour and might negatively impact your score:
\nReading prepared spectrograms from disk - fast (50-90% GPU usage depending on format/compression, 1 channel bmps are fastest)
\nPlaying with prefetch settings/custom dataloaders/raw numpy arrays - fastest (should be able to 100% load your GPU, i did not go that far myself)

\n

For debugging, print elapsed time after each command in your getitem and after each completed batch, then consider if you really need to do all that stuff on the fly.

", "votes": null, "replies": [] } ] }, { "id": 983333, "author_name": "brianfeeny", "author_url": "", "post_date": "08/24/2020 07:42:54", "content": "

@mpware was your choice of batch size due to memory constraints or because you found that to work best in your model?

", "votes": null, "replies": [] }, { "id": 983911, "author_name": "mpware", "author_url": "", "post_date": "08/24/2020 17:21:54", "content": "

@brianfeeny Yes memory constraints, I'm trying to use the highest possible but with P100 GPU I'm limited. I've another model trained with Batch Size = 36 that scores similarly so I don't think there is a big difference with batch sizes within [36 - 48]

", "votes": null, "replies": [] }, { "id": 983941, "author_name": "tanulsingh077", "author_url": "", "post_date": "08/24/2020 17:44:23", "content": "

Thanks for sharing @mpware , I would be greatful if you answer the following questions

\n", "votes": null, "replies": [] }, { "id": 983983, "author_name": "mpware", "author_url": "", "post_date": "08/24/2020 18:30:56", "content": "

No PCEN. I've tried SED but it does not work for me. I mean it does not work better than a model with regular average/max pooling. SED just adds an attention pooling, it generates a lot of false positive so I guess training procedure is super important to benefit from it but I didn't figure out how. I have not tried other pooling solutions yet.

", "votes": null, "replies": [] }, { "id": 984123, "author_name": "brianfeeny", "author_url": "", "post_date": "08/24/2020 21:01:53", "content": "

@MPWARE have you tried AMP? It should give you plenty of memory head room and likely speed up training considerably.

", "votes": null, "replies": [] }, { "id": 984131, "author_name": "mpware", "author_url": "", "post_date": "08/24/2020 21:24:37", "content": "

Not tried yet, just wondering if it would work on inference kernel as it requires PyTorch 1.6, correct? Current Kaggle docker comes with PyTorch 1.5.1 and I've read that Apex installation was a pain for offline kernel.

", "votes": null, "replies": [] }, { "id": 984157, "author_name": "brianfeeny", "author_url": "", "post_date": "08/24/2020 22:18:58", "content": "

@MPWARE good point I was not aware of that. I am not sure if you can train in fp16 and then do prediction in fp32. The main benefit of AMP is in training. With inference, we can use large batch sizes, with no negative effect, since we are not needing to store gradients/backprop.

", "votes": null, "replies": [] }, { "id": 985095, "author_name": "nacirbouazizi", "author_url": "", "post_date": "08/25/2020 13:57:10", "content": "

@mpware could you please give some insights or ressources to do augmentations ?

", "votes": null, "replies": [] }, { "id": 987415, "author_name": "roguekk007", "author_url": "", "post_date": "08/27/2020 08:33:55", "content": "

@mpware Thanks for your generous sharing. If you don't mind disclosing:

\n
    \n
  1. Do you validate using 5s clips randomly cropped from each audio? (If I understand correctly, you are also using clips of this format to train).
    2. \n
  2. For calculation of f1, do you use threshold of .5?
    3. \n
  3. For calculation of f1, how do you account for \"nocall\" class? Or just calculate f1 without \"nocall\" considered?
    4. \n
\n

By the way, for installation of apex, maybe you can do this:

\n
%%bash\ngit clone https://github.com/NVIDIA/apex (replace it with .zip you download from github)\ncd apex\npip install -v --no-cache-dir --global-option=\"--cpp_ext\" --global-option=\"--cuda_ext\" ./\n
\n

Thanks in advance

", "votes": null, "replies": [] }, { "id": 987775, "author_name": "mpware", "author_url": "", "post_date": "08/27/2020 14:03:09", "content": "

@roguekk007

\n
    \n
  1. I pick 5s randomly for train, but for eval I used the first 5s only.
    2. \n
  2. For F1, yes, threshold=0.5. I'm also monitoring mAP.
    3. \n
  3. I've some models with 264 classes and some with 265 including nocall class (with data including noise). I would expect 265 classes models to work quite better than 264 ones but it's quite similar.
    4. \n
", "votes": null, "replies": [] }, { "id": 988003, "author_name": "rguo97", "author_url": "", "post_date": "08/27/2020 17:33:18", "content": "

@mpware , Did you notice higher F1 in validation when you lowering the threshold? In my case, 0.3 threshold is 0.02-0.03 higher than 0.5 threshold. But this is inverse on LB.

", "votes": null, "replies": [] }, { "id": 988007, "author_name": "rohitsingh9990", "author_url": "", "post_date": "08/27/2020 17:34:44", "content": "

@rguo97 you are right, i noticed the same.

", "votes": null, "replies": [] }, { "id": 988048, "author_name": "mpware", "author_url": "", "post_date": "08/27/2020 18:18:37", "content": "

Are you also monitoring mAP, is it correlated to what you describe?

", "votes": null, "replies": [] }, { "id": 988392, "author_name": "rguo97", "author_url": "", "post_date": "08/28/2020 02:40:12", "content": "

mAP do not need threshold. I got around 0.73 mAP in my models, but I cannot get a stable LB…

", "votes": null, "replies": [] }, { "id": 988607, "author_name": "tonychenxyz", "author_url": "", "post_date": "08/28/2020 06:41:59", "content": "

@mpware Thanks for sharing this.

\n
\n

I've some models with 264 classes and some with 265 including nocall class (with data including noise). I would expect 265 classes models to work quite better than 264 ones but it's quite similar.

\n
\n

I was wondering what are you referreing to by data including noise?

\n

Thanks!

", "votes": null, "replies": [] }, { "id": 988613, "author_name": "mpware", "author_url": "", "post_date": "08/28/2020 06:46:47", "content": "

You can try to split data to isolate noise/background, then you can use it as another \"nocall\" class.

", "votes": null, "replies": [] }, { "id": 988615, "author_name": "mpware", "author_url": "", "post_date": "08/28/2020 06:49:30", "content": "

@rguo97 Yes, but when F1 score is high with 0.3 threshold, is mAP score also high for the same epoch?

", "votes": null, "replies": [] }, { "id": 989348, "author_name": "rguo97", "author_url": "", "post_date": "08/28/2020 18:21:26", "content": "

Yes. They are correlated.

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "962348": "Didn't see this post in this competition. Thought might be important.", "962444": "Yep. If we have some reasonable validation policy", "964256": "Single EfficientNet B1 CV5 model (**S**tate **O**f the **A**rt model with 5 seconds slices followed by MEL-spectrogram to feed CNN):\nCV=0.672/0.675/0.667/0.668/0.685 \nLB=0.572", "964537": "Thanks for sharing. What does CV5 mean? Also, could you share your CV strategy? @mpware", "964879": "Cross Validation with 5 folds. I'm using `StratifiedKFold` (K=5) to balance target among each folds. Nothing more than a regular CV strategy. `CosineAnnealingLR` is used too (1 cycle). I've also removed too short (to have enough data) or too long clips (to speed up training). Here is one fold example:\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F698363%2F90316b6aa7f2a8271044563c12568e93%2Ftrain.png?generation=1597049654653056&alt=media)", "965023": "mpware That's amazing. Thank you for replying. Are you using sklearn for f1 calculation? or something else.", "965032": "Yes,\n\n```\nAVG = 'samples'\n\ndef f1(y_true, y_pred):\n return metrics.f1_score(y_true, y_pred, average=AVG) \n```", "966190": "mpware Thanks!", "967778": "Thanks for your insight @mpware . Glad to see you in this comp .", "968350": "Thanks for sharing I have also tried Efficient Net B1 with 5 fold CV but it did not reflect on my LB. Can you share the batch size which was possible through your system.", "968729": "Batch size = 48", "970312": "Thanks for sharing! That's good, that even with roughly speaking constant train loss, you get increasing train metrics.", "970409": "We tried Efficient Net b5 with various combination of batch sizes and resolution, but we always ended up with a poor LB score, while during CV, our model performed really great. What resolution of image did u opt for these results?", "970422": "Image size is 240x445. Training procedure is important, you need to add some augmentations (add noise, backgrounds, ...). What is your CV? \n\nI've also trained a ResneSt50 with the same procedure and I got same LB even if CV was a bit better CV: 0.703/0.706/0.685/0.708/0.703.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F698363%2Fd1742929a2dc2be03cb747dd1d1991a6%2Ftrain.png?generation=1597409518166977&alt=media)\n\nAlso, main difficulty is that test set is soundscape with a lot background noise that differs from the train set (which is cleaner).", "970564": "Seems like going with a shallower model and increasing batches helped u a lot. My best so far has been 0.62 (organic), we have tried a couple of noise augmentations but we are still loitering around that score.\nThanx for the hint, kind sir. Cheers!!\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1604687%2Fef0384238ad2650804f1b090f6390834%2Fscreenshot-app.wandb.ai-2020.08.14-20_31_00.png?generation=1597417293673418&alt=media)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1604687%2Ffc70d306a57e5148b091dbf48ef6db89%2Fscreenshot-app.wandb.ai-2020.08.14-20_30_47.png?generation=1597417347235344&alt=media)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1604687%2F6c02b010dd032b2a717e9642fd8aafd3%2Fscreenshot-app.wandb.ai-2020.08.14-20_30_13.png?generation=1597417374634914&alt=media)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1604687%2F929b9555cd0d5f10ec70a261a7579f4d%2Fscreenshot-app.wandb.ai-2020.08.14-20_29_50.png?generation=1597417425828374&alt=media)", "970830": "I have also tried adding background clips but I cannot figure with the noise to data ratio. If you can give a hint regarding augmentations and how you go about it will be very very helpful. Thanks", "981555": "mpware Can u share how many epochs does it take for you to reach 0.70+ f1 score with Resnest50.", "981657": "What do you mean by 'organic'?", "981682": "Epochs = 64", "983333": "mpware was your choice of batch size due to memory constraints or because you found that to work best in your model?", "983911": "brianfeeny Yes memory constraints, I'm trying to use the highest possible but with P100 GPU I'm limited. I've another model trained with Batch Size = 36 that scores similarly so I don't think there is a big difference with batch sizes within [36 - 48]", "983941": "Thanks for sharing @mpware , I would be greatful if you answer the following questions\n* Did you try PCEN mel spectograms?\n* Did you try SED ?\nWhat do you think about the large Cv/LB gap?", "983983": "No PCEN. I've tried SED but it does not work for me. I mean it does not work better than a model with regular average/max pooling. SED just adds an attention pooling, it generates a lot of false positive so I guess training procedure is super important to benefit from it but I didn't figure out how. I have not tried other pooling solutions yet.", "984123": "MPWARE have you tried AMP? It should give you plenty of memory head room and likely speed up training considerably.", "984131": "Not tried yet, just wondering if it would work on inference kernel as it requires [PyTorch 1.6](https://pytorch.org/blog/accelerating-training-on-nvidia-gpus-with-pytorch-automatic-mixed-precision/), correct? Current Kaggle docker comes with PyTorch 1.5.1 and I've read that Apex installation was a pain for offline kernel.", "984157": "MPWARE good point I was not aware of that. I am not sure if you can train in fp16 and then do prediction in fp32. The main benefit of AMP is in training. With inference, we can use large batch sizes, with no negative effect, since we are not needing to store gradients/backprop.", "984659": "Single Resnest50, single fold CV(f1_score)=0.6877, LB:0.570. (Thanks to @mpware )\n\n> [UPDATE]: CV(5folds): 0.687/0.684/0.683/0.695/0.691 , LB: 0.572", "984677": "CV against what? Just the regular sounds you are training on? Or against BirdCheck or what? What metric are you measuring for CV?", "984736": "brianfeeny everything is almost same as what @mpware mentioned.", "984789": "Thanks I didn't see what you all are measuring though to see how your model is converging", "984806": "rohitsingh9990 Can you please tell if you added background noise etc , or you have got this just by changing hyperparameters??", "984808": "brianfeeny we all measuring f1_score. as already mentioned above by mpware\n\n```\nAVG = 'samples'\n\ndef f1(y_true, y_pred):\n return metrics.f1_score(y_true, y_pred, average=AVG)\n```", "984811": "tanulsingh077 yes, i added a lot of augmentations, and one of them is noise addition. Augmentations are really helpful in this task, if you apply them correctly.", "985025": "Organic as in without using public kernels", "985095": "mpware could you please give some insights or ressources to do augmentations ?", "985503": "So when you all do your multiple folds, are you combining/averaging your class probabilities and then predicting classes or do you let each fold predict classes and then do some sort of \"vote\" such as 2 of 4, or 3 of 4 folds agree etc?", "986917": "Did you apply a threshold on prediction and use the 'samples' F1 score?", "987415": "mpware Thanks for your generous sharing. If you don't mind disclosing:\n1. Do you validate using 5s clips randomly cropped from each audio? (If I understand correctly, you are also using clips of this format to train).\n2. For calculation of f1, do you use threshold of .5?\n3. For calculation of f1, how do you account for \"nocall\" class? Or just calculate f1 without \"nocall\" considered?\n\nBy the way, for installation of apex, maybe you can do this:\n```\n%%bash\ngit clone https://github.com/NVIDIA/apex (replace it with .zip you download from github)\ncd apex\npip install -v --no-cache-dir --global-option=\"--cpp_ext\" --global-option=\"--cuda_ext\" ./\n```\nThanks in advance", "987444": "rguo97 not in current implementation, but i am planning to do that and will share the results.", "987775": "roguekk007 \n1. I pick 5s randomly for train, but for eval I used the first 5s only. \n2. For F1, yes, threshold=0.5. I'm also monitoring mAP.\n3. I've some models with 264 classes and some with 265 including nocall class (with data including noise). I would expect 265 classes models to work quite better than 264 ones but it's quite similar.", "988003": "mpware , Did you notice higher F1 in validation when you lowering the threshold? In my case, 0.3 threshold is 0.02-0.03 higher than 0.5 threshold. But this is inverse on LB.", "988007": "rguo97 you are right, i noticed the same.", "988048": "Are you also monitoring mAP, is it correlated to what you describe?", "988392": "mAP do not need threshold. I got around 0.73 mAP in my models, but I cannot get a stable LB...", "988607": "mpware Thanks for sharing this.\n\n> I've some models with 264 classes and some with 265 including nocall class (with data including noise). I would expect 265 classes models to work quite better than 264 ones but it's quite similar.\n\nI was wondering what are you referreing to by data including noise?\n\nThanks!", "988613": "You can try to split data to isolate noise/background, then you can use it as another \"nocall\" class.", "988615": "rguo97 Yes, but when F1 score is high with 0.3 threshold, is mAP score also high for the same epoch?", "988637": "I'm seeing a lot of f1 scores in the cv here, which is great, but I'm wondering how you guys are choosing your thresholds for the f1 score...", "989348": "Yes. They are correlated.", "990208": "mpware , how much training time did one epoch took after adding augmentations(noise, pitching etc) ?. I am trying this but with kaggle GPU its taking 1 hr for 1 epoch.", "990224": "Felt like sharing our current LB CV scores, although they don't really mean anything.\n\nI use stratified 5-folds as everyone else, but when making the prediction I crop the 5 first second of the clip (I prefered this instead of randomly chosing a 5s crop, for reproducibility), which is why my CV scores are higher than those reported by everyone else.\n\n\n```\nFold 1 : micro_f1=0.809 \t samples_f1=0.757\nFold 2 : micro_f1=0.807 \t samples_f1=0.762\nFold 3 : micro_f1=0.809 \t samples_f1=0.755\nFold 4 : micro_f1=0.808 \t samples_f1=0.757\nFold 5 : micro_f1=0.800 \t samples_f1=0.747\n```\n> LB : 0.617", "990297": "Did the 0.617 of LB score come from 5fold ensemble?", "990417": "Yep, that's the 5 folds described above", "990475": "It's around 8-10 hours for 64 epochs. I've an optimized pipeline running on 12 CPUs + 1 GPU.", "990484": "hidehisaarai1213 Is your 0.618 LB sub a single fold, 5 folds, or did you start some blending ?", "990542": "Thanks , I suppose , since you are also using P100 its a lot faster", "990566": "Yes, P100.", "990588": "Also , It would be helpfull if you can share how you optimized your pipeline, **when competition ends**. It would help in future.", "990693": "This competition and best-scored public kernel are heavily limited by CPU preprocessing.\n\nReading mp3s every time you need a 5 second fragment - slowest (5-10% GPU usage)\nReading zipped wavs every time you need a 5 second fragment - slow (20-30% GPU usage)\nReading raw wavs every time you need a 5 second fragment - average (did not try it due to disk space requirement)\n\nFollowing steps require additional consideration with edge cases, data integrity and training behaviour and might negatively impact your score:\nReading prepared spectrograms from disk - fast (50-90% GPU usage depending on format/compression, 1 channel bmps are fastest)\nPlaying with prefetch settings/custom dataloaders/raw numpy arrays - fastest (should be able to 100% load your GPU, i did not go that far myself)\n\nFor debugging, print elapsed time after each command in your __getitem__ and after each completed batch, then consider if you _really_ need to do all that stuff on the fly.", "990856": "Thx for sharing!\nMy 0.618 score is a blend of two single fold models (same arch, different settings)." }, "source": "meta" }