{ "id": 176959, "title": "Any hints to beat the 0.568 baseline?", "url": "/competitions/birdsong-recognition/discussion/176959", "author_name": "Salaryman", "post_date": "2020-08-24T08:46:12.989000", "votes": 28, "comment_count": 63, "views": 0, "content": "

Among 1019 teams only 50 teams can beat the baseline ach >= 0.569 in public LB so far.

\n

Any hints to beat the 0.568 baseline?🙏🙏
\nfinetune the CNN? PANN? other ensemble structure? proper augmentation?
\nexternal dataset?

\n

Much appreciate!
\nThanks so much!

", "messages": [ { "id": 983376, "postDate": "2020-08-24T08:46:12.990Z", "content": "

Among 1019 teams only 50 teams can beat the baseline ach >= 0.569 in public LB so far.

\n

Any hints to beat the 0.568 baseline?🙏🙏
\nfinetune the CNN? PANN? other ensemble structure? proper augmentation?
\nexternal dataset?

\n

Much appreciate!
\nThanks so much!

", "rawMarkdown": "Among 1019 teams only 50 teams can beat the baseline ach >= 0.569 in public LB so far.\n\nAny hints to beat the 0.568 baseline?🙏🙏\nfinetune the CNN? PANN? other ensemble structure? proper augmentation?\nexternal dataset?\n\nMuch appreciate!\nThanks so much!\n ", "votes": 28 }, { "id": 983419, "postDate": "2020-08-24T09:44:02.147Z", "content": "

Single CNN model with melspectrograms (derived from public notebooks), with no augmentation, no mixup, using only primary label, with slightly modified training and inference mechanisms can get you to 0.575 Public LB.

\n

Echoing what others have said in the forum, the biggest gain I've seen so far did not come from model architecture or ensembling, rather from how training and inference methods addresses the things that make this competition challenging.

", "rawMarkdown": "Single CNN model with melspectrograms (derived from public notebooks), with no augmentation, no mixup, using only primary label, with slightly modified training and inference mechanisms can get you to 0.575 Public LB. \n\nEchoing what others have said in the forum, the biggest gain I've seen so far did not come from model architecture or ensembling, rather from how training and inference methods addresses the things that make this competition challenging.", "votes": 26, "replies": [ { "id": 983471, "postDate": "2020-08-24T10:29:47.070Z", "content": "

Can you give any insight about your training mechanism?

", "rawMarkdown": "Can you give any insight about your training mechanism?", "votes": -2 }, { "id": 983693, "postDate": "2020-08-24T14:15:54.223Z", "content": "

@ramarlina Thanks so much!
\nI will stick on fine tuning a single CNN and hope to beat the baseline😃

", "rawMarkdown": "@ramarlina Thanks so much! \nI will stick on fine tuning a single CNN and hope to beat the baseline😃", "votes": 2 }, { "id": 984154, "postDate": "2020-08-24T22:05:54.810Z", "content": "

Thanks for sharing mate!

", "rawMarkdown": "Thanks for sharing mate! " }, { "id": 986492, "postDate": "2020-08-26T14:48:14.077Z", "content": "

@ramarlina Can you please tell which architecture you are using ?? is it possible to go above 0.57+ with an effnet??

", "rawMarkdown": "@ramarlina Can you please tell which architecture you are using ?? is it possible to go above 0.57+ with an effnet??" }, { "id": 986560, "postDate": "2020-08-26T15:32:50.610Z", "content": "

@tanulsingh077 I am trying my reliable effnet only.

", "rawMarkdown": "@tanulsingh077 I am trying my reliable effnet only." }, { "id": 986623, "postDate": "2020-08-26T16:30:17.827Z", "content": "

@tanulsingh077 I'm using a resnet right now, but my CV tells me that you could get comparable results (potentially better) with an effnet. I'm not focused on selecting the most optimal backbone yet at this point.

", "rawMarkdown": "@tanulsingh077 I'm using a resnet right now, but my CV tells me that you could get comparable results (potentially better) with an effnet. I'm not focused on selecting the most optimal backbone yet at this point.", "votes": 5 }, { "id": 986631, "postDate": "2020-08-26T16:33:15.177Z", "content": "

@tanulsingh077 It is possible to beat 0.57 with an EfficientNet

", "rawMarkdown": "@tanulsingh077 It is possible to beat 0.57 with an EfficientNet ", "votes": 5 }, { "id": 986643, "postDate": "2020-08-26T16:50:35.870Z", "content": "

Thanks @ramarlina and @theoviel for the answer . Also @theoviel Is it possible to do that without augmentations , mixups , introducing noise etc and with single label multiclass approach ?? I have been trying from two days but my cv f1 score remains at 0 , even the train f1 score remains at zero , I am using everything same as public notebooks ☹️

", "rawMarkdown": "Thanks @ramarlina and @theoviel for the answer . Also @theoviel Is it possible to do that without augmentations , mixups , introducing noise etc and with single label multiclass approach ?? I have been trying from two days but my cv f1 score remains at 0 , even the train f1 score remains at zero , I am using everything same as public notebooks ☹️" }, { "id": 986663, "postDate": "2020-08-26T17:09:40.367Z", "content": "

even my cv f1 score is not 0… the model I trained for 2 days just got LB=0.0😂😂😂
\nshould be sth wrong with the prediction loop/ mapping

", "rawMarkdown": "even my cv f1 score is not 0... the model I trained for 2 days just got LB=0.0😂😂😂\nshould be sth wrong with the prediction loop/ mapping" }, { "id": 986676, "postDate": "2020-08-26T17:17:41.360Z", "content": "

@fiyeroleung

\n

`
\nclass BirdcallModel(nn.Module):
\n def init(self,num_classes):
\n super(BirdcallModel, self).init()

\n
    self.num_classes = num_classes\n    self.model = EfficientNet.from_pretrained(\n        f\"efficientnet-b{EFFNET_VER}\"\n    )\n\n    self.model._fc = nn.Sequential(\n        nn.Linear(1280, 1024), nn.ReLU(), nn.Dropout(p=0.2),\n        nn.Linear(1024, 1024), nn.ReLU(), nn.Dropout(p=0.2),\n        nn.Linear(1024, self.num_classes))\n\ndef forward(self, image):\n    return self.model(image)\n
\n

`

\n

This is the model and then I take softmax followed by argmax to get the predicted values , I don't know what's going wrong

", "rawMarkdown": "@fiyeroleung \n\n\n`\nclass BirdcallModel(nn.Module):\n def __init__(self,num_classes):\n super(BirdcallModel, self).__init__()\n \n self.num_classes = num_classes\n self.model = EfficientNet.from_pretrained(\n f\"efficientnet-b{EFFNET_VER}\"\n )\n \n self.model._fc = nn.Sequential(\n nn.Linear(1280, 1024), nn.ReLU(), nn.Dropout(p=0.2),\n nn.Linear(1024, 1024), nn.ReLU(), nn.Dropout(p=0.2),\n nn.Linear(1024, self.num_classes))\n \n def forward(self, image):\n return self.model(image)\n\n`\n\nThis is the model and then I take softmax followed by argmax to get the predicted values , I don't know what's going wrong" }, { "id": 986700, "postDate": "2020-08-26T17:39:07.307Z", "rawMarkdown": "", "isDeleted": true }, { "id": 986702, "postDate": "2020-08-26T17:43:56.337Z", "content": "

@tanulsingh077 Don't get me wrong, I also have the problem of LB=0😂
\nim reading line by line to see which part goes wrong

", "rawMarkdown": "@tanulsingh077 Don't get me wrong, I also have the problem of LB=0😂\nim reading line by line to see which part goes wrong" }, { "id": 987001, "postDate": "2020-08-26T22:44:16.497Z", "content": "

If your f1 score remains at 0 you probably have an implementation mistake somewhere

", "rawMarkdown": "If your f1 score remains at 0 you probably have an implementation mistake somewhere" }, { "id": 987214, "postDate": "2020-08-27T05:03:05.400Z", "content": "

@ramarlina no augmentation mean that you use the random audio with duration (s) or you cut/ padding with the fix length ?

", "rawMarkdown": "@ramarlina no augmentation mean that you use the random audio with duration (s) or you cut/ padding with the fix length ?", "votes": 1 }, { "id": 987281, "postDate": "2020-08-27T06:16:54.763Z", "content": "

Data augmentations techniques for spectrograms like warping, masking blocks of frequency channels, and masking blocks of time steps. Maybe it's my implementation, but it hasn't helped my CV so far.

", "rawMarkdown": "Data augmentations techniques for spectrograms like warping, masking blocks of frequency channels, and masking blocks of time steps. Maybe it's my implementation, but it hasn't helped my CV so far." }, { "id": 987346, "postDate": "2020-08-27T07:15:14.727Z", "content": "

Sorry! But how did you choose 5seconds.

", "rawMarkdown": "Sorry! But how did you choose 5seconds." }, { "id": 987423, "postDate": "2020-08-27T08:38:29.993Z", "content": "

I've experimented with a lot of things to ultimately resort back to random sampling clips for training.

", "rawMarkdown": "I've experimented with a lot of things to ultimately resort back to random sampling clips for training. ", "votes": 1 }, { "id": 987440, "postDate": "2020-08-27T09:05:01.073Z", "content": "

@theoviel @ramarlina I am really sorry I am asking , but its been three days and I have not been able to solve this , I will be highly grateful if you can have a look at my code here
\nhttps://drive.google.com/file/d/1O3eBTMcqV4d7jp0DhRSFwwBQ2ySLcjga/view?usp=sharing

\n

I am really sorry for asking for this , I don't want spoon feeding but I just can't figure out what's wrong and I have also wasted this week's gpu quota as well

", "rawMarkdown": "@theoviel @ramarlina I am really sorry I am asking , but its been three days and I have not been able to solve this , I will be highly grateful if you can have a look at my code here\nhttps://drive.google.com/file/d/1O3eBTMcqV4d7jp0DhRSFwwBQ2ySLcjga/view?usp=sharing\n\nI am really sorry for asking for this , I don't want spoon feeding but I just can't figure out what's wrong and I have also wasted this week's gpu quota as well" }, { "id": 987452, "postDate": "2020-08-27T09:20:11.113Z", "content": "

@tanulsingh077 been there, I have 4 hours of GPU left, so I feel ya. Are you having any specific issue?

", "rawMarkdown": "@tanulsingh077 been there, I have ~~5 hours~~ 4 hours of GPU left, so I feel ya. Are you having any specific issue?" }, { "id": 987455, "postDate": "2020-08-27T09:23:28.590Z", "content": "

Yes @ramarlina My f1 score remains at zero and just doesn't increase no matter which model I use , I have read the training code , evaluation code , dataset code , f1 score implementation code line by line ,but I can't find what's wrong , I am not doing anything fancy but still

", "rawMarkdown": "Yes @ramarlina My f1 score remains at zero and just doesn't increase no matter which model I use , I have read the training code , evaluation code , dataset code , f1 score implementation code line by line ,but I can't find what's wrong , I am not doing anything fancy but still" }, { "id": 987471, "postDate": "2020-08-27T09:43:44.893Z", "content": "

Does your training loss improve? How long did you run the model?

\n

One thing I can point out immediately is that, I only iterate the learning rate after the model completes an epoch. I call scheduler.step() outside and after the for loop.

", "rawMarkdown": "Does your training loss improve? How long did you run the model?\n\nOne thing I can point out immediately is that, I only iterate the learning rate after the model completes an epoch. I call scheduler.step() outside and after the for loop." }, { "id": 987482, "postDate": "2020-08-27T09:49:56.693Z", "content": "

I run it for 50 epochs , my train_loss decreases and valid loss also decreases , thanks for the scheduler tip I will us it

", "rawMarkdown": "I run it for 50 epochs , my train_loss decreases and valid loss also decreases , thanks for the scheduler tip I will us it" }, { "id": 987492, "postDate": "2020-08-27T09:56:37.087Z", "content": "

It's probably an issue with your f1 computation. What's your best val loss after 50 epochs? Also do you mean you get 0.0 LB f1 or 0.0 validation f1?

", "rawMarkdown": "It's probably an issue with your f1 computation. What's your best val loss after 50 epochs? Also do you mean you get 0.0 LB f1 or 0.0 validation f1?" }, { "id": 987551, "postDate": "2020-08-27T10:37:59.410Z", "content": "

Trying with many mods so that it runs on my 1080, maybe I could reproduce your issue. Is that the Train_F1_score? It soon increases to around 0.5 and decreases later on.

\n
\n
######### Starting Training for epoch 1
\n

1%| | 11/1066 [00:11<15:24, 1.14it/s, Train_F1_score=0.24, Train_Loss=0.225]

\n
\n

Please don't hope too much, but would be able to take some time to look into tonight. But again, please don't expect me to.

", "rawMarkdown": "Trying with many mods so that it runs on my 1080, maybe I could reproduce your issue. Is that the Train_F1_score? It soon increases to around 0.5 and decreases later on.\n\n> ############### Starting Training for epoch 1 #################\n> 1%| | 11/1066 [00:11<15:24, 1.14it/s, Train_F1_score=0.24, Train_Loss=0.225] \n\nPlease don't hope too much, but would be able to take some time to look into tonight. But again, please don't expect me to." }, { "id": 987571, "postDate": "2020-08-27T10:58:22.250Z", "content": "

@tanulsingh077 If I print your model's prediction stat like this, model is predicting smaller values.
\nIn F1Meter:

\n
    def update(self, y_true, y_pred,threshold=0.5):\n        y_true = y_true.detach().cpu().numpy()\n        yp = y_pred.detach().cpu().numpy()\n        print(yp.min(), yp.mean(), yp.max()) # <<<<<<<==== debugging here\n        y_pred = F.sigmoid(y_pred).detach().cpu().numpy()\n\n        y_pred[y_pred<threshold] = 0\n        y_pred[y_pred>threshold] = 1\n
\n

Printed like this, then many values after sigmoid() will be lower than threshold -> almost zero y_pred
\n -> less TP -> F1 might be smaller. it might be the direct reason for that.

\n
-0.30662918 0.00048712618 0.29636607\n  0%|          | 2/1066 [00:04<56:05,  3.16s/it, Train_F1_score=0.0428, Train_Loss=0.668]    \n-1.2256229 -0.10857863 0.36215597\n  0%|          | 3/1066 [00:05<42:13,  2.38s/it, Train_F1_score=0.238, Train_Loss=0.609] \n-3.2570004 -0.4962771 1.3743259\n  0%|          | 4/1066 [00:06<32:42,  1.85s/it, Train_F1_score=0.429, Train_Loss=0.518]\n-4.288979 -1.3962975 2.694023\n  :\n-41.389378 -20.891447 -0.23979145\n  1%|          | 11/1066 [00:11<16:00,  1.10it/s, Train_F1_score=0.247, Train_Loss=0.227]\n-57.430416 -22.627079 -0.5330948\n
\n

Did you check your data generated by Dataset is healthy?

", "rawMarkdown": "@tanulsingh077 If I print your model's prediction stat like this, model is predicting smaller values.\nIn F1Meter:\n```\n def update(self, y_true, y_pred,threshold=0.5):\n y_true = y_true.detach().cpu().numpy()\n yp = y_pred.detach().cpu().numpy()\n print(yp.min(), yp.mean(), yp.max()) # <<<<<<<==== debugging here\n y_pred = F.sigmoid(y_pred).detach().cpu().numpy()\n\n y_pred[y_predthreshold] = 1\n```\nPrinted like this, then many values after sigmoid() will be lower than threshold -> almost zero y_pred \n -> less TP -> F1 might be smaller. it might be the direct reason for that.\n```\n-0.30662918 0.00048712618 0.29636607\n 0%| | 2/1066 [00:04<56:05, 3.16s/it, Train_F1_score=0.0428, Train_Loss=0.668] \n-1.2256229 -0.10857863 0.36215597\n 0%| | 3/1066 [00:05<42:13, 2.38s/it, Train_F1_score=0.238, Train_Loss=0.609] \n-3.2570004 -0.4962771 1.3743259\n 0%| | 4/1066 [00:06<32:42, 1.85s/it, Train_F1_score=0.429, Train_Loss=0.518]\n-4.288979 -1.3962975 2.694023\n :\n-41.389378 -20.891447 -0.23979145\n 1%| | 11/1066 [00:11<16:00, 1.10it/s, Train_F1_score=0.247, Train_Loss=0.227]\n-57.430416 -22.627079 -0.5330948\n```\n\nDid you check your data generated by Dataset is healthy?", "votes": 2 }, { "id": 987605, "postDate": "2020-08-27T11:42:33Z", "content": "

Hello @daisukelab thanks for taking out the time to help me out , I really appreciate it . As you can see I am using the dataset class from public kernel without any changes I don't think that the dataset is corrupted ,and for confirmation I have checked it again

", "rawMarkdown": "Hello @daisukelab thanks for taking out the time to help me out , I really appreciate it . As you can see I am using the dataset class from public kernel without any changes I don't think that the dataset is corrupted ,and for confirmation I have checked it again " }, { "id": 987639, "postDate": "2020-08-27T12:17:05.230Z", "content": "

One small problem, DataLoader's shuffle is False by default. Maybe everybody just follow the notebook could be shuffle=False for training loader….

", "rawMarkdown": "One small problem, DataLoader's shuffle is False by default. Maybe everybody just follow the notebook could be shuffle=False for training loader....", "votes": 1 }, { "id": 987697, "postDate": "2020-08-27T13:00:29.050Z", "content": "

@daisukelab @ramarlina I have tried measuring F1 score between epochs of this public kernel
\nhttps://www.kaggle.com/ttahara/training-birdsong-baseline-resnest50-fast
\nand it also gives zero f1 score , Now I am more confused , Just to check if my f1 meter class is not wring I tried measuring F1 at the end of each epoch using sklearn's f1 score but it suffered the same fate

", "rawMarkdown": "@daisukelab @ramarlina I have tried measuring F1 score between epochs of this public kernel \nhttps://www.kaggle.com/ttahara/training-birdsong-baseline-resnest50-fast\nand it also gives zero f1 score , Now I am more confused , Just to check if my f1 meter class is not wring I tried measuring F1 at the end of each epoch using sklearn's f1 score but it suffered the same fate" }, { "id": 987714, "postDate": "2020-08-27T13:13:27.397Z", "content": "

In my case F1score is decreasing with training

", "rawMarkdown": "In my case F1score is decreasing with training" }, { "id": 987728, "postDate": "2020-08-27T13:25:39.997Z", "content": "

Keep in mind that, with this training procedure, one \"epoch\" means the model has \"seen\" exactly a single random chunk of 5s from each audio in your dataset. The average duration of each file is 55s, that means you can expect the model to have seen ~9% of your data within one epoch. You'll need at least 12 epochs to complete the equivalent of a single full pass over the data. On top of that, the model is looking at random timestamps within the audio, is the model actually looking at birdcalls? noise? silence?

\n

TLDR; you'll need to let the model run for a few epochs before it can pick anything significant.

", "rawMarkdown": "Keep in mind that, with this training procedure, one \"epoch\" means the model has \"seen\" exactly a single random chunk of 5s from each audio in your dataset. The average duration of each file is 55s, that means you can expect the model to have seen ~9% of your data within one epoch. You'll need at least 12 epochs to complete the equivalent of a single full pass over the data. On top of that, the model is looking at random timestamps within the audio, is the model actually looking at birdcalls? noise? silence? \n\nTLDR; you'll need to let the model run for a few epochs before it can pick anything significant.", "votes": 5 }, { "id": 988025, "postDate": "2020-08-27T17:52:53.763Z", "content": "

@tanulsingh077 I made further check - but not solved. Here's what have found so far.

\n\n

Umm.. my time is up. :(

\n

One last thing I can suggest is, we have this basic.

\n
\n

torch.Tensor([[-1, -0.1, 0, 0.1, 1]]).sigmoid() # ==> [[0.0474, 0.2689, 0.4750, 0.5000, 0.5250, 0.7311]]

\n
\n

As long as the F1 calculation threshould is 0.5, your model has to predict 0 or larger, but it outputs minus for the target label class. I would follow the basics if I could continue:

\n\n

Good luck…

", "rawMarkdown": "@tanulsingh077 I made further check - but not solved. Here's what have found so far.\n\n- Generated image looks wrong? -> Negative, visualized fine.\n- Image goes beyond normal value range? -> Negative, confirmed to be [0, 1] range.\n- F1 calculation goes wrong? -> Negative, it worked fine when using it with other correctly running program.\n\nUmm.. my time is up. :(\n\nOne last thing I can suggest is, we have this basic.\n\n> torch.Tensor([[-1, -0.1, 0, 0.1, 1]]).sigmoid() # ==> [[0.0474, 0.2689, 0.4750, 0.5000, 0.5250, 0.7311]]\n\nAs long as the F1 calculation threshould is 0.5, your model has to predict 0 or larger, but it outputs minus for the target label class. I would follow the basics if I could continue:\n\n- Debug the rest of things, confirm inputs/outputs are correct.\n- Replace each parts with working-for-sure existing ones.\n\nGood luck...", "votes": 1 }, { "id": 988160, "postDate": "2020-08-27T20:34:00.553Z", "content": "

Thanks @daisukelab for helping me with this and your suggestions ,It's because of people like you ,people like me get the oppurtunity to flourish, I will work my way again through the code , if I am not able to debug it , guess I will use the public kernels then

", "rawMarkdown": "Thanks @daisukelab for helping me with this and your suggestions ,It's because of people like you ,people like me get the oppurtunity to flourish, I will work my way again through the code , if I am not able to debug it , guess I will use the public kernels then " }, { "id": 988359, "postDate": "2020-08-28T02:03:50.887Z", "content": "

@tanulsingh077 here is a kernel that implements an F1 that seems to work with no issues in this competition. Maybe that may help you.

", "rawMarkdown": "@tanulsingh077 [here](https://github.com/earthspecies/birdcall/blob/master/02j_train_on_melspectrograms_pytorch_lme_pool_all_classes_simple.ipynb) is a kernel that implements an F1 that seems to work with no issues in this competition. Maybe that may help you.", "votes": 1 }, { "id": 988451, "postDate": "2020-08-28T04:08:43.363Z", "content": "

@nacirbouazizi I saw your kernel. There your model expect input shape of (bs,im_num, channel, height, width) what is im_num here

", "rawMarkdown": "@nacirbouazizi I saw your kernel. There your model expect input shape of (bs,im_num, channel, height, width) what is im_num here" }, { "id": 989145, "postDate": "2020-08-28T15:19:50.583Z", "content": "

Thanks @nacirbouazizi I will try it

", "rawMarkdown": "Thanks @nacirbouazizi I will try it ", "votes": 1 }, { "id": 1001421, "postDate": "2020-09-07T10:16:41.090Z", "content": "

@tanulsingh077
\nI saw your code. Your code namespace has 2 \"f1_score\" variables - One from sklearn and the other that you defined.

\n

If I were you, I would check if this namespace confusion is causing a problem.

", "rawMarkdown": "@tanulsingh077 \nI saw your code. Your code namespace has 2 \"f1_score\" variables - One from sklearn and the other that you defined.\n\nIf I were you, I would check if this namespace confusion is causing a problem." }, { "id": 1029605, "postDate": "2020-09-27T23:30:00.667Z", "content": "

Hi @ramarlina

\n
\n

Single CNN model with melspectrograms (derived from public notebooks), with no augmentation, no mixup, using only primary label, with slightly modified training and inference mechanisms can get you to 0.575 Public LB.

\n
\n

This is so awesome.
\nI had the same idea and tried many things, but I couldn't make such a significant improvement.

\n

Now that the competition is over, can you tell us how to do that, if you would?

", "rawMarkdown": "Hi @ramarlina \n\n> Single CNN model with melspectrograms (derived from public notebooks), with no augmentation, no mixup, using only primary label, with slightly modified training and inference mechanisms can get you to 0.575 Public LB.\n\nThis is so awesome.\nI had the same idea and tried many things, but I couldn't make such a significant improvement.\n\nNow that the competition is over, can you tell us how to do that, if you would?" } ] }, { "id": 983520, "postDate": "2020-08-24T11:27:01.530Z", "content": "

Augmentations and proper training strategy

", "rawMarkdown": "Augmentations and proper training strategy", "votes": 5, "replies": [ { "id": 983605, "postDate": "2020-08-24T12:44:41.590Z", "content": "

Can you please explain what training strategy is good?

\n

Thank you

", "rawMarkdown": "Can you please explain what training strategy is good?\n\nThank you", "votes": -4 }, { "id": 983701, "postDate": "2020-08-24T14:23:50.483Z", "content": "

Thanks so much! will explore augmentation

", "rawMarkdown": "Thanks so much! will explore augmentation ", "votes": 1 }, { "id": 987735, "postDate": "2020-08-27T13:32:02.293Z", "content": "

I implemented some basic augmentations (shifts, noise, time-freq masks etc.), training and validation f1 both increase, but LB falls drastically. What could be the problem?

", "rawMarkdown": "I implemented some basic augmentations (shifts, noise, time-freq masks etc.), training and validation f1 both increase, but LB falls drastically. What could be the problem?" } ] }, { "id": 983481, "postDate": "2020-08-24T10:43:32.963Z", "content": "

I think Careful handling of \"No call\" in test set will give you +0.57x

\n

I joined the competition very lately..I will likely spend my time on this if ever I get serious about the comp :p

", "rawMarkdown": "I think Careful handling of \"No call\" in test set will give you +0.57x \n\nI joined the competition very lately..I will likely spend my time on this if ever I get serious about the comp :p", "votes": 6, "replies": [ { "id": 983699, "postDate": "2020-08-24T14:23:10.660Z", "rawMarkdown": "", "votes": -1, "isDeleted": true }, { "id": 984002, "postDate": "2020-08-24T18:53:18.910Z", "content": "

I'm wondering if some top teams are using a binary Bird/NoBird classifier (or additional head) prior to BirdCall prediction? I'm not but if someone have some feedback about such idea …

", "rawMarkdown": "I'm wondering if some top teams are using a binary Bird/NoBird classifier (or additional head) prior to BirdCall prediction? I'm not but if someone have some feedback about such idea ...", "votes": 5 }, { "id": 984070, "postDate": "2020-08-24T20:01:43.940Z", "content": "

@mpware We're not doing such thing but I've thought about it
\nThe main issue being that the whole training data is not labeled for the task

", "rawMarkdown": "@mpware We're not doing such thing but I've thought about it\nThe main issue being that the whole training data is not labeled for the task", "votes": 6 }, { "id": 984178, "postDate": "2020-08-24T23:07:15.720Z", "content": "

I'm not doing either.

", "rawMarkdown": "I'm not doing either.", "votes": 8 }, { "id": 984289, "postDate": "2020-08-25T02:52:19.263Z", "content": "

I just submitted using 2 models, one binary and one multiclass. Decreased score from .569 to 0.371. It feels like nothing I do in this competition works. Predict 'nocall' if binary < threshold1 (tuned from validation), else predict maximum confident bird

", "rawMarkdown": "I just submitted using 2 models, one binary and one multiclass. Decreased score from .569 to 0.371. It feels like nothing I do in this competition works. Predict 'nocall' if binary < threshold1 (tuned from validation), else predict maximum confident bird", "votes": 4 } ] }, { "id": 983684, "postDate": "2020-08-24T14:02:12.823Z", "content": "

If you try public models on test example data, you will see that they are not generalizing at all. The public LB score may be due to high percentage of \"nocall\" in public test data.

\n

I think with post processing (inference) and LB feedback, score might go up to 0.58 (just a speculation, haven't tried). But, if model is not generalizing it means nothing for private LB.

", "rawMarkdown": "If you try public models on test example data, you will see that they are not generalizing at all. The public LB score may be due to high percentage of \"nocall\" in public test data.\n\nI think with post processing (inference) and LB feedback, score might go up to 0.58 (just a speculation, haven't tried). But, if model is not generalizing it means nothing for private LB.", "votes": 3, "replies": [ { "id": 983705, "postDate": "2020-08-24T14:25:08.843Z", "content": "

True, huge shake up in recent comps.

", "rawMarkdown": "True, huge shake up in recent comps. ", "votes": 1 } ] }, { "id": 985571, "postDate": "2020-08-25T20:28:07.717Z", "content": "

you can try augmetation. It is helpful. or try anorther training stragy

", "rawMarkdown": "you can try augmetation. It is helpful. or try anorther training stragy", "votes": 1 }, { "id": 984980, "postDate": "2020-08-25T12:13:34.583Z", "content": "

Try to generate the vectors for the audio.

", "rawMarkdown": "Try to generate the vectors for the audio." }, { "id": 983909, "postDate": "2020-08-24T17:19:27.243Z", "content": "

@Ramarlina did you use any clip wise post-processing in interface

", "rawMarkdown": "@Ramarlina did you use any clip wise post-processing in interface", "replies": [ { "id": 984055, "postDate": "2020-08-24T19:49:32.447Z", "content": "

Not currently, but I think it's worth looking into. I'm working on incorporating some ideas from the SED
\n public notebook.

", "rawMarkdown": "Not currently, but I think it's worth looking into. I'm working on incorporating some ideas from the SED \n public notebook.", "votes": 2 }, { "id": 984056, "postDate": "2020-08-24T19:49:32.933Z", "rawMarkdown": "", "isDeleted": true }, { "id": 984097, "postDate": "2020-08-24T20:33:14.227Z", "content": "

thanks for the info

", "rawMarkdown": "thanks for the info\n" }, { "id": 984110, "postDate": "2020-08-24T20:46:22.153Z", "content": "

https://github.com/arunodhayan/cornelBirdCall-2020111/blob/master/tor-try.ipynb
\nThis is my baseline in interface can anyone suggest me any improvements

", "rawMarkdown": "https://github.com/arunodhayan/cornelBirdCall-2020111/blob/master/tor-try.ipynb\nThis is my baseline in interface can anyone suggest me any improvements", "votes": -3 }, { "id": 984246, "postDate": "2020-08-25T01:49:30.877Z", "content": "

I cannot show something too much, here's my 2 cents regarding site 3.
\nWhat happens if we take argmax for entire time axis, it could be:

\n", "rawMarkdown": "I cannot show something too much, here's my 2 cents regarding site 3.\nWhat happens if we take argmax for entire time axis, it could be:\n- Short chirp, clear, predicted strong, but last in short frames -> if we take average for entire frames...\n- Long chirp, dirty and not sure if it is really a chirp, predicted weak, but happens in entire frames -> ...", "votes": 2 }, { "id": 984799, "postDate": "2020-08-25T09:43:09.497Z", "content": "

so i should concentrate in site3 interface

", "rawMarkdown": "so i should concentrate in site3 interface" }, { "id": 985150, "postDate": "2020-08-25T14:35:17.690Z", "content": "

@arunodhayan I looked at your notebook, and here's my 2 cents.

\n

I'm not in any position to rate or anything, but I think you have all the elements in there to at least beat 0.568. You might want to validate each decision to make sure it helps not hurts. Decisions like: your choice of sampling rate, image sizes, sampling based on SNR etc.

\n

I haven't been able to make sampling based on sound-to-noise ratio work for me, but I know others have had some success with it.

", "rawMarkdown": "@arunodhayan I looked at your notebook, and here's my 2 cents.\n\nI'm not in any position to rate or anything, but I think you have all the elements in there to at least beat 0.568. You might want to validate each decision to make sure it helps not hurts. Decisions like: your choice of sampling rate, image sizes, sampling based on SNR etc.\n\nI haven't been able to make sampling based on sound-to-noise ratio work for me, but I know others have had some success with it.\n ", "votes": 1 }, { "id": 990970, "postDate": "2020-08-30T02:38:15.570Z", "content": "

@daisukelab

\n

I appreciate your input.

\n
\n

but last in short frames

\n
\n

What does this mean?
\nFor example, do you mean the last 1 second of 11 seconds of data?

", "rawMarkdown": "@daisukelab \n\n\nI appreciate your input.\n\n> but last in short frames\n\nWhat does this mean?\nFor example, do you mean the last 1 second of 11 seconds of data?\n" }, { "id": 991014, "postDate": "2020-08-30T04:06:25.987Z", "content": "

@fkubota please read this 'last' as 'exist'.

", "rawMarkdown": "@fkubota please read this 'last' as 'exist'.", "votes": 1 }, { "id": 991029, "postDate": "2020-08-30T04:48:19.117Z", "content": "

@daisukelab

\n

Thank you for your quick reply!
\nI understand now.

\n

I will use it as a reference.

", "rawMarkdown": "@daisukelab \n\nThank you for your quick reply!\nI understand now.\n\nI will use it as a reference." } ] } ], "comments": [ { "id": 983419, "author_name": "Mendrika Ramarlina", "author_url": "", "post_date": "2020-08-24T09:44:02.147000", "content": "

Single CNN model with melspectrograms (derived from public notebooks), with no augmentation, no mixup, using only primary label, with slightly modified training and inference mechanisms can get you to 0.575 Public LB.

\n

Echoing what others have said in the forum, the biggest gain I've seen so far did not come from model architecture or ensembling, rather from how training and inference methods addresses the things that make this competition challenging.

", "votes": 26, "replies": [ { "id": 983471, "author_name": "Testing", "author_url": "", "post_date": "2020-08-24T10:29:47.070000", "content": "

Can you give any insight about your training mechanism?

", "votes": -2, "replies": [] }, { "id": 983693, "author_name": "Salaryman", "author_url": "", "post_date": "2020-08-24T14:15:54.223000", "content": "

@ramarlina Thanks so much!
\nI will stick on fine tuning a single CNN and hope to beat the baseline😃

", "votes": 2, "replies": [] }, { "id": 984154, "author_name": "Louka Ewington-Pitsos", "author_url": "", "post_date": "2020-08-24T22:05:54.810000", "content": "

Thanks for sharing mate!

", "votes": 0, "replies": [] }, { "id": 986492, "author_name": "Mr_KnowNothing", "author_url": "", "post_date": "2020-08-26T14:48:14.077000", "content": "

@ramarlina Can you please tell which architecture you are using ?? is it possible to go above 0.57+ with an effnet??

", "votes": 0, "replies": [] }, { "id": 986560, "author_name": "yash chaudhary", "author_url": "", "post_date": "2020-08-26T15:32:50.610000", "content": "

@tanulsingh077 I am trying my reliable effnet only.

", "votes": 0, "replies": [] }, { "id": 986623, "author_name": "Mendrika Ramarlina", "author_url": "", "post_date": "2020-08-26T16:30:17.827000", "content": "

@tanulsingh077 I'm using a resnet right now, but my CV tells me that you could get comparable results (potentially better) with an effnet. I'm not focused on selecting the most optimal backbone yet at this point.

", "votes": 5, "replies": [] }, { "id": 986631, "author_name": "Theo Viel", "author_url": "", "post_date": "2020-08-26T16:33:15.177000", "content": "

@tanulsingh077 It is possible to beat 0.57 with an EfficientNet

", "votes": 5, "replies": [] }, { "id": 986643, "author_name": "Mr_KnowNothing", "author_url": "", "post_date": "2020-08-26T16:50:35.870000", "content": "

Thanks @ramarlina and @theoviel for the answer . Also @theoviel Is it possible to do that without augmentations , mixups , introducing noise etc and with single label multiclass approach ?? I have been trying from two days but my cv f1 score remains at 0 , even the train f1 score remains at zero , I am using everything same as public notebooks ☹️

", "votes": 0, "replies": [] }, { "id": 986663, "author_name": "Salaryman", "author_url": "", "post_date": "2020-08-26T17:09:40.367000", "content": "

even my cv f1 score is not 0… the model I trained for 2 days just got LB=0.0😂😂😂
\nshould be sth wrong with the prediction loop/ mapping

", "votes": 0, "replies": [] }, { "id": 986676, "author_name": "Mr_KnowNothing", "author_url": "", "post_date": "2020-08-26T17:17:41.360000", "content": "

@fiyeroleung

\n

`
\nclass BirdcallModel(nn.Module):
\n def init(self,num_classes):
\n super(BirdcallModel, self).init()

\n
    self.num_classes = num_classes\n    self.model = EfficientNet.from_pretrained(\n        f\"efficientnet-b{EFFNET_VER}\"\n    )\n\n    self.model._fc = nn.Sequential(\n        nn.Linear(1280, 1024), nn.ReLU(), nn.Dropout(p=0.2),\n        nn.Linear(1024, 1024), nn.ReLU(), nn.Dropout(p=0.2),\n        nn.Linear(1024, self.num_classes))\n\ndef forward(self, image):\n    return self.model(image)\n
\n

`

\n

This is the model and then I take softmax followed by argmax to get the predicted values , I don't know what's going wrong

", "votes": 0, "replies": [] }, { "id": 986700, "author_name": "", "author_url": "", "post_date": "2020-08-26T17:39:07.307000", "content": "", "votes": 0, "replies": [] }, { "id": 986702, "author_name": "Salaryman", "author_url": "", "post_date": "2020-08-26T17:43:56.337000", "content": "

@tanulsingh077 Don't get me wrong, I also have the problem of LB=0😂
\nim reading line by line to see which part goes wrong

", "votes": 0, "replies": [] }, { "id": 987001, "author_name": "Theo Viel", "author_url": "", "post_date": "2020-08-26T22:44:16.497000", "content": "

If your f1 score remains at 0 you probably have an implementation mistake somewhere

", "votes": 0, "replies": [] }, { "id": 987214, "author_name": "( ͡° ͜ʖ ͡°)", "author_url": "", "post_date": "2020-08-27T05:03:05.400000", "content": "

@ramarlina no augmentation mean that you use the random audio with duration (s) or you cut/ padding with the fix length ?

", "votes": 1, "replies": [] }, { "id": 987281, "author_name": "Mendrika Ramarlina", "author_url": "", "post_date": "2020-08-27T06:16:54.763000", "content": "

Data augmentations techniques for spectrograms like warping, masking blocks of frequency channels, and masking blocks of time steps. Maybe it's my implementation, but it hasn't helped my CV so far.

", "votes": 0, "replies": [] }, { "id": 987346, "author_name": "Manh Lab", "author_url": "", "post_date": "2020-08-27T07:15:14.727000", "content": "

Sorry! But how did you choose 5seconds.

", "votes": 0, "replies": [] }, { "id": 987423, "author_name": "Mendrika Ramarlina", "author_url": "", "post_date": "2020-08-27T08:38:29.993000", "content": "

I've experimented with a lot of things to ultimately resort back to random sampling clips for training.

", "votes": 1, "replies": [] }, { "id": 987440, "author_name": "Mr_KnowNothing", "author_url": "", "post_date": "2020-08-27T09:05:01.073000", "content": "

@theoviel @ramarlina I am really sorry I am asking , but its been three days and I have not been able to solve this , I will be highly grateful if you can have a look at my code here
\nhttps://drive.google.com/file/d/1O3eBTMcqV4d7jp0DhRSFwwBQ2ySLcjga/view?usp=sharing

\n

I am really sorry for asking for this , I don't want spoon feeding but I just can't figure out what's wrong and I have also wasted this week's gpu quota as well

", "votes": 0, "replies": [] }, { "id": 987452, "author_name": "Mendrika Ramarlina", "author_url": "", "post_date": "2020-08-27T09:20:11.113000", "content": "

@tanulsingh077 been there, I have 4 hours of GPU left, so I feel ya. Are you having any specific issue?

", "votes": 0, "replies": [] }, { "id": 987455, "author_name": "Mr_KnowNothing", "author_url": "", "post_date": "2020-08-27T09:23:28.590000", "content": "

Yes @ramarlina My f1 score remains at zero and just doesn't increase no matter which model I use , I have read the training code , evaluation code , dataset code , f1 score implementation code line by line ,but I can't find what's wrong , I am not doing anything fancy but still

", "votes": 0, "replies": [] }, { "id": 987471, "author_name": "Mendrika Ramarlina", "author_url": "", "post_date": "2020-08-27T09:43:44.893000", "content": "

Does your training loss improve? How long did you run the model?

\n

One thing I can point out immediately is that, I only iterate the learning rate after the model completes an epoch. I call scheduler.step() outside and after the for loop.

", "votes": 0, "replies": [] }, { "id": 987482, "author_name": "Mr_KnowNothing", "author_url": "", "post_date": "2020-08-27T09:49:56.693000", "content": "

I run it for 50 epochs , my train_loss decreases and valid loss also decreases , thanks for the scheduler tip I will us it

", "votes": 0, "replies": [] }, { "id": 987492, "author_name": "Mendrika Ramarlina", "author_url": "", "post_date": "2020-08-27T09:56:37.087000", "content": "

It's probably an issue with your f1 computation. What's your best val loss after 50 epochs? Also do you mean you get 0.0 LB f1 or 0.0 validation f1?

", "votes": 0, "replies": [] }, { "id": 987551, "author_name": "daisukelab", "author_url": "", "post_date": "2020-08-27T10:37:59.410000", "content": "

Trying with many mods so that it runs on my 1080, maybe I could reproduce your issue. Is that the Train_F1_score? It soon increases to around 0.5 and decreases later on.

\n
\n
######### Starting Training for epoch 1
\n

1%| | 11/1066 [00:11<15:24, 1.14it/s, Train_F1_score=0.24, Train_Loss=0.225]

\n
\n

Please don't hope too much, but would be able to take some time to look into tonight. But again, please don't expect me to.

", "votes": 0, "replies": [] }, { "id": 987571, "author_name": "daisukelab", "author_url": "", "post_date": "2020-08-27T10:58:22.250000", "content": "

@tanulsingh077 If I print your model's prediction stat like this, model is predicting smaller values.
\nIn F1Meter:

\n
    def update(self, y_true, y_pred,threshold=0.5):\n        y_true = y_true.detach().cpu().numpy()\n        yp = y_pred.detach().cpu().numpy()\n        print(yp.min(), yp.mean(), yp.max()) # <<<<<<<==== debugging here\n        y_pred = F.sigmoid(y_pred).detach().cpu().numpy()\n\n        y_pred[y_pred<threshold] = 0\n        y_pred[y_pred>threshold] = 1\n
\n

Printed like this, then many values after sigmoid() will be lower than threshold -> almost zero y_pred
\n -> less TP -> F1 might be smaller. it might be the direct reason for that.

\n
-0.30662918 0.00048712618 0.29636607\n  0%|          | 2/1066 [00:04<56:05,  3.16s/it, Train_F1_score=0.0428, Train_Loss=0.668]    \n-1.2256229 -0.10857863 0.36215597\n  0%|          | 3/1066 [00:05<42:13,  2.38s/it, Train_F1_score=0.238, Train_Loss=0.609] \n-3.2570004 -0.4962771 1.3743259\n  0%|          | 4/1066 [00:06<32:42,  1.85s/it, Train_F1_score=0.429, Train_Loss=0.518]\n-4.288979 -1.3962975 2.694023\n  :\n-41.389378 -20.891447 -0.23979145\n  1%|          | 11/1066 [00:11<16:00,  1.10it/s, Train_F1_score=0.247, Train_Loss=0.227]\n-57.430416 -22.627079 -0.5330948\n
\n

Did you check your data generated by Dataset is healthy?

", "votes": 2, "replies": [] }, { "id": 987605, "author_name": "Mr_KnowNothing", "author_url": "", "post_date": "2020-08-27T11:42:33", "content": "

Hello @daisukelab thanks for taking out the time to help me out , I really appreciate it . As you can see I am using the dataset class from public kernel without any changes I don't think that the dataset is corrupted ,and for confirmation I have checked it again

", "votes": 0, "replies": [] }, { "id": 987639, "author_name": "daisukelab", "author_url": "", "post_date": "2020-08-27T12:17:05.230000", "content": "

One small problem, DataLoader's shuffle is False by default. Maybe everybody just follow the notebook could be shuffle=False for training loader….

", "votes": 1, "replies": [] }, { "id": 987697, "author_name": "Mr_KnowNothing", "author_url": "", "post_date": "2020-08-27T13:00:29.050000", "content": "

@daisukelab @ramarlina I have tried measuring F1 score between epochs of this public kernel
\nhttps://www.kaggle.com/ttahara/training-birdsong-baseline-resnest50-fast
\nand it also gives zero f1 score , Now I am more confused , Just to check if my f1 meter class is not wring I tried measuring F1 at the end of each epoch using sklearn's f1 score but it suffered the same fate

", "votes": 0, "replies": [] }, { "id": 987714, "author_name": "Testing", "author_url": "", "post_date": "2020-08-27T13:13:27.397000", "content": "

In my case F1score is decreasing with training

", "votes": 0, "replies": [] }, { "id": 987728, "author_name": "Mendrika Ramarlina", "author_url": "", "post_date": "2020-08-27T13:25:39.997000", "content": "

Keep in mind that, with this training procedure, one \"epoch\" means the model has \"seen\" exactly a single random chunk of 5s from each audio in your dataset. The average duration of each file is 55s, that means you can expect the model to have seen ~9% of your data within one epoch. You'll need at least 12 epochs to complete the equivalent of a single full pass over the data. On top of that, the model is looking at random timestamps within the audio, is the model actually looking at birdcalls? noise? silence?

\n

TLDR; you'll need to let the model run for a few epochs before it can pick anything significant.

", "votes": 5, "replies": [] }, { "id": 988025, "author_name": "daisukelab", "author_url": "", "post_date": "2020-08-27T17:52:53.763000", "content": "

@tanulsingh077 I made further check - but not solved. Here's what have found so far.

\n\n

Umm.. my time is up. :(

\n

One last thing I can suggest is, we have this basic.

\n
\n

torch.Tensor([[-1, -0.1, 0, 0.1, 1]]).sigmoid() # ==> [[0.0474, 0.2689, 0.4750, 0.5000, 0.5250, 0.7311]]

\n
\n

As long as the F1 calculation threshould is 0.5, your model has to predict 0 or larger, but it outputs minus for the target label class. I would follow the basics if I could continue:

\n\n

Good luck…

", "votes": 1, "replies": [] }, { "id": 988160, "author_name": "Mr_KnowNothing", "author_url": "", "post_date": "2020-08-27T20:34:00.553000", "content": "

Thanks @daisukelab for helping me with this and your suggestions ,It's because of people like you ,people like me get the oppurtunity to flourish, I will work my way again through the code , if I am not able to debug it , guess I will use the public kernels then

", "votes": 0, "replies": [] }, { "id": 988359, "author_name": "Nacir Bouazizi", "author_url": "", "post_date": "2020-08-28T02:03:50.887000", "content": "

@tanulsingh077 here is a kernel that implements an F1 that seems to work with no issues in this competition. Maybe that may help you.

", "votes": 1, "replies": [] }, { "id": 988451, "author_name": "PIkachu", "author_url": "", "post_date": "2020-08-28T04:08:43.363000", "content": "

@nacirbouazizi I saw your kernel. There your model expect input shape of (bs,im_num, channel, height, width) what is im_num here

", "votes": 0, "replies": [] }, { "id": 989145, "author_name": "Mr_KnowNothing", "author_url": "", "post_date": "2020-08-28T15:19:50.583000", "content": "

Thanks @nacirbouazizi I will try it

", "votes": 1, "replies": [] }, { "id": 1001421, "author_name": "Vee", "author_url": "", "post_date": "2020-09-07T10:16:41.090000", "content": "

@tanulsingh077
\nI saw your code. Your code namespace has 2 \"f1_score\" variables - One from sklearn and the other that you defined.

\n

If I were you, I would check if this namespace confusion is causing a problem.

", "votes": 0, "replies": [] }, { "id": 1029605, "author_name": "fkubota", "author_url": "", "post_date": "2020-09-27T23:30:00.667000", "content": "

Hi @ramarlina

\n
\n

Single CNN model with melspectrograms (derived from public notebooks), with no augmentation, no mixup, using only primary label, with slightly modified training and inference mechanisms can get you to 0.575 Public LB.

\n
\n

This is so awesome.
\nI had the same idea and tried many things, but I couldn't make such a significant improvement.

\n

Now that the competition is over, can you tell us how to do that, if you would?

", "votes": 0, "replies": [] } ] }, { "id": 983520, "author_name": "Volodymyr", "author_url": "", "post_date": "2020-08-24T11:27:01.530000", "content": "

Augmentations and proper training strategy

", "votes": 5, "replies": [ { "id": 983605, "author_name": "Testing", "author_url": "", "post_date": "2020-08-24T12:44:41.590000", "content": "

Can you please explain what training strategy is good?

\n

Thank you

", "votes": -4, "replies": [] }, { "id": 983701, "author_name": "Salaryman", "author_url": "", "post_date": "2020-08-24T14:23:50.483000", "content": "

Thanks so much! will explore augmentation

", "votes": 1, "replies": [] }, { "id": 987735, "author_name": "Stanislav Blinov", "author_url": "", "post_date": "2020-08-27T13:32:02.293000", "content": "

I implemented some basic augmentations (shifts, noise, time-freq masks etc.), training and validation f1 both increase, but LB falls drastically. What could be the problem?

", "votes": 0, "replies": [] } ] }, { "id": 983481, "author_name": "Serigne ", "author_url": "", "post_date": "2020-08-24T10:43:32.963000", "content": "

I think Careful handling of \"No call\" in test set will give you +0.57x

\n

I joined the competition very lately..I will likely spend my time on this if ever I get serious about the comp :p

", "votes": 6, "replies": [ { "id": 983699, "author_name": "", "author_url": "", "post_date": "2020-08-24T14:23:10.660000", "content": "", "votes": -1, "replies": [] }, { "id": 984002, "author_name": "MPWARE", "author_url": "", "post_date": "2020-08-24T18:53:18.910000", "content": "

I'm wondering if some top teams are using a binary Bird/NoBird classifier (or additional head) prior to BirdCall prediction? I'm not but if someone have some feedback about such idea …

", "votes": 5, "replies": [] }, { "id": 984070, "author_name": "Theo Viel", "author_url": "", "post_date": "2020-08-24T20:01:43.940000", "content": "

@mpware We're not doing such thing but I've thought about it
\nThe main issue being that the whole training data is not labeled for the task

", "votes": 6, "replies": [] }, { "id": 984178, "author_name": "Hidehisa Arai", "author_url": "", "post_date": "2020-08-24T23:07:15.720000", "content": "

I'm not doing either.

", "votes": 8, "replies": [] }, { "id": 984289, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2020-08-25T02:52:19.263000", "content": "

I just submitted using 2 models, one binary and one multiclass. Decreased score from .569 to 0.371. It feels like nothing I do in this competition works. Predict 'nocall' if binary < threshold1 (tuned from validation), else predict maximum confident bird

", "votes": 4, "replies": [] } ] }, { "id": 983684, "author_name": "Manjesh Gupta", "author_url": "", "post_date": "2020-08-24T14:02:12.823000", "content": "

If you try public models on test example data, you will see that they are not generalizing at all. The public LB score may be due to high percentage of \"nocall\" in public test data.

\n

I think with post processing (inference) and LB feedback, score might go up to 0.58 (just a speculation, haven't tried). But, if model is not generalizing it means nothing for private LB.

", "votes": 3, "replies": [ { "id": 983705, "author_name": "Salaryman", "author_url": "", "post_date": "2020-08-24T14:25:08.843000", "content": "

True, huge shake up in recent comps.

", "votes": 1, "replies": [] } ] }, { "id": 985571, "author_name": "Manh Lab", "author_url": "", "post_date": "2020-08-25T20:28:07.717000", "content": "

you can try augmetation. It is helpful. or try anorther training stragy

", "votes": 1, "replies": [] }, { "id": 984980, "author_name": "itsshavar", "author_url": "", "post_date": "2020-08-25T12:13:34.583000", "content": "

Try to generate the vectors for the audio.

", "votes": 0, "replies": [] }, { "id": 983909, "author_name": "Arunodhayan", "author_url": "", "post_date": "2020-08-24T17:19:27.243000", "content": "

@Ramarlina did you use any clip wise post-processing in interface

", "votes": 0, "replies": [ { "id": 984055, "author_name": "Mendrika Ramarlina", "author_url": "", "post_date": "2020-08-24T19:49:32.447000", "content": "

Not currently, but I think it's worth looking into. I'm working on incorporating some ideas from the SED
\n public notebook.

", "votes": 2, "replies": [] }, { "id": 984056, "author_name": "", "author_url": "", "post_date": "2020-08-24T19:49:32.933000", "content": "", "votes": 0, "replies": [] }, { "id": 984097, "author_name": "Arunodhayan", "author_url": "", "post_date": "2020-08-24T20:33:14.227000", "content": "

thanks for the info

", "votes": 0, "replies": [] }, { "id": 984110, "author_name": "Arunodhayan", "author_url": "", "post_date": "2020-08-24T20:46:22.153000", "content": "

https://github.com/arunodhayan/cornelBirdCall-2020111/blob/master/tor-try.ipynb
\nThis is my baseline in interface can anyone suggest me any improvements

", "votes": -3, "replies": [] }, { "id": 984246, "author_name": "daisukelab", "author_url": "", "post_date": "2020-08-25T01:49:30.877000", "content": "

I cannot show something too much, here's my 2 cents regarding site 3.
\nWhat happens if we take argmax for entire time axis, it could be:

\n", "votes": 2, "replies": [] }, { "id": 984799, "author_name": "Arunodhayan", "author_url": "", "post_date": "2020-08-25T09:43:09.497000", "content": "

so i should concentrate in site3 interface

", "votes": 0, "replies": [] }, { "id": 985150, "author_name": "Mendrika Ramarlina", "author_url": "", "post_date": "2020-08-25T14:35:17.690000", "content": "

@arunodhayan I looked at your notebook, and here's my 2 cents.

\n

I'm not in any position to rate or anything, but I think you have all the elements in there to at least beat 0.568. You might want to validate each decision to make sure it helps not hurts. Decisions like: your choice of sampling rate, image sizes, sampling based on SNR etc.

\n

I haven't been able to make sampling based on sound-to-noise ratio work for me, but I know others have had some success with it.

", "votes": 1, "replies": [] }, { "id": 990970, "author_name": "fkubota", "author_url": "", "post_date": "2020-08-30T02:38:15.570000", "content": "

@daisukelab

\n

I appreciate your input.

\n
\n

but last in short frames

\n
\n

What does this mean?
\nFor example, do you mean the last 1 second of 11 seconds of data?

", "votes": 0, "replies": [] }, { "id": 991014, "author_name": "daisukelab", "author_url": "", "post_date": "2020-08-30T04:06:25.987000", "content": "

@fkubota please read this 'last' as 'exist'.

", "votes": 1, "replies": [] }, { "id": 991029, "author_name": "fkubota", "author_url": "", "post_date": "2020-08-30T04:48:19.117000", "content": "

@daisukelab

\n

Thank you for your quick reply!
\nI understand now.

\n

I will use it as a reference.

", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "983376": "Among 1019 teams only 50 teams can beat the baseline ach >= 0.569 in public LB so far.\n\nAny hints to beat the 0.568 baseline?🙏🙏\nfinetune the CNN? PANN? other ensemble structure? proper augmentation?\nexternal dataset?\n\nMuch appreciate!\nThanks so much!\n ", "983419": "Single CNN model with melspectrograms (derived from public notebooks), with no augmentation, no mixup, using only primary label, with slightly modified training and inference mechanisms can get you to 0.575 Public LB. \n\nEchoing what others have said in the forum, the biggest gain I've seen so far did not come from model architecture or ensembling, rather from how training and inference methods addresses the things that make this competition challenging.", "983520": "Augmentations and proper training strategy", "983481": "I think Careful handling of \"No call\" in test set will give you +0.57x \n\nI joined the competition very lately..I will likely spend my time on this if ever I get serious about the comp :p", "983684": "If you try public models on test example data, you will see that they are not generalizing at all. The public LB score may be due to high percentage of \"nocall\" in public test data.\n\nI think with post processing (inference) and LB feedback, score might go up to 0.58 (just a speculation, haven't tried). But, if model is not generalizing it means nothing for private LB.", "985571": "you can try augmetation. It is helpful. or try anorther training stragy", "984980": "Try to generate the vectors for the audio.", "983909": "@Ramarlina did you use any clip wise post-processing in interface" } }