Maybe this loss function can be of use here, anyone tried it with success or can rule it out?
\nThe idea is that, when during training, a random ~5s clip is picked, that sound may not represent the label call well, it become mislabeled in the training. The Bi-Tempered Logistic Loss is created to handle this kind of problem.
\nhttps://ai.googleblog.com/2019/08/bi-tempered-logistic-loss-for-training.html
\nhttps://arxiv.org/pdf/1906.03361.pdf
Update I:
\nAfter a full train run - it works better(F1 validation score) than BCE(WL), and that with only a first test fold run without tuning the loss settings.
Update II:
\nDid a new run, with less randomness, given that mostly of the sounds are cut in random, so kept that same during both testing.
\nHad better validation score again and also a better LB score vs BCE loss, a small gain though but still, the val score also increases with some tuning of parameter settings, so maybe there are more gain to find. Or maybe this is all in the margin of error and just luck, but this is what I found within this loss.
\nI will keep the BiTLoss for now in my model training, maybe ensemble it with others, time will tell. Too much tuning, testing and training in other parts of this competition, so I'll leave further loss tuning for now :)
Maybe this loss function can be of use here, anyone tried it with success or can rule it out?
\nThe idea is that, when during training, a random ~5s clip is picked, that sound may not represent the label call well, it become mislabeled in the training. The Bi-Tempered Logistic Loss is created to handle this kind of problem.
\nhttps://ai.googleblog.com/2019/08/bi-tempered-logistic-loss-for-training.html
\nhttps://arxiv.org/pdf/1906.03361.pdf
Update I:
\nAfter a full train run - it works better(F1 validation score) than BCE(WL), and that with only a first test fold run without tuning the loss settings.
Update II:
\nDid a new run, with less randomness, given that mostly of the sounds are cut in random, so kept that same during both testing.
\nHad better validation score again and also a better LB score vs BCE loss, a small gain though but still, the val score also increases with some tuning of parameter settings, so maybe there are more gain to find. Or maybe this is all in the margin of error and just luck, but this is what I found within this loss.
\nI will keep the BiTLoss for now in my model training, maybe ensemble it with others, time will tell. Too much tuning, testing and training in other parts of this competition, so I'll leave further loss tuning for now :)
In other competition, I used the Bi-Tempered Logistic Loss function.
\nPlease use it if necessary. :)
Thanks. Yes I also had the same version saved from the same competition :) trying a custom version now for a fold run, with different settings, see how it goes.
", "rawMarkdown": "Thanks. Yes I also had the same version saved from the same competition :) trying a custom version now for a fold run, with different settings, see how it goes.", "votes": null }, { "id": "1316667", "postDate": "05/20/2021 19:02:46", "content": "I hadn't seen this before, thanks! Just quickly skimming so far… The paper seems focused on softmax loss, but we're generally allowed to have multiple birds in a segment (so, treating each label as an independent binary classification problem is usually seen as preferable to softmax'ing). Do you make any modifications for allowing multiple labels?
", "rawMarkdown": "I hadn't seen this before, thanks! Just quickly skimming so far... The paper seems focused on softmax loss, but we're generally allowed to have multiple birds in a segment (so, treating each label as an independent binary classification problem is usually seen as preferable to softmax'ing). Do you make any modifications for allowing multiple labels?", "votes": null }, { "id": "1317620", "postDate": "05/21/2021 14:52:49", "content": "Yes it's a great loss function in general.
\nIt’s just an idea to the problem with the mislabeled croped samples and maybe find a solution for it, which I wanted to share the community and take feedback on, if some already had tested this or other similar techniques. 🙂 This loss function was just a quick idea that crossed my mind, there are many losses out there for handle mislabeled images to test on.
\nI'm still testing it and I'm passing it through a sigmoid part for the probabilities instead of directly using the softmax part of it, and with some other changes to the default settings - it works, if it's better than say bce, the full run will tell.
", "rawMarkdown": "Yes it's a great loss function in general.\n\nIt’s just an idea to the problem with the mislabeled croped samples and maybe find a solution for it, which I wanted to share the community and take feedback on, if some already had tested this or other similar techniques. 🙂 This loss function was just a quick idea that crossed my mind, there are many losses out there for handle mislabeled images to test on.\n\nI'm still testing it and I'm passing it through a sigmoid part for the probabilities instead of directly using the softmax part of it, and with some other changes to the default settings - it works, if it's better than say bce, the full run will tell.", "votes": null }, { "id": "1318451", "postDate": "05/22/2021 10:45:35", "content": "Update: After a full train run - it works better(F1 validation score) than BCE(WL), and that with only a first test fold run without tuning the loss settings.
", "rawMarkdown": "Update: After a full train run - it works better(F1 validation score) than BCE(WL), and that with only a first test fold run without tuning the loss settings.", "votes": null }, { "id": "1319222", "postDate": "05/23/2021 03:20:30", "content": "great works,thanks
", "rawMarkdown": "great works,thanks", "votes": null }, { "id": "1319397", "postDate": "05/23/2021 06:45:36", "content": "@kirderf thanks you get a better lB also with it ?
", "rawMarkdown": "kirderf thanks you get a better lB also with it ?", "votes": null }, { "id": "1319771", "postDate": "05/23/2021 13:39:49", "content": "@kirderf thanks, good idea. what did you use for loss settings for the first run t1 and t2?
", "rawMarkdown": "kirderf thanks, good idea. what did you use for loss settings for the first run t1 and t2?", "votes": null }, { "id": "1321600", "postDate": "05/24/2021 19:35:06", "content": "Yes slightly better LB score as well
", "rawMarkdown": "Yes slightly better LB score as well", "votes": null }, { "id": "1321603", "postDate": "05/24/2021 19:48:52", "content": "Update II:
\nDid a new run, with less randomness, given that mostly of the sounds are cut in random, so kept that same during both testing.
\nHad better validation score again and also a better LB score vs BCE loss, a small gain though but still, the val score also increases with some tuning of parameter settings, so maybe there are more gain to find. Or maybe this is all in the margin of error and just luck, but this is what I found within this loss.
\nI will keep the BiTLoss for now in my model training, maybe ensemble it with others, time will tell. Too much tuning, testing and training in other parts of this competition, so I'll leave further loss tuning for now :)
Thanks for looking into this, Kerderf!
\nI've got a number of 'good heuristic' methods for getting much-better-than-random training segments from the Xeno-Canto recordings. It seems like one would want to do this kind of loss tuning on top of easy heuristics, to reduce the amount+impact of 'bad' training segments in the first place.
\nI also found this interesting paper on label smoothing with weak labels, after wondering why there wasn't some comparison to label smoothing in the bitempered loss paper:
\nhttp://www.sanjivk.com/LabelSmoothing_ICML20.pdf
Thanks 👍 Yes I'm also using label smoothing in the BiTLoss. Also testing other methods like majority/full mixup/cutmix training with extra 'no call/noisy label' to help(?) the classifing, and then find a good PP to the prob..Have seen in a paper that fmix and cutmix is slighly better than mixup, link below. Though I'm only using cutmix/mixup so far. Best is of course is to try only segment true calls before training.
\nTo small variations between the result to be groundbreaking but maybe in a final ensembling it can help. Just ideas, fun to evaluate …🙂
\n\nB. Audio Classification
\n
\n We now perform experiments on the Google Commands
\n data set, which was created to promote deep learning research
\n on speech recognition problems. It is comprised of 65,000 one
\n second utterances of one of 30 words, with 10 of those words
\n being the target classes and the rest considered unrelated or
\n background noise. We perform MSDA on a Mel-frequency
\n spectrogram of each utterance. The results for a PreAct
\n ResNet-18 are given in Table II. We evaluate FMix, MixUp,
\n and CutMix for the standard α = 1 used for the majority of
\n our experiments and α = 0.2 recommended by Zhang et al.
\n [2] for MixUp. We see in both cases that FMix and CutMix
\n improve performance over MixUp outside the margin of error,
\n with the best result achieved by FMix with α = 1.
https://arxiv.org/abs/2002.12047
", "rawMarkdown": "Thanks 👍 Yes I'm also using label smoothing in the BiTLoss. Also testing other methods like majority/full mixup/cutmix training with extra 'no call/noisy label' to help(?) the classifing, and then find a good PP to the prob..Have seen in a paper that fmix and cutmix is slighly better than mixup, link below. Though I'm only using cutmix/mixup so far. Best is of course is to try only segment true calls before training.\nTo small variations between the result to be groundbreaking but maybe in a final ensembling it can help. Just ideas, fun to evaluate …🙂\n\n> B. Audio Classification\nWe now perform experiments on the Google Commands\ndata set, which was created to promote deep learning research\non speech recognition problems. It is comprised of 65,000 one\nsecond utterances of one of 30 words, with 10 of those words\nbeing the target classes and the rest considered unrelated or\nbackground noise. We perform MSDA on a Mel-frequency\nspectrogram of each utterance. The results for a PreAct\nResNet-18 are given in Table II. We evaluate FMix, MixUp,\nand CutMix for the standard α = 1 used for the majority of\nour experiments and α = 0.2 recommended by Zhang et al.\n[2] for MixUp. We see in both cases that FMix and CutMix\nimprove performance over MixUp outside the margin of error,\nwith the best result achieved by FMix with α = 1.\n\nhttps://arxiv.org/abs/2002.12047", "votes": null }, { "id": "1329130", "postDate": "05/30/2021 21:38:17", "content": "Hmm, I don't seem to be doing it right. The blog says, \"Setting both t1 and t2 to 1.0 recovers the logistic loss function,\" but, when I try those settings, I don't get the same results as BCEWithLogitsLoss.
\nimport torch\nfrom torch import nn\n\ndevice = \"cpu\"\n\nactivations = torch.FloatTensor([[-0.5, 0.1, 2.0]]).to(device)\nlabels = torch.FloatTensor([[0.2, 0.5, 0.3]]).to(device)\n\nloss = bi_tempered_logistic_loss(activations=activations, labels=labels, t1=1.0, t2=1.0)\nprint(\"Loss, t1=1.0, t2=1.0: \", loss)\n\ncriterion = nn.BCEWithLogitsLoss()\nloss = criterion(activations, labels).item()\nprint(\"Loss BCEWithLogitLoss: \", loss)\nThese numbers should be equal, right?
\nOriginal Github Repot
In other competition, I used the Bi-Tempered Logistic Loss function.
\nPlease use it if necessary. :)
Thanks. Yes I also had the same version saved from the same competition :) trying a custom version now for a fold run, with different settings, see how it goes.
", "votes": null, "replies": [] }, { "id": 1319222, "author_name": "hanson0910", "author_url": "", "post_date": "05/23/2021 03:20:30", "content": "great works,thanks
", "votes": null, "replies": [] } ] }, { "id": 1316667, "author_name": "tomdenton", "author_url": "", "post_date": "05/20/2021 19:02:46", "content": "I hadn't seen this before, thanks! Just quickly skimming so far… The paper seems focused on softmax loss, but we're generally allowed to have multiple birds in a segment (so, treating each label as an independent binary classification problem is usually seen as preferable to softmax'ing). Do you make any modifications for allowing multiple labels?
", "votes": null, "replies": [ { "id": 1317620, "author_name": "kirderf", "author_url": "", "post_date": "05/21/2021 14:52:49", "content": "Yes it's a great loss function in general.
\nIt’s just an idea to the problem with the mislabeled croped samples and maybe find a solution for it, which I wanted to share the community and take feedback on, if some already had tested this or other similar techniques. 🙂 This loss function was just a quick idea that crossed my mind, there are many losses out there for handle mislabeled images to test on.
\nI'm still testing it and I'm passing it through a sigmoid part for the probabilities instead of directly using the softmax part of it, and with some other changes to the default settings - it works, if it's better than say bce, the full run will tell.
", "votes": null, "replies": [] }, { "id": 1318451, "author_name": "kirderf", "author_url": "", "post_date": "05/22/2021 10:45:35", "content": "Update: After a full train run - it works better(F1 validation score) than BCE(WL), and that with only a first test fold run without tuning the loss settings.
", "votes": null, "replies": [] }, { "id": 1319397, "author_name": "jaideepvalani", "author_url": "", "post_date": "05/23/2021 06:45:36", "content": "@kirderf thanks you get a better lB also with it ?
", "votes": null, "replies": [] }, { "id": 1319771, "author_name": "aliabdin1", "author_url": "", "post_date": "05/23/2021 13:39:49", "content": "@kirderf thanks, good idea. what did you use for loss settings for the first run t1 and t2?
", "votes": null, "replies": [] }, { "id": 1321600, "author_name": "kirderf", "author_url": "", "post_date": "05/24/2021 19:35:06", "content": "Yes slightly better LB score as well
", "votes": null, "replies": [] } ] }, { "id": 1321603, "author_name": "kirderf", "author_url": "", "post_date": "05/24/2021 19:48:52", "content": "Update II:
\nDid a new run, with less randomness, given that mostly of the sounds are cut in random, so kept that same during both testing.
\nHad better validation score again and also a better LB score vs BCE loss, a small gain though but still, the val score also increases with some tuning of parameter settings, so maybe there are more gain to find. Or maybe this is all in the margin of error and just luck, but this is what I found within this loss.
\nI will keep the BiTLoss for now in my model training, maybe ensemble it with others, time will tell. Too much tuning, testing and training in other parts of this competition, so I'll leave further loss tuning for now :)
Thanks for looking into this, Kerderf!
\nI've got a number of 'good heuristic' methods for getting much-better-than-random training segments from the Xeno-Canto recordings. It seems like one would want to do this kind of loss tuning on top of easy heuristics, to reduce the amount+impact of 'bad' training segments in the first place.
\nI also found this interesting paper on label smoothing with weak labels, after wondering why there wasn't some comparison to label smoothing in the bitempered loss paper:
\nhttp://www.sanjivk.com/LabelSmoothing_ICML20.pdf
Thanks 👍 Yes I'm also using label smoothing in the BiTLoss. Also testing other methods like majority/full mixup/cutmix training with extra 'no call/noisy label' to help(?) the classifing, and then find a good PP to the prob..Have seen in a paper that fmix and cutmix is slighly better than mixup, link below. Though I'm only using cutmix/mixup so far. Best is of course is to try only segment true calls before training.
\nTo small variations between the result to be groundbreaking but maybe in a final ensembling it can help. Just ideas, fun to evaluate …🙂
\n\nB. Audio Classification
\n
\n We now perform experiments on the Google Commands
\n data set, which was created to promote deep learning research
\n on speech recognition problems. It is comprised of 65,000 one
\n second utterances of one of 30 words, with 10 of those words
\n being the target classes and the rest considered unrelated or
\n background noise. We perform MSDA on a Mel-frequency
\n spectrogram of each utterance. The results for a PreAct
\n ResNet-18 are given in Table II. We evaluate FMix, MixUp,
\n and CutMix for the standard α = 1 used for the majority of
\n our experiments and α = 0.2 recommended by Zhang et al.
\n [2] for MixUp. We see in both cases that FMix and CutMix
\n improve performance over MixUp outside the margin of error,
\n with the best result achieved by FMix with α = 1.
https://arxiv.org/abs/2002.12047
", "votes": null, "replies": [] } ] }, { "id": 1329130, "author_name": "philipkd", "author_url": "", "post_date": "05/30/2021 21:38:17", "content": "Hmm, I don't seem to be doing it right. The blog says, \"Setting both t1 and t2 to 1.0 recovers the logistic loss function,\" but, when I try those settings, I don't get the same results as BCEWithLogitsLoss.
\nimport torch\nfrom torch import nn\n\ndevice = \"cpu\"\n\nactivations = torch.FloatTensor([[-0.5, 0.1, 2.0]]).to(device)\nlabels = torch.FloatTensor([[0.2, 0.5, 0.3]]).to(device)\n\nloss = bi_tempered_logistic_loss(activations=activations, labels=labels, t1=1.0, t2=1.0)\nprint(\"Loss, t1=1.0, t2=1.0: \", loss)\n\ncriterion = nn.BCEWithLogitsLoss()\nloss = criterion(activations, labels).item()\nprint(\"Loss BCEWithLogitLoss: \", loss)\nThese numbers should be equal, right?
\nOriginal Github Repot