This CBR competition is very tough as the test dataset is absolutely more complicated than the training one. There is no direct way to cross validate our models and getting consistent OOF w.r.t. the LB is just too hard. Sharing some ideas about my own journey could help some of us to re-orient our researchs:\n* Mels specs resizing just works fine : perhaps my backbone model design enforces some threshold w.r.t. to image size, things like kernel_size and model stride could explain that\n* Changing the backbone model has little effect\n* Big backbone model, while they don't necessarily lead to LB boost for me, didn't seem to overfit\n* Pretrained weights help in generalization\n* Feeding complicated neural wave embeddings to my model seems not to be great\n* ....
\n\nWhat about you, what works (or not) for you ?
\n\n\n", "messages": [ { "id": 950118, "postDate": "2020-07-29T07:41:55.677Z", "content": "Daring Ideas Are Like Chessmen Moved Forward: They May Be Beaten, But They May Start A Winning Game. – Johann Wolfgang Von Goethe
\n
This CBR competition is very tough as the test dataset is absolutely more complicated than the training one. There is no direct way to cross validate our models and getting consistent OOF w.r.t. the LB is just too hard. Sharing some ideas about my own journey could help some of us to re-orient our researchs:\n* Mels specs resizing just works fine : perhaps my backbone model design enforces some threshold w.r.t. to image size, things like kernel_size and model stride could explain that\n* Changing the backbone model has little effect\n* Big backbone model, while they don't necessarily lead to LB boost for me, didn't seem to overfit\n* Pretrained weights help in generalization\n* Feeding complicated neural wave embeddings to my model seems not to be great\n* ....
\n\nWhat about you, what works (or not) for you ?
\n\n\n", "rawMarkdown": "This CBR competition is very tough as the test dataset is absolutely more complicated than the training one. There is no direct way to cross validate our models and getting consistent OOF w.r.t. the LB is just too hard. Sharing some ideas about my own journey could help some of us to re-orient our researchs:\n* Mels specs resizing just works fine : perhaps my backbone model design enforces some threshold w.r.t. to image size, things like **kernel_size** and model **stride** could explain that\n* Changing the backbone model has little effect\n* Big backbone model, while they don't necessarily lead to LB boost for me, didn't seem to overfit\n* Pretrained weights help in generalization\n* Feeding complicated neural wave embeddings to my model seems not to be great\n* ....\n\nWhat about you, what works (or not) for you ? \n\n>**Daring Ideas Are Like Chessmen Moved Forward: They May Be Beaten, But They May Start A Winning Game. – Johann Wolfgang Von Goethe**", "votes": 47 }, { "id": 950317, "postDate": "2020-07-29T10:33:18.920Z", "content": "Daring Ideas Are Like Chessmen Moved Forward: They May Be Beaten, But They May Start A Winning Game. – Johann Wolfgang Von Goethe
\n
in my case \nadding random noise helped. \nResizing Mel did not have significant impact (~0.003). \nusing background labels did not help.
", "rawMarkdown": "in my case \nadding random noise helped. \nResizing Mel did not have significant impact (~0.003). \nusing background labels did not help. ", "votes": 11, "replies": [ { "id": 950327, "postDate": "2020-07-29T10:40:45.510Z", "content": "The random noise idea is a great one. Personnally, it's hard to me to find a good augmentation scheme as Mels Specs are not real images even if there are some augementation techniques for mel sepcs (Frequency Masking, Time Masking, ...)
", "rawMarkdown": "The random noise idea is a great one. Personnally, it's hard to me to find a good augmentation scheme as Mels Specs are not real images even if there are some augementation techniques for mel sepcs (Frequency Masking, Time Masking, ...)", "votes": 5 }, { "id": 950363, "postDate": "2020-07-29T11:06:22.720Z", "content": "i added random noise on the wavedata and then generate Mels spec out of it. i did try Mels spec augmentation but, at this LB i am not sure if it impacted my model .
", "rawMarkdown": "i added random noise on the wavedata and then generate Mels spec out of it. i did try Mels spec augmentation but, at this LB i am not sure if it impacted my model . ", "votes": 5 }, { "id": 951101, "postDate": "2020-07-29T21:59:52.977Z", "content": "I'm also using augmentations for waveform and it seems they are working both on CV / pubLB. As for melspec augmentations, I tried SpecAugment and Cutout but they don't help. I also used Mixup but it doesn't help either so far. I am thinking possible explanations for this, since I at first thought it might works, and thought out an idea that annotation level of train / test may not be the same:
\n\nI think just mixing raw waveform with some decaying factors and label it with the labels of the sources will do, but I haven’t tried this because there’s some more to do before this.
\n\n> unlock RCNN
\n\nI’ve already tried and thought it is only another alternative of aggregation over features from CNN to use recurrent layer: basically attention will do the same and better. Maybe good for ensemble, but it costs a bit in terms of the speed.
", "rawMarkdown": "I think just mixing raw waveform with some decaying factors and label it with the labels of the sources will do, but I haven’t tried this because there’s some more to do before this.\n\n> unlock RCNN \n\nI’ve already tried and thought it is only another alternative of aggregation over features from CNN to use recurrent layer: basically attention will do the same and better. Maybe good for ensemble, but it costs a bit in terms of the speed.", "votes": 5 }, { "id": 951168, "postDate": "2020-07-30T00:38:18.113Z", "content": "@hidehisaarai1213 You helped me in the other post, so I can make a suggestion for you here. Maybe you should try SpecMix. It combines SpecAugment with Mixup. (Source: https://www.kaggle.com/c/freesound-audio-tagging-2019/discussion/95382) Perhaps try it with high Mixup value parameters (30-50%) and you can get good results.
", "rawMarkdown": "@hidehisaarai1213 You helped me in the other post, so I can make a suggestion for you here. Maybe you should try SpecMix. It combines SpecAugment with Mixup. (Source: https://www.kaggle.com/c/freesound-audio-tagging-2019/discussion/95382) Perhaps try it with high Mixup value parameters (30-50%) and you can get good results. ", "votes": 4 }, { "id": 951253, "postDate": "2020-07-30T02:41:42.563Z", "rawMarkdown": "", "votes": -16, "isDeleted": true }, { "id": 951268, "postDate": "2020-07-30T03:02:14.443Z", "content": "> Maybe you should try SpecMix. It combines SpecAugment with Mixup.
\n\nThanks! I also joined FAT2019 and remember well some people got luck with SpecMix.
\n\nI just forgot to note that there is another problem in Mixup; if we mix melspecs naively then we may loose the co-occurence information. Therefore if we'd like to mix them, it's better to use location information to rule out inexistent co-occurence.
", "rawMarkdown": "> Maybe you should try SpecMix. It combines SpecAugment with Mixup.\n\nThanks! I also joined FAT2019 and remember well some people got luck with SpecMix. \n\nI just forgot to note that there is another problem in Mixup; if we mix melspecs naively then we may loose the co-occurence information. Therefore if we'd like to mix them, it's better to use location information to rule out inexistent co-occurence.", "votes": 4 }, { "id": 951273, "postDate": "2020-07-30T03:07:42.913Z", "content": "> An attention will really not be able to do
\n\nI do not get your point. Why do you think so?
\n\n> 2 More Hints (Had to give you were too far off) :-
\n\nI appreciate your kindness but if you believe you are hinting me, why not you use that information and prove that good on LB? I just don't trust something works / not works information without any experiment.
I've already tried attention (from the very beginning of this competition) and it gave me comparatively good result for both CV and pubLB. I've also found combining self-attention aggregation and GlobalMaxPooling aggregation does better and reported that here.
", "rawMarkdown": "I've already tried attention (from the very beginning of this competition) and it gave me comparatively good result for both CV and pubLB. I've also found combining self-attention aggregation and GlobalMaxPooling aggregation does better and reported that [here](https://www.kaggle.com/c/birdsong-recognition/discussion/167611).", "votes": 5 }, { "id": 951962, "postDate": "2020-07-30T14:21:44.727Z", "rawMarkdown": "", "votes": 3, "isDeleted": true }, { "id": 952582, "postDate": "2020-07-31T04:09:55.330Z", "content": "@hidehisaarai1213 Thanks for your good baseline and all these sharing. About this, I'm a little confused:
\n\n\n\n\nTherefore if we'd like to mix them, it's better to use location information to rule out inexistent co-occurence.
\n
Could you talk more details about this? Like how to use location information and how to rule out inexistent co-occurence , that would be very kind.
I haven't made concrete idea of this but let me try to explain...
\n\nAs you can see in https://ebird.org/home , different species live in different places. For example amewoo(https://ebird.org/species/amewoo) and lewwoo(https://ebird.org/species/lewwoo) do not overlap in terms of where they live(well it overlaps a bit but only a bit). We can also see co-occurrence of different species using primary label and secondary labels: some pairs of species never appear together.
This co-occurrence may be very beneficial because we can correct the model prediction with this and/or our model also uses this co-occurrence information implicitly.
\n\nIf we mix audio clips naively we may loose this, therefore we should not mix the calls from species pairs that are less likely to happen.
\n\nSad to say that this is not very easy I think, we need to check which species lives where...
", "rawMarkdown": "I haven't made concrete idea of this but let me try to explain...\n\nAs you can see in https://ebird.org/home , different species live in different places. For example `amewoo`(https://ebird.org/species/amewoo) and `lewwoo`(https://ebird.org/species/lewwoo) do not overlap in terms of where they live(well it overlaps a bit but only a bit). We can also see co-occurrence of different species using primary label and secondary labels: some pairs of species never appear together.\n\nThis co-occurrence may be very beneficial because we can correct the model prediction with this and/or our model also uses this co-occurrence information implicitly.\n\nIf we mix audio clips naively we may loose this, therefore we should not mix the calls from species pairs that are less likely to happen.\n\nSad to say that this is not very easy I think, we need to check which species lives where...", "votes": 12 }, { "id": 952595, "postDate": "2020-07-31T04:30:35.227Z", "content": "\n\n\nSad to say that this is not very easy I think, we need to check which species lives where…
\n
I agree with you. But this is a very brilliant idea, thanks for explaining it.
", "rawMarkdown": "> Sad to say that this is not very easy I think, we need to check which species lives where…\n\nI agree with you. But this is a very brilliant idea, thanks for explaining it.", "votes": 4 }, { "id": 965936, "postDate": "2020-08-11T01:53:03.813Z", "content": "@yuvaramsingh What is ur noise to data ratio in terms of size. I am trying to add noise but seems like I am adding too few. A bit of help would be greatly appreciated.
", "rawMarkdown": "@yuvaramsingh What is ur noise to data ratio in terms of size. I am trying to add noise but seems like I am adding too few. A bit of help would be greatly appreciated. ", "votes": 1 }, { "id": 966468, "postDate": "2020-08-11T12:47:12.003Z", "rawMarkdown": "" } ] }, { "id": 951111, "postDate": "2020-07-29T22:16:40.660Z", "content": "Is anyone else having issues getting the model to generalise within the Xeno-Canto recordings / being very picky about learning rate? I've made my own train/validation splits, but the validation loss ends up 3x higher, even with dropout and some basic augmentation.
", "rawMarkdown": "Is anyone else having issues getting the model to generalise within the Xeno-Canto recordings / being very picky about learning rate? I've made my own train/validation splits, but the validation loss ends up 3x higher, even with dropout and some basic augmentation.", "votes": 3, "replies": [ { "id": 951256, "postDate": "2020-07-30T02:44:53.167Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 950724, "postDate": "2020-07-29T15:13:42.807Z", "content": "well done.
", "rawMarkdown": "well done." }, { "id": 950286, "postDate": "2020-07-29T10:14:52.183Z", "rawMarkdown": "", "votes": 1, "isDeleted": true, "replies": [ { "id": 950322, "postDate": "2020-07-29T10:35:08.440Z", "content": "We all know that feeling when after reading your favorite paper about an over complicated model, you come to implementation and guess what : total failure ☹️ !
", "rawMarkdown": "We all know that feeling when after reading your favorite paper about an over complicated model, you come to implementation and guess what : total failure ☹️ !", "votes": 5 }, { "id": 950995, "postDate": "2020-07-29T19:31:07.763Z", "rawMarkdown": "", "votes": -7, "isDeleted": true }, { "id": 951039, "postDate": "2020-07-29T20:41:12.033Z", "content": "Ah ! There still some hope then ! Good luck.
", "rawMarkdown": "Ah ! There still some hope then ! Good luck." }, { "id": 951082, "postDate": "2020-07-29T21:29:54.837Z", "rawMarkdown": "", "votes": -6, "isDeleted": true }, { "id": 951114, "postDate": "2020-07-29T22:21:13.823Z", "content": "I would love hearing more about the next fixed LRCNN model if it's possible 👍
", "rawMarkdown": "I would love hearing more about the next fixed LRCNN model if it's possible 👍 ", "votes": 1 }, { "id": 951140, "postDate": "2020-07-29T23:29:23.330Z", "rawMarkdown": "", "votes": -6, "isDeleted": true }, { "id": 953215, "postDate": "2020-07-31T15:52:58.213Z", "rawMarkdown": "", "votes": -3, "isDeleted": true } ] } ], "comments": [ { "id": 950317, "author_name": "yuvaramsingh", "author_url": "", "post_date": "2020-07-29T10:33:18.920000", "content": "in my case \nadding random noise helped. \nResizing Mel did not have significant impact (~0.003). \nusing background labels did not help.
", "votes": 11, "replies": [ { "id": 950327, "author_name": "kkiller", "author_url": "", "post_date": "2020-07-29T10:40:45.510000", "content": "The random noise idea is a great one. Personnally, it's hard to me to find a good augmentation scheme as Mels Specs are not real images even if there are some augementation techniques for mel sepcs (Frequency Masking, Time Masking, ...)
", "votes": 5, "replies": [] }, { "id": 950363, "author_name": "yuvaramsingh", "author_url": "", "post_date": "2020-07-29T11:06:22.720000", "content": "i added random noise on the wavedata and then generate Mels spec out of it. i did try Mels spec augmentation but, at this LB i am not sure if it impacted my model .
", "votes": 5, "replies": [] }, { "id": 951101, "author_name": "Hidehisa Arai", "author_url": "", "post_date": "2020-07-29T21:59:52.977000", "content": "I'm also using augmentations for waveform and it seems they are working both on CV / pubLB. As for melspec augmentations, I tried SpecAugment and Cutout but they don't help. I also used Mixup but it doesn't help either so far. I am thinking possible explanations for this, since I at first thought it might works, and thought out an idea that annotation level of train / test may not be the same:
\n\nI think just mixing raw waveform with some decaying factors and label it with the labels of the sources will do, but I haven’t tried this because there’s some more to do before this.
\n\n> unlock RCNN
\n\nI’ve already tried and thought it is only another alternative of aggregation over features from CNN to use recurrent layer: basically attention will do the same and better. Maybe good for ensemble, but it costs a bit in terms of the speed.
", "votes": 5, "replies": [] }, { "id": 951168, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2020-07-30T00:38:18.113000", "content": "@hidehisaarai1213 You helped me in the other post, so I can make a suggestion for you here. Maybe you should try SpecMix. It combines SpecAugment with Mixup. (Source: https://www.kaggle.com/c/freesound-audio-tagging-2019/discussion/95382) Perhaps try it with high Mixup value parameters (30-50%) and you can get good results.
", "votes": 4, "replies": [] }, { "id": 951253, "author_name": "", "author_url": "", "post_date": "2020-07-30T02:41:42.563000", "content": "", "votes": -16, "replies": [] }, { "id": 951268, "author_name": "Hidehisa Arai", "author_url": "", "post_date": "2020-07-30T03:02:14.443000", "content": "> Maybe you should try SpecMix. It combines SpecAugment with Mixup.
\n\nThanks! I also joined FAT2019 and remember well some people got luck with SpecMix.
\n\nI just forgot to note that there is another problem in Mixup; if we mix melspecs naively then we may loose the co-occurence information. Therefore if we'd like to mix them, it's better to use location information to rule out inexistent co-occurence.
", "votes": 4, "replies": [] }, { "id": 951273, "author_name": "Hidehisa Arai", "author_url": "", "post_date": "2020-07-30T03:07:42.913000", "content": "> An attention will really not be able to do
\n\nI do not get your point. Why do you think so?
\n\n> 2 More Hints (Had to give you were too far off) :-
\n\nI appreciate your kindness but if you believe you are hinting me, why not you use that information and prove that good on LB? I just don't trust something works / not works information without any experiment.
I've already tried attention (from the very beginning of this competition) and it gave me comparatively good result for both CV and pubLB. I've also found combining self-attention aggregation and GlobalMaxPooling aggregation does better and reported that here.
", "votes": 5, "replies": [] }, { "id": 951962, "author_name": "", "author_url": "", "post_date": "2020-07-30T14:21:44.727000", "content": "", "votes": 3, "replies": [] }, { "id": 952582, "author_name": "Yu Kang", "author_url": "", "post_date": "2020-07-31T04:09:55.330000", "content": "@hidehisaarai1213 Thanks for your good baseline and all these sharing. About this, I'm a little confused:
\n\n\n\n\nTherefore if we'd like to mix them, it's better to use location information to rule out inexistent co-occurence.
\n
Could you talk more details about this? Like how to use location information and how to rule out inexistent co-occurence , that would be very kind.
I haven't made concrete idea of this but let me try to explain...
\n\nAs you can see in https://ebird.org/home , different species live in different places. For example amewoo(https://ebird.org/species/amewoo) and lewwoo(https://ebird.org/species/lewwoo) do not overlap in terms of where they live(well it overlaps a bit but only a bit). We can also see co-occurrence of different species using primary label and secondary labels: some pairs of species never appear together.
This co-occurrence may be very beneficial because we can correct the model prediction with this and/or our model also uses this co-occurrence information implicitly.
\n\nIf we mix audio clips naively we may loose this, therefore we should not mix the calls from species pairs that are less likely to happen.
\n\nSad to say that this is not very easy I think, we need to check which species lives where...
", "votes": 12, "replies": [] }, { "id": 952595, "author_name": "Yu Kang", "author_url": "", "post_date": "2020-07-31T04:30:35.227000", "content": "\n\n\nSad to say that this is not very easy I think, we need to check which species lives where…
\n
I agree with you. But this is a very brilliant idea, thanks for explaining it.
", "votes": 4, "replies": [] }, { "id": 965936, "author_name": "Pipe Runner (Old Account)", "author_url": "", "post_date": "2020-08-11T01:53:03.813000", "content": "@yuvaramsingh What is ur noise to data ratio in terms of size. I am trying to add noise but seems like I am adding too few. A bit of help would be greatly appreciated.
", "votes": 1, "replies": [] }, { "id": 966468, "author_name": "Kuldeep Vansadia", "author_url": "", "post_date": "2020-08-11T12:47:12.003000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 951111, "author_name": "errorfixrepeat", "author_url": "", "post_date": "2020-07-29T22:16:40.660000", "content": "Is anyone else having issues getting the model to generalise within the Xeno-Canto recordings / being very picky about learning rate? I've made my own train/validation splits, but the validation loss ends up 3x higher, even with dropout and some basic augmentation.
", "votes": 3, "replies": [ { "id": 951256, "author_name": "", "author_url": "", "post_date": "2020-07-30T02:44:53.167000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 950724, "author_name": "Elliot", "author_url": "", "post_date": "2020-07-29T15:13:42.807000", "content": "well done.
", "votes": 0, "replies": [] }, { "id": 950286, "author_name": "", "author_url": "", "post_date": "2020-07-29T10:14:52.183000", "content": "", "votes": 1, "replies": [ { "id": 950322, "author_name": "kkiller", "author_url": "", "post_date": "2020-07-29T10:35:08.440000", "content": "We all know that feeling when after reading your favorite paper about an over complicated model, you come to implementation and guess what : total failure ☹️ !
", "votes": 5, "replies": [] }, { "id": 950995, "author_name": "", "author_url": "", "post_date": "2020-07-29T19:31:07.763000", "content": "", "votes": -7, "replies": [] }, { "id": 951039, "author_name": "kkiller", "author_url": "", "post_date": "2020-07-29T20:41:12.033000", "content": "Ah ! There still some hope then ! Good luck.
", "votes": 0, "replies": [] }, { "id": 951082, "author_name": "", "author_url": "", "post_date": "2020-07-29T21:29:54.837000", "content": "", "votes": -6, "replies": [] }, { "id": 951114, "author_name": "kkiller", "author_url": "", "post_date": "2020-07-29T22:21:13.823000", "content": "I would love hearing more about the next fixed LRCNN model if it's possible 👍
", "votes": 1, "replies": [] }, { "id": 951140, "author_name": "", "author_url": "", "post_date": "2020-07-29T23:29:23.330000", "content": "", "votes": -6, "replies": [] }, { "id": 953215, "author_name": "", "author_url": "", "post_date": "2020-07-31T15:52:58.213000", "content": "", "votes": -3, "replies": [] } ] } ], "raw_markdown_by_id": { "950118": "This CBR competition is very tough as the test dataset is absolutely more complicated than the training one. There is no direct way to cross validate our models and getting consistent OOF w.r.t. the LB is just too hard. Sharing some ideas about my own journey could help some of us to re-orient our researchs:\n* Mels specs resizing just works fine : perhaps my backbone model design enforces some threshold w.r.t. to image size, things like **kernel_size** and model **stride** could explain that\n* Changing the backbone model has little effect\n* Big backbone model, while they don't necessarily lead to LB boost for me, didn't seem to overfit\n* Pretrained weights help in generalization\n* Feeding complicated neural wave embeddings to my model seems not to be great\n* ....\n\nWhat about you, what works (or not) for you ? \n\n>**Daring Ideas Are Like Chessmen Moved Forward: They May Be Beaten, But They May Start A Winning Game. – Johann Wolfgang Von Goethe**", "950317": "in my case \nadding random noise helped. \nResizing Mel did not have significant impact (~0.003). \nusing background labels did not help. ", "951111": "Is anyone else having issues getting the model to generalise within the Xeno-Canto recordings / being very picky about learning rate? I've made my own train/validation splits, but the validation loss ends up 3x higher, even with dropout and some basic augmentation.", "950724": "well done.", "950286": "" } }