I am using SED method discussed here. My best 1 fold model achieves 0.80 on LB.
\nI was expecting that combining 5 models from 5 folds would improve my result, but my 5 folds model performs worse than a single fold model.
\nHere is what I did:
\nStandard Stratified 5 fold Splits
\nI trained each fold model for 30 epochs and save the model that achieves the highest LWLRAP on the validation set for that fold. Then I try to methods to combine the test prediction: 1) averaging 2) approach detailed here.
\nAppreciate your input. Thanks ! 🙏
", "messages": [ { "id": 1161648, "postDate": "2021-01-20T17:05:56.443Z", "content": "I am using SED method discussed here. My best 1 fold model achieves 0.80 on LB.
\nI was expecting that combining 5 models from 5 folds would improve my result, but my 5 folds model performs worse than a single fold model.
\nHere is what I did:
\nStandard Stratified 5 fold Splits
\nI trained each fold model for 30 epochs and save the model that achieves the highest LWLRAP on the validation set for that fold. Then I try to methods to combine the test prediction: 1) averaging 2) approach detailed here.
\nAppreciate your input. Thanks ! 🙏
", "rawMarkdown": "I am using SED method discussed [here](https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/208830). My best 1 fold model achieves 0.80 on LB.\n\nI was expecting that combining 5 models from 5 folds would improve my result, but my 5 folds model performs worse than a single fold model. \n\nHere is what I did:\n\nStandard Stratified 5 fold Splits \n\nI trained each fold model for 30 epochs and save the model that achieves the highest LWLRAP on the validation set for that fold. Then I try to methods to combine the test prediction: 1) averaging 2) approach detailed [here](https://www.kaggle.com/kneroma/rfcx-bagging). \n\n### Questions:\n\n1. Are there any obvious mistake in my approach ? \n2. For people who had success in improving score after combining 5 folds, what did you do differently ? \n\n\nAppreciate your input. Thanks ! 🙏", "votes": 7 }, { "id": 1162198, "postDate": "2021-01-21T03:31:31.500Z", "content": "I think you are right.
I used EfficientNetB3 with BCE. And a single model's score was 0.86-0.89. Finally, these 5-folds average ensemble score was 0.913.
\n\nI trained each fold model for 30 epochs and save the model that achieves the highest LWLRAP on the validation set for that fold.
\n
Your local LWLRAP may not be correlation with LB. In my environment, it is more correlated with Loss value than LWLRAP.
", "rawMarkdown": "1. I think you are right.\n\n2. I used EfficientNetB3 with BCE. And a single model's score was 0.86-0.89. Finally, these 5-folds average ensemble score was 0.913. \n\n> I trained each fold model for 30 epochs and save the model that achieves the highest LWLRAP on the validation set for that fold. \n\nYour local LWLRAP may not be correlation with LB. In my environment, it is more correlated with Loss value than LWLRAP.", "votes": 1, "replies": [ { "id": 1162441, "postDate": "2021-01-21T06:37:27.650Z", "content": "@shinmura0
\nBTW,
\nWhat is your experience with using larger backbone ?
One of your earlier post mentioned that you are doing EfficientNetB0. Did B3 outperform B0 in your setup ?
\nI saw another person mentioned he uses B5 ( same PANNs arch).
\nI didn't try larger backbone because I saw other discussion about how larger model leads to worse result. But if larger model does help, then I will go ahead and try it out
", "rawMarkdown": "@shinmura0 \nBTW,\nWhat is your experience with using larger backbone ? \n\nOne of your earlier post mentioned that you are doing EfficientNetB0. Did B3 outperform B0 in your setup ?\n\nI saw another person mentioned he uses B5 ( same PANNs arch).\n\nI didn't try larger backbone because I saw other discussion about how larger model leads to worse result. But if larger model does help, then I will go ahead and try it out" }, { "id": 1162561, "postDate": "2021-01-21T08:15:26.857Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1162560, "postDate": "2021-01-21T08:15:26.857Z", "content": "\n\nDid B3 outperform B0 in your setup ?
\n
Yes. The following result is 5-folds average ensemble.
\n@trooperog
\nThanks for sharing your model score.
Here is a summary my SED approach. Any hint on how may i improve toward your single model result ?
\nArchitecture = PANNs SED architecture
\nBackbone = EfficientNetB0
\nAugmentation = [MixUp, RandomCrop of tp audio]
\nOptimizer = Adam with Cosine Annealing
\nData: log mel spectrogram
\nLoss: BCE
my SED arch is almost the same as yours , i did some changes in Panns official architecture and some more augmentations and a custom loss and also The backbone i used was efficientnet_b5_ns from timm github repo.
", "rawMarkdown": "my SED arch is almost the same as yours , i did some changes in Panns official architecture and some more augmentations and a custom loss and also The backbone i used was efficientnet_b5_ns from timm github repo.", "votes": 1 }, { "id": 1161869, "postDate": "2021-01-20T19:30:28.330Z", "content": "\nAny hint for where I should shop for custom loss ? Like what keyword to search in literature / medium.
\nDoes your loss specifically optimize for ranking objective ?
", "rawMarkdown": "@trooperog \n\nAny hint for where I should shop for custom loss ? Like what keyword to search in literature / medium. \n\nDoes your loss specifically optimize for ranking objective ? " }, { "id": 1161884, "postDate": "2021-01-20T19:45:09.967Z", "content": "in tensorflow there is a loss called SigmoidFocalCrossEntropy my custom loss is a pytorch implementation of this with some changes in its hyper parameters .
", "rawMarkdown": "in tensorflow there is a loss called SigmoidFocalCrossEntropy my custom loss is a pytorch implementation of this with some changes in its hyper parameters .", "votes": 2 }, { "id": 1161951, "postDate": "2021-01-20T20:58:16.930Z", "content": "I will look into it. Many thanks !
", "rawMarkdown": "I will look into it. Many thanks !" } ] }, { "id": 1162076, "postDate": "2021-01-21T00:36:11.163Z", "content": "It's an interesting question here.
\nPerhaps I just didn't get what other people are doing. As far as I see, it's pretty hard to get a proper local estimation of your model's LWLRAP for 60s audio. I don't have it at all.
\nThe given train_tp and train_fp only contains some information of the training audios, namely, in specific windows there is or is not a particular species. It doesn't say anything about other species in the specified windows, nor anything outside of the windows. So, we don't have accurate full set of species that exists in one 60s audio, which is necessary to compute a trustworthy LWLRAP.
What do you think?
", "rawMarkdown": "It's an interesting question here. \n\nPerhaps I just didn't get what other people are doing. As far as I see, it's pretty hard to get a proper local estimation of your model's LWLRAP for 60s audio. I don't have it at all.\n\nThe given `train_tp` and `train_fp` only contains some information of the training audios, namely, in specific windows there is or is not a particular species. It doesn't say anything about other species in the specified windows, nor anything outside of the windows. So, we don't have accurate full set of species that exists in one 60s audio, which is necessary to compute a trustworthy LWLRAP.\n\nWhat do you think?", "replies": [ { "id": 1162086, "postDate": "2021-01-21T01:17:29.050Z", "content": "You are saying that the tp labels do not provide sufficient info for us to compute clipwise_lwrap. Therefore, my 5 fold best model based on clipwise_lwrap are probably sub-optimal / random ?
\nWhat kind of validation scheme are you doing ? I used to validate on random 10 secs crop of audio that contains the annotated event. But I am getting overly optimistic performance estimate( val lwrap = 0.87 and LB = 0.80). After I did clipwise_lwrap, the over estimation issue is much reduced.
", "rawMarkdown": "You are saying that the tp labels do not provide sufficient info for us to compute clipwise_lwrap. Therefore, my 5 fold best model based on clipwise_lwrap are probably sub-optimal / random ?\n\nWhat kind of validation scheme are you doing ? I used to validate on random 10 secs crop of audio that contains the annotated event. But I am getting overly optimistic performance estimate( val lwrap = 0.87 and LB = 0.80). After I did clipwise_lwrap, the over estimation issue is much reduced.\n\n", "votes": 1 }, { "id": 1162094, "postDate": "2021-01-21T01:43:04.023Z", "content": "\n\n`You are saying that the tp labels do not provide sufficient info for us to compute clipwise_lwrap. Therefore, my 5 fold best model based on clipwise_lwrap are probably sub-optimal / random ?
\n
Yes, I think the raw train_tp does not have enough information for either 60s audios or the small windows it points to. In some sense, I didn't validate, I just train for a fixed number of epochs.
Perhaps one-hot type of training labels made simply from train_tp is also an inaccurate loss. Bird calls together, but we have only one non-zero label for each sample.
I think you are right.
I used EfficientNetB3 with BCE. And a single model's score was 0.86-0.89. Finally, these 5-folds average ensemble score was 0.913.
\n\nI trained each fold model for 30 epochs and save the model that achieves the highest LWLRAP on the validation set for that fold.
\n
Your local LWLRAP may not be correlation with LB. In my environment, it is more correlated with Loss value than LWLRAP.
", "votes": 1, "replies": [ { "id": 1162441, "author_name": "NakedKoala", "author_url": "", "post_date": "2021-01-21T06:37:27.650000", "content": "@shinmura0
\nBTW,
\nWhat is your experience with using larger backbone ?
One of your earlier post mentioned that you are doing EfficientNetB0. Did B3 outperform B0 in your setup ?
\nI saw another person mentioned he uses B5 ( same PANNs arch).
\nI didn't try larger backbone because I saw other discussion about how larger model leads to worse result. But if larger model does help, then I will go ahead and try it out
", "votes": 0, "replies": [] }, { "id": 1162561, "author_name": "", "author_url": "", "post_date": "2021-01-21T08:15:26.857000", "content": "", "votes": 0, "replies": [] }, { "id": 1162560, "author_name": "shinmura0", "author_url": "", "post_date": "2021-01-21T08:15:26.857000", "content": "\n\nDid B3 outperform B0 in your setup ?
\n
Yes. The following result is 5-folds average ensemble.
\n@trooperog
\nThanks for sharing your model score.
Here is a summary my SED approach. Any hint on how may i improve toward your single model result ?
\nArchitecture = PANNs SED architecture
\nBackbone = EfficientNetB0
\nAugmentation = [MixUp, RandomCrop of tp audio]
\nOptimizer = Adam with Cosine Annealing
\nData: log mel spectrogram
\nLoss: BCE
my SED arch is almost the same as yours , i did some changes in Panns official architecture and some more augmentations and a custom loss and also The backbone i used was efficientnet_b5_ns from timm github repo.
", "votes": 1, "replies": [] }, { "id": 1161869, "author_name": "NakedKoala", "author_url": "", "post_date": "2021-01-20T19:30:28.330000", "content": "\nAny hint for where I should shop for custom loss ? Like what keyword to search in literature / medium.
\nDoes your loss specifically optimize for ranking objective ?
", "votes": 0, "replies": [] }, { "id": 1161884, "author_name": "Shubham Thapa", "author_url": "", "post_date": "2021-01-20T19:45:09.967000", "content": "in tensorflow there is a loss called SigmoidFocalCrossEntropy my custom loss is a pytorch implementation of this with some changes in its hyper parameters .
", "votes": 2, "replies": [] }, { "id": 1161951, "author_name": "NakedKoala", "author_url": "", "post_date": "2021-01-20T20:58:16.930000", "content": "I will look into it. Many thanks !
", "votes": 0, "replies": [] } ] }, { "id": 1162076, "author_name": "Buffalo Spdwy", "author_url": "", "post_date": "2021-01-21T00:36:11.163000", "content": "It's an interesting question here.
\nPerhaps I just didn't get what other people are doing. As far as I see, it's pretty hard to get a proper local estimation of your model's LWLRAP for 60s audio. I don't have it at all.
\nThe given train_tp and train_fp only contains some information of the training audios, namely, in specific windows there is or is not a particular species. It doesn't say anything about other species in the specified windows, nor anything outside of the windows. So, we don't have accurate full set of species that exists in one 60s audio, which is necessary to compute a trustworthy LWLRAP.
What do you think?
", "votes": 0, "replies": [ { "id": 1162086, "author_name": "NakedKoala", "author_url": "", "post_date": "2021-01-21T01:17:29.050000", "content": "You are saying that the tp labels do not provide sufficient info for us to compute clipwise_lwrap. Therefore, my 5 fold best model based on clipwise_lwrap are probably sub-optimal / random ?
\nWhat kind of validation scheme are you doing ? I used to validate on random 10 secs crop of audio that contains the annotated event. But I am getting overly optimistic performance estimate( val lwrap = 0.87 and LB = 0.80). After I did clipwise_lwrap, the over estimation issue is much reduced.
", "votes": 1, "replies": [] }, { "id": 1162094, "author_name": "Buffalo Spdwy", "author_url": "", "post_date": "2021-01-21T01:43:04.023000", "content": "\n\n`You are saying that the tp labels do not provide sufficient info for us to compute clipwise_lwrap. Therefore, my 5 fold best model based on clipwise_lwrap are probably sub-optimal / random ?
\n
Yes, I think the raw train_tp does not have enough information for either 60s audios or the small windows it points to. In some sense, I didn't validate, I just train for a fixed number of epochs.
Perhaps one-hot type of training labels made simply from train_tp is also an inaccurate loss. Bird calls together, but we have only one non-zero label for each sample.